search

telefonicaid / fiware-pep-steelskin

Telefonica's implementation of the FIWARE PEP GE
GNU Affero General Public License v3.0
0 stars 0 forks source link

Enable CPU profiling for better observability #489

Closed rg2011 closed 2 years ago

rg2011 commented 2 years ago

Since version 10.x, node.js provides a CPU profiling API that can be used to collect CPU profiles on-demand: https://nodejs.org/docs/latest-v10.x/api/inspector.html. The API was added as experimental in release 10.x and marked as stable in release 14.x

Having the ability to collect CPU profiles on-demand would greatly help to troubleshoot issues with PEP in production, where sometimes we find CPU usages unexpectedly high for the number of requests served.

These traces are usually enabled by adding an HTTP GET endpoint to the application, that enables trace-collection on demand for a few seconds, then disables it, and returns the profile collected during that interval.

AlvaroVega commented 2 years ago

Maybe a new docker image flavour? https://github.com/telefonicaid/fiware-pep-steelskin/blob/master/docker/Dockerfile or just a new parameter to start docker image? https://github.com/telefonicaid/fiware-pep-steelskin/blob/0c7bb57a1f2cf3a22e92d9d94e7298af4870f68a/docker/entrypoint.sh#L47

fgalan commented 2 years ago

Maybe a new docker image flavour? https://github.com/telefonicaid/fiware-pep-steelskin/blob/master/docker/Dockerfile or just a new parameter to start docker image?

Even better, maybe something that can be switch on/off via API? Eg:

PUT /admin/profiler?level=1 (to enable it)
PUT /admin/profiler?level=0 (to disable it)

although not sure if this is technically possible (I don't know anything about the Node inspector and how it works :)

(Note we already are using /admin API for log configuration, eg. PUT /admin/log?level=DEBUG)

fgalan commented 2 years ago

Btw, this issue can apply to other node-based componentes:

We maybe should create twin issues for them in their respective repositories.

AlvaroVega commented 2 years ago

btw I'm using 

node --inspect-brk=0.0.0.0 /opt/fiware-pep-steelskin/bin/pepProxy

in this point:

https://github.com/telefonicaid/fiware-pep-steelskin/blob/2b04b9f757f20ce5dfa92da711cce5e48f5292e6/docker/entrypoint.sh#L49

AlvaroVega commented 2 years ago

Btw, this issue can apply to other node-based componentes:

* IOTAs (all the suite)

* Perseo-FE

* STH

We maybe should create twin issues for them in their respective repositories.

For iotas you can use current image and add this sentence to docker-compose.yml:

command: node --inspect-brk=0.0.0.0 /opt/iotagent-json/bin/iotagent-json
fgalan commented 2 years ago

Fixed in PR https://github.com/telefonicaid/fiware-pep-steelskin/pull/515