Open maybephilipp opened 1 month ago
I tried to create a repro repository mimicing what is happaning on my actual code and... maybe I miss something, but repro code doesn't create the same issue. I'm kinda tired and bleary-eyed already as I investigate it for 3 days...
Could someone from newrelic point me to other places for debugging newrelic somehow? Some extra debug logs that could show, for example very frequent execution of some newrelic functionality? My guess is that newrelic's green thread is ticking very frequently... Disabling newrelic on my prod for now... No idea how to solve
Let me know if you'd find a repro useful and I can try to produce one, but I'm also seeing this. It is most pronounced in CPU-constrained environments, I didn't notice it locally, but once I deployed to Google Cloud Run it quickly maxed the CPU.
Invoking Gunicorn like:
gunicorn --pythonpath /my_django_app --chdir /my_django_app --worker-tmp-dir /dev/shm --worker-class gevent --workers=5 --max-requests=1000 --access-logfile '-' --error-logfile '-' --bind 0.0.0.0:8000 wsgi:application
Starting new relic in my wsgi.py
file by simply doing:
newrelic.agent.initialize()
Hope that helps. A single request will trigger it in the case of Cloud Run.
I have New Relic turned off for my service that uses gevent worker model until this is resolved, so would be great if it gets figured out!
Thanks for the information to help with the reproduction! Looking into this
Hi there -- Any luck looking into this? Would love to get back to instrumenting our gevent model gunicorn processes again.
Hey I'm picking this ticket up from Lalleh. That get_pid and epoll info is intriguing. I see there is this patch https://github.com/newrelic/newrelic-python-agent/blob/main/newrelic/hooks/coroutines_gevent.py#L17-L29 and I wonder if this could be causing an issue. Could you try running with this branch and see if this fixes the issue?
Python agent makes python process to consume a lot of CPU in idle state.
Description When I turn python agent on, upon server start, when first request came and some queries ran, python process starts to eat 40-100% of CPU. If I disable python agent and restart server, the CPU is 0-3%. The high CPU level with python agent keeps the same while being idle (100% sure no requests coming).
Stack: Python, Flask, Gevent (tried both – just PyWSGI server and gunicorn – the situation is exactly same), SocketIO (with simple-websocket), Nginx.
Debugging the app showed that most of CPU time is spent looping green threads. strace showed that a lot of epoll_wait's going on when newrelic is up.
Expected Behavior No impact or low impact.
Troubleshooting or NR Diag results
NR Diag (nrdiag/linux/nrdiag_arm64 --suites python):
Let me know how can I share the nrdiag-output.zip as I assume there is some sensitive info in it.
py-spy
(hub.py just keeps growing after time)
austin-tui
main process:
any other thread:
cProfile snakeviz
The most interesting part: strace
After a lot of time of investigation I found a stackoverflow question (https://stackoverflow.com/questions/58973698/uwsgi-with-gevent-takes-up-100-cpu) which led me to idea of monitoring sys calls and turned out something in my python app just annihilating my server with epoll_wait and getting pid:
Green threads dump:
https://pastebin.com/mVrqs4VY
Done with:
Steps to Reproduce will try to create
Your Environment
Additional context nothing