search

newrelic / newrelic-python-agent

New Relic Python Agent
https://docs.newrelic.com/docs/agents/python-agent
Apache License 2.0
174 stars 99 forks source link

Long running Django command terminated by newrelic-admin. #978

Closed thihara closed 8 months ago

thihara commented 10 months ago

Description We have a long running Django command that crunches some data, it runs around 3.5 to 4 hours in general once a week.

Recently due to a need to diagnose a performance issue I tried to integrate newrelic with this job.

The job terminated two times with the error message

Killed NEW_RELIC_CONFIG_FILE=newrelic.ini NEW_RELIC_ENVIRONMENT=production_env newrelic-admin run-python -u manage.py base_of_my_existance_today_job

What's interesting is that the job terminated in the middle once, with the same message. Then resumed, we have a resumption mechanism in place to retry a failed job.

Then when the job completed successfully, waited about 1.5 minutes and I saw the Kill message mentioned above.

The relevant part of the newrelic.ini file

[import-hook:django] instrumentation.scripts.django_admin = base_of_my_existance_today_job

Job is started with this command

NEW_RELIC_CONFIG_FILE=newrelic.ini NEW_RELIC_ENVIRONMENT=$CRONS_GROUP newrelic-admin run-python -u manage.py base_of_my_existance_today_job

Expected Behavior The job should complete without being killed and new relic should report the data in the portal.

Steps to Reproduce Enable new relic monitoring for a long running heavy Django command.

Your Environment Python version - 3.9 Django version - 3.2.14 newrelic agent version - 7.2.2.169

Additional context N/A

hmstepanek commented 8 months ago

The killed message just means that something killed the job. New Relic doesn't kill jobs like this. Likely it was issued from outside the job.