Exception Investigation: builtins:TypeError (NoneType)

[ccostino]

This is one of the errors we've seen captured in New Relic that we'd like to dig into and understand, if not also resolve.

Error message: unsupported operand type(s) for +: 'NoneType' and 'datetime.timedelta' Exception: builtins:TypeError

Traceback(most recent call last):
File "/home/vcap/deps/0/bin/celery", line 8, in <module>
… /home/vcap/deps/0/python/lib/python3.12/site-packages/celery/__main__.py, line 15, in main
…home/vcap/deps/0/python/lib/python3.12/site-packages/celery/bin/celery.py, line 236, in main
  File /home/vcap/deps/0/python/lib/python3.12/site-packages/click/core.py, line 1157, in __call__
  File /home/vcap/deps/0/python/lib/python3.12/site-packages/click/core.py, line 1078, in main
  File /home/vcap/deps/0/python/lib/python3.12/site-packages/click/core.py, line 1688, in invoke
  File /home/vcap/deps/0/python/lib/python3.12/site-packages/click/core.py, line 1434, in invoke
  File /home/vcap/deps/0/python/lib/python3.12/site-packages/click/core.py, line 783, in invoke
…/home/vcap/deps/0/python/lib/python3.12/site-packages/click/decorators.py, line 33, in new_func
… /home/vcap/deps/0/python/lib/python3.12/site-packages/celery/bin/base.py, line 135, in caller
…home/vcap/deps/0/python/lib/python3.12/site-packages/celery/bin/worker.py, line 356, in worker
…e/vcap/deps/0/python/lib/python3.12/site-packages/celery/worker/worker.py, line 202, in start
…/home/vcap/deps/0/python/lib/python3.12/site-packages/celery/bootsteps.py, line 116, in start
…/home/vcap/deps/0/python/lib/python3.12/site-packages/celery/bootsteps.py, line 365, in start
…cap/deps/0/python/lib/python3.12/site-packages/celery/concurrency/base.py, line 130, in start
…/deps/0/python/lib/python3.12/site-packages/celery/concurrency/prefork.py, line 109, in on_start
…deps/0/python/lib/python3.12/site-packages/celery/concurrency/asynpool.py, line 464, in __init__
…le /home/vcap/deps/0/python/lib/python3.12/site-packages/billiard/pool.py, line 1045, in __init__
…deps/0/python/lib/python3.12/site-packages/celery/concurrency/asynpool.py, line 482, in _create_worker_process
…le /home/vcap/deps/0/python/lib/python3.12/site-packages/billiard/pool.py, line 1157, in _create_worker_process
…/home/vcap/deps/0/python/lib/python3.12/site-packages/billiard/process.py, line 120, in start
…/home/vcap/deps/0/python/lib/python3.12/site-packages/billiard/context.py, line 331, in _Popen
…me/vcap/deps/0/python/lib/python3.12/site-packages/billiard/popen_fork.py, line 22, in __init__
…me/vcap/deps/0/python/lib/python3.12/site-packages/billiard/popen_fork.py, line 77, in _launch
…/home/vcap/deps/0/python/lib/python3.12/site-packages/billiard/process.py, line 323, in _bootstrap
…/home/vcap/deps/0/python/lib/python3.12/site-packages/billiard/process.py, line 110, in run
…le /home/vcap/deps/0/python/lib/python3.12/site-packages/billiard/pool.py, line 291, in __call__
…le /home/vcap/deps/0/python/lib/python3.12/site-packages/billiard/pool.py, line 361, in workloop
…/home/vcap/deps/0/python/lib/python3.12/site-packages/celery/app/trace.py, line 651, in fast_trace_task
…/home/vcap/deps/0/python/lib/python3.12/site-packages/celery/app/trace.py, line 453, in trace_task
                                       File /home/vcap/app/app/__init__.py, line 412, in __call__
…/python/lib/python3.12/site-packages/newrelic/hooks/application_celery.py, line 123, in wrapper
…/home/vcap/deps/0/python/lib/python3.12/site-packages/celery/app/trace.py, line 736, in __protected_call__
                           File /home/vcap/app/app/celery/nightly_tasks.py, line 54, in cleanup_unfinished_jobs

Implementation Sketch and Acceptance Criteria

• [ ] Look through our log history at https://logs.fr.cloud.gov/ and see if any patterns emerge that might indicate what conditions cause this
• [ ] Try and replicate the error, if possible
• [ ] Follow up in this issue with any findings and possible remediation steps

Security Considerations

• Be mindful of not sharing sensitive information.
• We'd like to reduce the errors that occur in the application where possible to maintain application stability and performance, and reduce the chances that the application ends up in an error or invalid state.

[xlorepdarkhelm]

The line in the stack trace with the problem is:

acceptable_finish_time = job.processing_started + timedelta(minutes=5)

Which suggests that the job.processing_started is None when it is run, for this error. I'll begin tracing through the code to see where that might be the case.

[ccostino]

@xlorepdarkhelm, were you able to uncover anything more with this or is it dependent on some of the new logging that is slated to go out in the next release?

[xlorepdarkhelm]

It was dependent on some logging.