Set celery task limits and fix worker timeout

[aarontp]

Description of the change

• Sets various retry and timeout options for celery task setup
• Updates the worker timeout to kill all child processes if execution time limit is reached
• Adds counters for soft timeouts and uncaught exceptions
• Also fixes issue with server timeout which was causing timed out tasks not to be cleaned up properly.
• Removes unused SINGLE_RUN feature and config option.
• Changes worker start args from '--pool=solo to --concurrency=1 as a solo pool executes tasks directly instead of forking and cannot process celery timeouts.

After this change there will be four different timeouts that can occur:

• Turbinia worker timeout: This timeout happens when executing anything through TurbiniaTask.execute() and uses the timeout values set in the Job config in the turbinia config file.
• Celery Soft timeout: This will send a catchable celery.exceptions.SoftTimeLimitExceeded exception to the worker and should be caught in the TurbiniaTask.execute() or TurbiniaTask.run_wrapper() methods depending on the state of the Task. This is set slightly longer than the Turbinia workout timeout to allow the worker to clean-up and give results when possible. The buffer is defined by task_manager.CELERY_SOFT_TIMEOUT_BUFFER.
• Celery Hard timeout: After some buffer time after the other timeouts (task_manager.CELERY_HARD_TIMEOUT_BUFFER), celery is configured to kill the worker running the task. This is not catchable so we will not get any results back from the worker so the server will also need to time out the task.
• Turbinia Server task timeout: If the server does not get any results back from the worker it will time out the task and consider it unrecoverable. This happens task_manager.SERVER_TASK_TIMEOUT_BUFFER seconds after the Task timeout. This buffer is large because the timer starts from when the task is enqueued and can't account for any scheduling and other wait time. This is to avoid timing out valid tasks due to worker congestion.

Applicable issues

• fixes #818

Additional information

Checklist

• [x] All tests were successful.
• [ ] Unit tests added.
• [ ] Documentation updated.

[jleaniz]

@aarontp removing '--pool=solo' from worker.py:253 should enable time limits and avoid the blocking issues we were experiencing.

[aarontp]

@aarontp removing '--pool=solo' from worker.py:253 should enable time limits and avoid the blocking issues we were experiencing.

Indeed, that worked, thanks!

[aarontp]

PTAL, I added the soft limit exception to the execute() method as well.