search

oss-aspen / 8Knot

Dash app in development to serve open source community visualizations using GitHub data from Augur. Hosted app: https://eightknot.osci.io
MIT License
47 stars 59 forks source link

OCP app hanging, workers hitting time limit: #440

Closed JamesKunstle closed 11 months ago

JamesKunstle commented 1 year ago

Traceback:

[2023-09-11 15:13:10,393: ERROR/MainProcess] Task handler raised error: TimeLimitExceeded(84600) Traceback (most recent call last): File "/opt/app-root/lib64/python3.9/site-packages/billiard/pool.py", line 684, in on_hard_timeout raise TimeLimitExceeded(job._timeout) billiard.exceptions.TimeLimitExceeded: TimeLimitExceeded(84600,) [2023-09-11 15:13:10,393: ERROR/MainProcess] Hard time limit (84600s) exceeded for long_callback_af0d10c6ed5f47471a01d685531fadf619049c1a[b7166630-15f0-4f13-b758-e47822b06387] [2023-09-11 15:13:10,502: ERROR/MainProcess] Process 'ForkPoolWorker-12' pid:59 exited with 'signal 9 (SIGKILL)'

Unsure why this is happening- 84600 is a day in seconds so process was running for a long time, hogging queue space.

JamesKunstle commented 12 months ago

Solved, I think: background callbacks that wait for data to become available can hang forever if the data never becomes available.