expose django-q timeout and retry to env vars

askap-vast / vast-pipeline

This repository holds the code of the Radio Transient detection pipeline for the VAST project.

https://vast-survey.org/vast-pipeline/

MIT License

7 stars 3 forks source link

expose django-q timeout and retry to env vars #589

Closed marxide closed 2 years ago

marxide commented 2 years ago

Allows the Django-Q config variables timeout and retry to be configured with env vars. Defaults to existing values if not configured in .env. [skip ci]

marxide commented 2 years ago

Our largest runs usually take about 12 hours. I'm not sure what caused the one that exceeded the 24h timeout, but a likely scenario is that the system was under load with other processes (data transfers, Jupyter servers, or something else). The run after the one that timed out took ~12 hours.

marxide commented 2 years ago

We're also going to remove the measurement pairs from the database soon (WIP) which will significantly reduce the running time.