Feature request | Allow configuration of visibility timeout

[steffiprofileprint]

Hi guys,

Our team has been using Celery Director for our workflows. Some of our tasks take longer than 1 hour to run, and we are running into this Celery + Redis issue, where long tasks are added to the queue repeatedly. The quick fix for this is to increase visibility_timeout in the broker transport options. I would like to propose to add a new optional environment variable DIRECTOR_VISIBILITY_TIMEOUT for the hidden config. The changes would something like the following in /director/settings.py:

...
# Celery configuration
  self.CELERY_CONF = {
      "task_always_eager": False,
      "broker_url": env.str("DIRECTOR_BROKER_URI", "redis://localhost:6379/0"),
      "result_backend": env.str(
          "DIRECTOR_RESULT_BACKEND_URI", "redis://localhost:6379/1"
      ),
      ### MODIFIED
      "broker_transport_options": {
          "master_name": "director",
          "visibility_timeout": env.int("DIRECTOR_VISIBILITY_TIMEOUT", 3600)
      },
     ###
  }
...

Thanks for reading and for the project, it has been very helpful to us! 👍

[ncrocfer]

Hello @steffiprofileprint ,

Indeed it can be a new setting to improve the user experience of Celery Director 👍