Background Tasks UI Broken when using Redis Sentinel

[tagur87]

Deployment Type

Self-hosted

NetBox Version

v3.7.0

Python Version

3.11

Steps to Reproduce

1. Deploy Redis Sentinel instance
2. Deploy netbox with the following configuration for the REDIS_KEY

   REDIS = {
   'tasks': {
       'SENTINELS': [('<sentinel>', <port>)],
       'SENTINEL_SERVICE': 'mymaster',
       'SENTINEL_TIMEOUT': 10,
       'PASSWORD': '',
       'DATABASE': 0,
       'SSL': False,
       'INSECURE_SKIP_TLS_VERIFY': False,
   },
   'caching': {
       'SENTINELS': [('<sentinel>', <port>)],
       'SENTINEL_SERVICE': 'mymaster',
       'SENTINEL_TIMEOUT': 10,
       'PASSWORD': '',
       'DATABASE': 1,
       'SSL': False,
       'INSECURE_SKIP_TLS_VERIFY': False,
   },
   }

3. Browse to http://..../admin/background-tasks/
4. See error message

Expected Behavior

Background tasks page should work as normal when a regular redis instance is used.

Observed Behavior

Following error is returned from debug output:

Environment:

Request Method: GET
Request URL: https://admin-jdt-nsot.o11n-dev.jw-cd-lab-1.10aws.org/admin/background-tasks/

Django Version: 4.2.8
Python Version: 3.11.4
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.humanize',
 'django.forms',
 'corsheaders',
 'debug_toolbar',
 'graphiql_debug_toolbar',
 'django_filters',
 'django_tables2',
 'django_prometheus',
 'graphene_django',
 'mptt',
 'rest_framework',
 'social_django',
 'taggit',
 'timezone_field',
 'core',
 'account',
 'circuits',
 'dcim',
 'ipam',
 'extras',
 'tenancy',
 'users',
 'utilities',
 'virtualization',
 'vpn',
 'wireless',
 'django_rq',
 'drf_spectacular',
 'drf_spectacular_sidecar',
 'netbox_ip_reservations.IPReservationConfig']
Installed Middleware:
['graphiql_debug_toolbar.middleware.DebugToolbarMiddleware',
 'django_prometheus.middleware.PrometheusBeforeMiddleware',
 'corsheaders.middleware.CorsMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'django.middleware.security.SecurityMiddleware',
 'netbox.middleware.RemoteUserMiddleware',
 'netbox.middleware.CoreMiddleware',
 'netbox.middleware.MaintenanceModeMiddleware',
 'django_prometheus.middleware.PrometheusAfterMiddleware']

Traceback (most recent call last):
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/views/decorators/cache.py", line 62, in _wrapper_view_func
    response = view_func(request, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/contrib/auth/decorators.py", line 23, in _wrapper_view
    return view_func(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django_rq/views.py", line 37, in stats
    **get_scheduler_statistics(),
      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django_rq/utils.py", line 106, in get_scheduler_statistics
    conn_key = f"{connection_kwargs['host']}:{connection_kwargs['port']}/{connection_kwargs.get('db', 0)}"
                  ^^^^^^^^^^^^^^^^^^^^^^^^^

Exception Type: KeyError at /admin/background-tasks/
Exception Value: 'host'

[moonrail]

We run into this issue as well - it seems to have been introduced in django-rq 2.9.0 via this commit: https://github.com/rq/django-rq/commit/df3271341a3668717844dcdc9a4a77e64a6316d0

Not completely sure, but seeing our own settings, your settings and the examples of django-rq in their README.md it seems they did not test with Sentinel and simply broke support with this change.

Edit: We've downgraded django-rq to 2.8.1 and there the error does not occur. If someone else might verify this as well, we should open an issue at django-rq´s repository.

[jeremystretch]

@tagur87 @moonrail could one of you please raise a bug report with django-rq to confirm that this is an uptsream issue?

[arthanson]

There is a ticket in django-rq: https://github.com/rq/django-rq/issues/642