search

mdawar / rq-exporter

Prometheus metrics exporter for Python RQ (Redis Queue).
MIT License
65 stars 28 forks source link

"Not a valid RQ worker key" error when using custom classes #6

Closed PaulFlanaganGenscape closed 3 years ago

PaulFlanaganGenscape commented 3 years ago 
$ docker-compose -f docker-compose.yml -f docker-compose.custom.yml up
Recreating rq-exporter_grafana_1                  ... done
Starting rq-exporter_redis_1        ... done
Recreating rq-exporter_prometheus_1               ... done
Starting rq-exporter_dashboard_1                  ... done
Starting rq-exporter_rq_exporter_1                ... done
Starting rq-exporter_rq_exporter_custom_classes_1 ... done
Starting rq-exporter_enqueue_1                    ... done
Starting rq-exporter_enqueue_custom_classes_1     ... done
Starting rq-exporter_worker_1                     ... done
Starting rq-exporter_worker_2                     ... done
Starting rq-exporter_worker_custom_classes_1      ... done
Starting rq-exporter_worker_custom_classes_2      ... done
Attaching to rq-exporter_redis_1, rq-exporter_grafana_1, rq-exporter_dashboard_1, rq-exporter_rq_exporter_1, rq-exporter_rq_exporter_custom_classes_1, rq-exporter_prometheus_1, rq-exporter_enqueue_1, rq-exporter_enqueue_custom_classes_1, rq-exporter_worker_1, rq-exporter_worker_2, rq-exporter_worker_custom_classes_1, rq-exporter_worker_custom_classes_2
dashboard_1                   | RQ Dashboard version 0.6.0
enqueue_custom_classes_1      | Using queues with custom RQ classes
enqueue_custom_classes_1      | Enqueuing job "process_data" on queue "high"
enqueue_1                     | Enqueuing job "process_data" on queue "low"
dashboard_1                   |  * Running on 0.0.0.0:9181
grafana_1                     | t=2020-11-30T17:24:16+0000 lvl=warn msg="phantomJS is deprecated and will be removed in a future release. You should consider migrating from phantomJS to grafana-image-renderer plugin. Read more at https://grafana.com/docs/grafana/latest/administration/image_rendering/" logger=rendering renderer=phantomJS
dashboard_1                   |  * Serving Flask app "rq_dashboard.cli" (lazy loading)
dashboard_1                   |  * Environment: production
dashboard_1                   |    WARNING: This is a development server. Do not use it in a production deployment.
dashboard_1                   |    Use a production WSGI server instead.
dashboard_1                   |  * Debug mode: off
redis_1                       | 1:C 30 Nov 2020 17:24:13.503 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1                       | 1:C 30 Nov 2020 17:24:13.503 # Redis version=5.0.10, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1                       | 1:C 30 Nov 2020 17:24:13.503 # Configuration loaded
rq_exporter_1                 | [2020-11-30 17:24:18] [rq_exporter] [INFO]: Serving the application on 0.0.0.0:9726
rq_exporter_custom_classes_1  | [2020-11-30 17:24:18] [rq_exporter] [INFO]: Serving the application on 0.0.0.0:9726
redis_1                       | 1:M 30 Nov 2020 17:24:13.505 * Running mode=standalone, port=6379.
redis_1                       | 1:M 30 Nov 2020 17:24:13.505 # Server initialized
redis_1                       | 1:M 30 Nov 2020 17:24:13.505 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
worker_2                      | 17:24:20 Worker rq:worker:c1f6df2c68ed486292a0deccabc6a049: started, version 1.6.1
worker_2                      | 17:24:20 Subscribing to channel rq:pubsub:c1f6df2c68ed486292a0deccabc6a049
redis_1                       | 1:M 30 Nov 2020 17:24:13.506 * Ready to accept connections
worker_2                      | 17:24:20 *** Listening on high, default, low...
worker_2                      | 17:24:20 Cleaning registries for queue: high
worker_2                      | 17:24:20 Cleaning registries for queue: default
worker_2                      | 17:24:20 Cleaning registries for queue: low
worker_2                      | 17:24:20 low: jobs.process_data(2) (55375777-b6d9-4ca5-80d9-3f3008366af1)
rq-exporter_prometheus_1 exited with code 1
worker_custom_classes_2       | 17:24:20 Worker rq:custom:worker:cc69e2c9c6c145ea823ccf84981bae67: started, version 1.6.1
worker_custom_classes_2       | 17:24:20 Subscribing to channel rq:pubsub:cc69e2c9c6c145ea823ccf84981bae67
worker_custom_classes_2       | 17:24:20 *** Listening on high, default, low...
worker_custom_classes_2       | 17:24:20 Cleaning registries for queue: high
worker_1                      | 17:24:20 Worker rq:worker:358aca2223bd4d2aa04c8ab57044f34e: started, version 1.6.1
worker_1                      | 17:24:20 Subscribing to channel rq:pubsub:358aca2223bd4d2aa04c8ab57044f34e
worker_custom_classes_1       | Traceback (most recent call last):
worker_custom_classes_1       |   File "/usr/local/bin/rq", line 8, in <module>
worker_custom_classes_1       |     sys.exit(main())
worker_custom_classes_1       |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
worker_custom_classes_1       |     return self.main(*args, **kwargs)
worker_custom_classes_1       |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
worker_custom_classes_1       |     rv = self.invoke(ctx)
worker_custom_classes_1       |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
worker_custom_classes_1       |     return _process_result(sub_ctx.command.invoke(sub_ctx))
worker_custom_classes_1       |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
worker_custom_classes_1       |     return ctx.invoke(self.callback, **ctx.params)
worker_custom_classes_1       |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
worker_custom_classes_1       |     return callback(*args, **kwargs)
worker_custom_classes_1       |   File "/usr/local/lib/python3.8/site-packages/rq/cli/cli.py", line 78, in wrapper
worker_custom_classes_1       |     return ctx.invoke(func, cli_config, *args[1:], **kwargs)
worker_custom_classes_1       |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
worker_custom_classes_1       |     return callback(*args, **kwargs)
worker_custom_classes_1       |   File "/usr/local/lib/python3.8/site-packages/rq/cli/cli.py", line 230, in worker
worker_custom_classes_1       |     cleanup_ghosts(cli_config.connection)
worker_custom_classes_1       |   File "/usr/local/lib/python3.8/site-packages/rq/contrib/legacy.py", line 25, in cleanup_ghosts
worker_custom_classes_1       |     for worker in Worker.all(connection=conn):
worker_custom_classes_1       |   File "/usr/local/lib/python3.8/site-packages/rq/worker.py", line 120, in all
worker_custom_classes_1       |     workers = [cls.find_by_key(as_text(key),
worker_custom_classes_1       |   File "/usr/local/lib/python3.8/site-packages/rq/worker.py", line 120, in <listcomp>
worker_custom_classes_1       |     workers = [cls.find_by_key(as_text(key),
worker_custom_classes_1       |   File "/usr/local/lib/python3.8/site-packages/rq/worker.py", line 146, in find_by_key
worker_custom_classes_1       |     raise ValueError('Not a valid RQ worker key: %s' % worker_key)
worker_custom_classes_1       | ValueError: Not a valid RQ worker key: rq:custom:worker:cc69e2c9c6c145ea823ccf84981bae67
worker_1                      | 17:24:20 *** Listening on high, default, low...
worker_custom_classes_2       | 17:24:20 Cleaning registries for queue: default
worker_custom_classes_2       | 17:24:20 Cleaning registries for queue: low
worker_custom_classes_2       | 17:24:20 high: jobs.process_data(2) (d04e44ff-31f8-4306-a987-331f0579ac36)
rq-exporter_worker_custom_classes_1 exited with code 1
worker_2                      | process_data: sleeping for 2 seconds
worker_2                      | 17:24:22 low: Job OK (55375777-b6d9-4ca5-80d9-3f3008366af1)
mdawar commented 3 years ago

Hi,

Thank you for reporting this error, actually this error is coming from the function rq.contrib.legacy.cleanup_ghosts, it is using the default Worker class, that's why the class method Worker.find_by_key is raising this error because it's checking if the worker name starts with the original prefix Worker.redis_worker_namespace_prefix (rq:worker:) and not the custom class' prefix CustomWorker.redis_worker_namespace_prefix (rq:custom:worker:).

This has to be fixed in the rq repo, the cleanup_ghosts must be fixed to use the custom worker class cli_config.worker_class.

If you want a temporary fix, use a custom worker prefix that starts with rq:worker:, for example:

class CustomWorker(Worker):
    redis_worker_namespace_prefix = 'rq:worker:custom'