search

neuromatch / NEXT

NEXT is a machine learning system that runs in the cloud and makes it easy to develop, evaluate, and apply active learning in the real-world. Ask better questions. Get better results. Faster. Automated.
http://nextml.org
Apache License 2.0
1 stars 0 forks source link

Fix the redis max connection bug #7

Closed iamzoltan closed 1 month ago

iamzoltan commented 2 months ago

This only occurred when stress testing 200 participants. I think it may be the cause of failures in the past. After restarting the dockers mid stress test, everything resolved as expected.

Possible solutions:

The combination of both would be useful.

iamzoltan commented 2 months ago 
 File "/next_backend/next/broker/broker.py", line 57, in applyAsync
nextbackenddocker-1  |     result = tasks.apply.apply_async(args=[app_id,
nextbackenddocker-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/app/task.py", line 594, in apply_async
nextbackenddocker-1  |     return app.send_task(
nextbackenddocker-1  |            ^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/app/base.py", line 800, in send_task
nextbackenddocker-1  |     self.backend.on_task_call(P, task_id)
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/backends/redis.py", line 373, in on_task_call
nextbackenddocker-1  |     self.result_consumer.consume_from(task_id)
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/backends/redis.py", line 169, in consume_from
nextbackenddocker-1  |     return self.start(task_id)
nextbackenddocker-1  |            ^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/backends/redis.py", line 147, in start
nextbackenddocker-1  |     self._consume_from(initial_task_id)
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/backends/redis.py", line 176, in _consume_from
nextbackenddocker-1  |     with self.reconnect_on_error():
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
nextbackenddocker-1  |     self.gen.throw(typ, value, traceback)
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/backends/redis.py", line 130, in reconnect_on_error
nextbackenddocker-1  |     self._ensure(self._reconnect_pubsub, ())
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/backends/redis.py", line 384, in ensure
nextbackenddocker-1  |     return retry_over_time(
nextbackenddocker-1  |            ^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/kombu/utils/functional.py", line 318, in retry_over_time
nextbackenddocker-1  |     return fun(*args, **kwargs)
nextbackenddocker-1  |            ^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/backends/redis.py", line 106, in _reconnect_pubsub
nextbackenddocker-1  |     metas = self.backend.client.mget(self.subscribed_to)
nextbackenddocker-1  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/commands/core.py", line 2016, in mget
nextbackenddocker-1  |     return self.execute_command("MGET", *args, **options)
nextbackenddocker-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 542, in execute_command
nextbackenddocker-1  |     conn = self.connection or pool.get_connection(command_name, **options)
nextbackenddocker-1  |                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 1109, in get_connection
nextbackenddocker-1  |     connection.connect()
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 288, in connect
nextbackenddocker-1  |     self.on_connect()
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 391, in on_connect
nextbackenddocker-1  |     self.read_response()
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 512, in read_response
nextbackenddocker-1  |     response = self._parser.read_response(disable_decoding=disable_decoding)
nextbackenddocker-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/_parsers/resp2.py", line 15, in read_response
nextbackenddocker-1  |     result = self._read_response(disable_decoding=disable_decoding)
nextbackenddocker-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/_parsers/resp2.py", line 38, in _read_response
nextbackenddocker-1  |     raise error
nextbackenddocker-1  | redis.exceptions.ConnectionError: max number of clients reached
nextbackenddocker-1  | [2024-05-14 21:31:26 +0000] [6] [DEBUG] Closing connection.
nextbackenddocker-1  | [2024-05-14 21:31:27,034] ERROR in app: Exception on /api/experiment/processAnswer [POST]
nextbackenddocker-1  | Traceback (most recent call last):
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/backends/redis.py", line 127, in reconnect_on_error
nextbackenddocker-1  |     yield
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/backends/redis.py", line 177, in _consume_from
nextbackenddocker-1  |     self._pubsub.subscribe(key)
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 929, in subscribe
nextbackenddocker-1  |     ret_val = self.execute_command("SUBSCRIBE", *new_channels.keys())
nextbackenddocker-1  |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 758, in execute_command
nextbackenddocker-1  |     self.connection = self.connection_pool.get_connection(
nextbackenddocker-1  |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 1109, in get_connection
nextbackenddocker-1  |     connection.connect()
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 288, in connect
nextbackenddocker-1  |     self.on_connect()
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 391, in on_connect
nextbackenddocker-1  |     self.read_response()
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 512, in read_response
nextbackenddocker-1  |     response = self._parser.read_response(disable_decoding=disable_decoding)
nextbackenddocker-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/_parsers/resp2.py", line 15, in read_response
nextbackenddocker-1  |     result = self._read_response(disable_decoding=disable_decoding)
nextbackenddocker-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/_parsers/resp2.py", line 38, in _read_response
nextbackenddocker-1  |     raise error
nextbackenddocker-1  | redis.exceptions.ConnectionError: max number of clients reached
nextbackenddocker-1  |
nextbackenddocker-1  | During handling of the above exception, another exception occurred:
nextbackenddocker-1  |
nextbackenddocker-1  | Traceback (most recent call last):
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 1473, in wsgi_app
nextbackenddocker-1  |     response = self.full_dispatch_request()
nextbackenddocker-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 882, in full_dispatch_request
nextbackenddocker-1  |     rv = self.handle_user_exception(e)
nextbackenddocker-1  |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/flask_restful/__init__.py", line 298, in error_router
nextbackenddocker-1  |     return original_handler(e)
nextbackenddocker-1  |            ^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/flask_restful/__init__.py", line 298, in error_router
nextbackenddocker-1  |     return original_handler(e)
nextbackenddocker-1  |            ^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/flask_restful/__init__.py", line 298, in error_router
nextbackenddocker-1  |     return original_handler(e)
nextbackenddocker-1  |            ^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 880, in full_dispatch_request
nextbackenddocker-1  |     rv = self.dispatch_request()
nextbackenddocker-1  |          ^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 865, in dispatch_request
nextbackenddocker-1  |     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
nextbackenddocker-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/flask_restful/__init__.py", line 489, in wrapper
nextbackenddocker-1  |     resp = resource(*args, **kwargs)
nextbackenddocker-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/flask/views.py", line 110, in view
nextbackenddocker-1  |     return current_app.ensure_sync(self.dispatch_request)(**kwargs)  # type: ignore[no-any-return]
nextbackenddocker-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/flask_restful/__init__.py", line 604, in dispatch_request
nextbackenddocker-1  |     resp = meth(*args, **kwargs)
nextbackenddocker-1  |            ^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/next_backend/next/api/resources/process_answer.py", line 66, in post
nextbackenddocker-1  |     response_json,didSucceed,message = broker.applyAsync(app_id,
nextbackenddocker-1  |                                        ^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/next_backend/next/broker/broker.py", line 57, in applyAsync
nextbackenddocker-1  |     result = tasks.apply.apply_async(args=[app_id,
nextbackenddocker-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/app/task.py", line 594, in apply_async
nextbackenddocker-1  |     return app.send_task(
nextbackenddocker-1  |            ^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/app/base.py", line 800, in send_task
nextbackenddocker-1  |     self.backend.on_task_call(P, task_id)
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/backends/redis.py", line 373, in on_task_call
nextbackenddocker-1  |     self.result_consumer.consume_from(task_id)
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/backends/redis.py", line 169, in consume_from
nextbackenddocker-1  |     return self.start(task_id)
nextbackenddocker-1  |            ^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/backends/redis.py", line 147, in start
nextbackenddocker-1  |     self._consume_from(initial_task_id)
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/backends/redis.py", line 176, in _consume_from
nextbackenddocker-1  |     with self.reconnect_on_error():
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
nextbackenddocker-1  |     self.gen.throw(typ, value, traceback)
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/backends/redis.py", line 130, in reconnect_on_error
nextbackenddocker-1  |     self._ensure(self._reconnect_pubsub, ())
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/backends/redis.py", line 384, in ensure
nextbackenddocker-1  |     return retry_over_time(
nextbackenddocker-1  |            ^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/kombu/utils/functional.py", line 318, in retry_over_time
nextbackenddocker-1  |     return fun(*args, **kwargs)
nextbackenddocker-1  |            ^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/backends/redis.py", line 106, in _reconnect_pubsub
nextbackenddocker-1  |     metas = self.backend.client.mget(self.subscribed_to)
nextbackenddocker-1  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/commands/core.py", line 2016, in mget
nextbackenddocker-1  |     return self.execute_command("MGET", *args, **options)
nextbackenddocker-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 542, in execute_command
nextbackenddocker-1  |     conn = self.connection or pool.get_connection(command_name, **options)
nextbackenddocker-1  |                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 1109, in get_connection
nextbackenddocker-1  |     connection.connect()
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 288, in connect
nextbackenddocker-1  |     self.on_connect()
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 391, in on_connect
nextbackenddocker-1  |     self.read_response()
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 512, in read_response
nextbackenddocker-1  |     response = self._parser.read_response(disable_decoding=disable_decoding)
nextbackenddocker-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/_parsers/resp2.py", line 15, in read_response
nextbackenddocker-1  |     result = self._read_response(disable_decoding=disable_decoding)
nextbackenddocker-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/_parsers/resp2.py", line 38, in _read_response
nextbackenddocker-1  |     raise error
nextbackenddocker-1  | redis.exceptions.ConnectionError: max number of clients reached
nextbackenddocker-1  | Exception on /api/experiment/processAnswer [POST]
nextbackenddocker-1  | Traceback (most recent call last):
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/backends/redis.py", line 127, in reconnect_on_error
nextbackenddocker-1  |     yield
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/backends/redis.py", line 177, in _consume_from
nextbackenddocker-1  |     self._pubsub.subscribe(key)
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 929, in subscribe
nextbackenddocker-1  |     ret_val = self.execute_command("SUBSCRIBE", *new_channels.keys())
nextbackenddocker-1  |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 758, in execute_command
nextbackenddocker-1  |     self.connection = self.connection_pool.get_connection(
nextbackenddocker-1  |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 1109, in get_connection
nextbackenddocker-1  |     connection.connect()
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 288, in connect
nextbackenddocker-1  |     self.on_connect()
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 391, in on_connect
nextbackenddocker-1  |     self.read_response()
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 512, in read_response
nextbackenddocker-1  |     response = self._parser.read_response(disable_decoding=disable_decoding)
nextbackenddocker-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/_parsers/resp2.py", line 15, in read_response
nextbackenddocker-1  |     result = self._read_response(disable_decoding=disable_decoding)
nextbackenddocker-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/_parsers/resp2.py", line 38, in _read_response
nextbackenddocker-1  |     raise error
nextbackenddocker-1  | redis.exceptions.ConnectionError: max number of clients reached
nextbackenddocker-1  |
nextbackenddocker-1  | During handling of the above exception, another exception occurred:
nextbackenddocker-1  |
nextbackenddocker-1  | Traceback (most recent call last):
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 1473, in wsgi_app
nextbackenddocker-1  |     response = self.full_dispatch_request()
nextbackenddocker-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 882, in full_dispatch_request
nextbackenddocker-1  |     rv = self.handle_user_exception(e)
nextbackenddocker-1  |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/flask_restful/__init__.py", line 298, in error_router
nextbackenddocker-1  |     return original_handler(e)
nextbackenddocker-1  |            ^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/flask_restful/__init__.py", line 298, in error_router
nextbackenddocker-1  |     return original_handler(e)
nextbackenddocker-1  |            ^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/flask_restful/__init__.py", line 298, in error_router
nextbackenddocker-1  |     return original_handler(e)
nextbackenddocker-1  |            ^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 880, in full_dispatch_request
nextbackenddocker-1  |     rv = self.dispatch_request()
nextbackenddocker-1  |          ^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 865, in dispatch_request
nextbackenddocker-1  |     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
nextbackenddocker-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/flask_restful/__init__.py", line 489, in wrapper
nextbackenddocker-1  |     resp = resource(*args, **kwargs)
nextbackenddocker-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/flask/views.py", line 110, in view
nextbackenddocker-1  |     return current_app.ensure_sync(self.dispatch_request)(**kwargs)  # type: ignore[no-any-return]
nextbackenddocker-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/flask_restful/__init__.py", line 604, in dispatch_request
nextbackenddocker-1  |     resp = meth(*args, **kwargs)
nextbackenddocker-1  |            ^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/next_backend/next/api/resources/process_answer.py", line 66, in post
nextbackenddocker-1  |     response_json,didSucceed,message = broker.applyAsync(app_id,
nextbackenddocker-1  |                                        ^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/next_backend/next/broker/broker.py", line 57, in applyAsync
nextbackenddocker-1  |     result = tasks.apply.apply_async(args=[app_id,
nextbackenddocker-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/app/task.py", line 594, in apply_async
nextbackenddocker-1  |     return app.send_task(
nextbackenddocker-1  |            ^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/app/base.py", line 800, in send_task
nextbackenddocker-1  |     self.backend.on_task_call(P, task_id)
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/backends/redis.py", line 373, in on_task_call
nextbackenddocker-1  |     self.result_consumer.consume_from(task_id)
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/backends/redis.py", line 169, in consume_from
nextbackenddocker-1  |     return self.start(task_id)
nextbackenddocker-1  |            ^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/backends/redis.py", line 147, in start
nextbackenddocker-1  |     self._consume_from(initial_task_id)
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/backends/redis.py", line 176, in _consume_from
nextbackenddocker-1  |     with self.reconnect_on_error():
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
nextbackenddocker-1  |     self.gen.throw(typ, value, traceback)
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/backends/redis.py", line 130, in reconnect_on_error
nextbackenddocker-1  |     self._ensure(self._reconnect_pubsub, ())
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/backends/redis.py", line 384, in ensure
nextbackenddocker-1  |     return retry_over_time(
nextbackenddocker-1  |            ^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/kombu/utils/functional.py", line 318, in retry_over_time
nextbackenddocker-1  |     return fun(*args, **kwargs)
nextbackenddocker-1  |            ^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/celery/backends/redis.py", line 106, in _reconnect_pubsub
nextbackenddocker-1  |     metas = self.backend.client.mget(self.subscribed_to)
nextbackenddocker-1  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/commands/core.py", line 2016, in mget
nextbackenddocker-1  |     return self.execute_command("MGET", *args, **options)
nextbackenddocker-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/client.py", line 542, in execute_command
nextbackenddocker-1  |     conn = self.connection or pool.get_connection(command_name, **options)
nextbackenddocker-1  |                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 1109, in get_connection
nextbackenddocker-1  |     connection.connect()
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 288, in connect
nextbackenddocker-1  |     self.on_connect()
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 391, in on_connect
nextbackenddocker-1  |     self.read_response()
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/connection.py", line 512, in read_response
nextbackenddocker-1  |     response = self._parser.read_response(disable_decoding=disable_decoding)
nextbackenddocker-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/_parsers/resp2.py", line 15, in read_response
nextbackenddocker-1  |     result = self._read_response(disable_decoding=disable_decoding)
nextbackenddocker-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nextbackenddocker-1  |   File "/usr/local/lib/python3.11/site-packages/redis/_parsers/resp2.py", line 38, in _read_response
nextbackenddocker-1  |     raise error
nextbackenddocker-1  | redis.exceptions.ConnectionError: max number of clients
iamzoltan commented 1 month ago

This was solved by closing connections that were stale for over 5 mins

iamzoltan commented 1 month ago

some additional tests need to be done

iamzoltan commented 1 month ago

merged solution in #1