search

certtools / intelmq

IntelMQ is a solution for IT security teams for collecting and processing security feeds using a message queuing protocol.
https://docs.intelmq.org/latest/
GNU Affero General Public License v3.0
976 stars 297 forks source link

Redis Pipeline error #2446

Open gethvi opened 10 months ago

gethvi commented 10 months ago

On some occasions (after restarting Redis) connection to Redis is refused and this exception is raised:

File "/opt/venv/lib/python3.9/site-packages/intelmq/lib/pipeline.py", line 311, in count_queued_messages
queue_dict[queue] = self.pipe.llen(queue)
File "/opt/venv/lib/python3.9/site-packages/redis/commands/core.py", line 2668, in llen
return self.execute_command("LLEN", name)
File "/opt/venv/lib/python3.9/site-packages/redis/client.py", line 1258, in execute_command
return conn.retry.call_with_retry(
File "/opt/venv/lib/python3.9/site-packages/redis/retry.py", line 49, in call_with_retry
fail(error)
File "/opt/venv/lib/python3.9/site-packages/redis/client.py", line 1262, in <lambda>
lambda error: self._disconnect_raise(conn, error),
File "/opt/venv/lib/python3.9/site-packages/redis/client.py", line 1248, in _disconnect_raise
raise error
File "/opt/venv/lib/python3.9/site-packages/redis/retry.py", line 46, in call_with_retry
return do()
File "/opt/venv/lib/python3.9/site-packages/redis/client.py", line 1259, in <lambda>
lambda: self._send_command_parse_response(
File "/opt/venv/lib/python3.9/site-packages/redis/client.py", line 1235, in _send_command_parse_response
return self.parse_response(conn, command_name, **options)
File "/opt/venv/lib/python3.9/site-packages/redis/client.py", line 1275, in parse_response
response = connection.read_response()
File "/opt/venv/lib/python3.9/site-packages/redis/connection.py", line 865, in read_response
response = self._parser.read_response(disable_decoding=disable_decoding)
File "/opt/venv/lib/python3.9/site-packages/redis/connection.py", line 346, in read_response
result = self._read_response(disable_decoding=disable_decoding)
File "/opt/venv/lib/python3.9/site-packages/redis/connection.py", line 369, in _read_response
raise error
redis.exceptions.BusyLoadingError: Redis is loading the dataset in memory

This just means we should just wait for a while for Redis to load the dataset in memory, not necessarily throw this on the user.

This particular case is from intelmq manager calling the command: sudo -u intelmq intelmqctl --type json list queues-and-status.

kamil-certat commented 10 months ago

Hmm, I believe that while bots should wait, the intelmqctl is a one-time tool, so it in my opinion should end with a user-friendly command indicating starting Redis. On the other hand, the API wrapper should handle it gracefully, and just retry. What do you think?

aaronkaplan commented 9 months ago

Well, for some intelmqctl is a CLI "API" ;-)

gethvi commented 9 months ago

Hmm, I believe that while bots should wait, the intelmqctl is a one-time tool, so it in my opinion should end with a user-friendly command indicating starting Redis. On the other hand, the API wrapper should handle it gracefully, and just retry. What do you think?

I agree that one-time tool should probably just notify the user that Redis is not (yet) ready and that the user should wait and try again.

The API wrapper should probably wait for a little while, but eventually do the same really. Redis can be unavailable for a quite a while if the dataset is big (or if it crashes again, this happend to us when redis - in docker container - ran out of memory).