search

certtools / intelmq

IntelMQ is a solution for IT security teams for collecting and processing security feeds using a message queuing protocol.
https://docs.intelmq.org/latest/
GNU Affero General Public License v3.0
963 stars 295 forks source link

HTTP collector: Log response size #1764

Open ghost opened 3 years ago

ghost commented 3 years ago

Proposal for the HTTP collector log. Instead of

INFO - Downloading report from 'https://urlhaus.abuse.ch/feeds/tld/at/'.

log the size in parenthesis:

INFO - Downloading report from 'https://urlhaus.abuse.ch/feeds/tld/at/' (1.4 KB).

Then it's easy to see if a response was empty (0B, see also https://lists.cert.at/pipermail/intelmq-users/2021-February/000202.html) or very big.

ghost commented 3 years ago

len(response) will give the size in bytes (after unpacking). We can add a function to intelmq.lib.utils for size-conversion to get a human-readable size specifier.