Open alexxroche opened 3 years ago
Hi @alexxroche
The Error 429 and message from Firefox "You are being rate limited" indicates that your IP address is on a sort of cool-down for trying to download the same files from database.clamav.net too frequently.
Background: A growing number of people had been using wget
and similar tools to frequently download the whole database set, or are deploying containers or VMs that don't contain a baseline set of databases and thus when started download the whole database sets. This has become increasingly costly for the ClamAV project. So the ClamAV project is making a best effort to require users to use FreshClam to update existing databases and to not download the entire database set unless absolutely necessary. To that end, programs like wget
have been blocked entirely (error code 403
) and the whole database files like daily.cvd
, main.cvd
, and bytecode.cvd
are being rate limited (error code 429
).
So it might not be your fault exactly. But you can take steps to prevent being affected. If you're using a public cloud or are behind a large NAT, it's likely that others are frequently downloading the whole database files and triggering the rate limiting.
The update patch files (the *.cdiff
files) are not rate limited, so good netizens can easily update their existing ClamAV installs by running FreshClam.
If you work for a large organization has a lot of ClamAV installations, it might be a good idea to set up a private mirror for your organization. This will also eliminate the rate limiting issue, and will help save the ClamAV project some $$. See https://www.clamav.net/documents/private-local-mirrors and https://pypi.org/project/cvdupdate/ for more details.
If you're using a public cloud or are behind a large NAT,
I'm on a dedicated single static IPv4 and an IPv6 /64. I use NAT for the IPv4 stack of my LAN, but I control every node within that LAN and ensure that no more than two machines are trying to download daily.cvd or main.cvd (and any other device can then be synchronised using rsync or by reading the cvd files from the LAN's NAS.)
it might be a good idea to set up a private mirror
I agree. (Which is why I've been doing that since 1999 with anti-virus databases and why I propose at each company that I work that they should be good netizens and offer to mirror important projects such as ClamAV. I believe that every ISP should have their own mirror of ClamAV specifically so that we don't have a single point of failure such as a CDN that could be silently dropping requests without anyone noticing. How can we as the ClamAV community know how many ClamAV instillation are now out of date because they are unable to update?)
@alexxroche are you still having issues downloading signature updates?
Not currently.
I found a copy of main.cld that wasn't on the CDN and downloaded and installed it. I just tested freshclam and it updated from 0.102.4 to 0.103.2 without error. (I still think that freshclam should fall back to using the old network of mirrors if the CDN times out N
times.
@alexxroche are you still having issues downloading signature updates?
I'm still getting database update error via freshclam
"Well there's your problem..." #lets_not_put_all_of_our_eggs_in_one_basket
Out of curiosity I opened https://database.clamav.net/daily.cvd in firefox and after cloudfair faffed about for most of a minute I was presented with:
"not exactly poggers".