search

Cisco-Talos / cvdupdate

ClamAV Private Database Mirror Updater Tool
Apache License 2.0
93 stars 35 forks source link

cvd update failing frequently #54

Closed Muzeebshaik closed 4 months ago

Muzeebshaik commented 9 months ago

`#16 1.949 2023-11-15 12:39:21 cvdupdate-1.1.1 WARNING Failed to download main.cvd.

16 1.949 2023-11-15 12:39:21 cvdupdate-1.1.1 WARNING Download request rejected because we've downloaded the same file too frequently.

16 1.949 2023-11-15 12:39:21 cvdupdate-1.1.1 WARNING We won't try main.cvd again for 13:20:14 hours.

16 1.952 2023-11-15 12:39:21 cvdupdate-1.1.1 DEBUG Checking daily.cvd for update from https://database.clamav.net/daily.cvd`

I have been using the CVD update for the past six months, but haven't come across these kind of failures, these is happening very frequently. How to avoid this from happening and is there any other way to get latest definition other than cvdupdate or freshclam

micahsnyder commented 9 months ago

What environment are you running this in? Is it a public cloud or other IP shared with a lot of hosts? It may be that others behind the same public IP are downloading the same files now.

The rate limiting set up for database.clamav.net through Cloudflare is supposed to be ratelimiting per client, but is unfortunately ratelimiting per IP instead. I am waiting for support from our web team as they work with Cloudflare to solve the problem. I will message againf or an update.

Muzeebshaik commented 9 months ago

Good day, @micahsnyder Thank you for responding so quickly. As I previously indicated, I have been using this configuration for a long time, but recently I am seeing this rate limiting issue. I am not operating in a public cloud, and I am sure there aren't likely to be many hosts attempting to download the same files.

micahsnyder commented 9 months ago

@Muzeebshaik I can't be certain why you're running into this issue more frequently now, then.

I checked in with our web team about the support request with Cloudflare and it seems the last meeting with Cloudflare fell through. So, at my request, we increased the rate limit to reduce frustration with database updates until it can be fixed the correct way.

Please let me know if the update stops failing, or continues to fail.

Muzeebshaik commented 9 months ago

@micahsnyder Thanks, I see that updates are not failing as they before did. Can I know is there any change to the rate limit recently that made updates stop?

micahsnyder commented 9 months ago

I don't think we changed the rate limit recently causing the failures, but we increased the limit two weeks ago a bandaid for the issue until we can fix it the correct way. I do not know why it started failing in our specific case when it had previously worked okay.

micahsnyder commented 4 months ago

I believe we've finally fixed the rate limit issue for freshclam and cvdupdate.