search

Cisco-Talos / clamav

ClamAV - Documentation is here: https://docs.clamav.net
https://www.clamav.net/
GNU General Public License v2.0
4.03k stars 675 forks source link

Blocked by CDN when trying to download latest database #1123

Open cazbhoy opened 7 months ago

cazbhoy commented 7 months ago

Describe the bug

When I try to run freshclam it fails and tells me im blocked by the CDN. I have tried letting the timeout expire and running it again but the issue persists.

C:\clamav>freshclam.exe

ClamAV update process started at Fri Dec 22 09:50:59 2023 daily database available for update (local version: 27129, remote version: 27131) Current database is 2 versions behind. Downloading database patch # 27130... Time: 0.2s, ETA: 0.0s [========================>] 3.37KiB/3.37KiB WARNING: downloadPatch: Can't download daily-27130.cdiff from https://database.clamav.net/daily-27130.cdiff WARNING: Incremental update failed, trying to download daily.cvd Time: 0.2s, ETA: 0.0s [========================>] 3.36KiB/3.36KiB WARNING: Can't download daily.cvd from https://database.clamav.net/daily.cvd WARNING: FreshClam received error code 403 from the ClamAV Content Delivery Network (CDN). This could mean several things:

  1. You are running an out-of-date version of ClamAV / FreshClam. Ensure you are the most updated version by visiting https://www.clamav.net/downloads
  2. Your network is explicitly denied by the FreshClam CDN. In order to rectify this please check that you are: a. Running an up-to-date version of FreshClam b. Running FreshClam no more than once an hour c. If you have checked (a) and (b), please open a ticket at https://github.com/Cisco-Talos/clamav/issues and we will investigate why your network is blocked. WARNING: You are on cool-down until after: 2023-12-23 09:51:05 ERROR: Database update process failed: Forbidden; Blocked by CDN ERROR: Update failed.

How to reproduce the problem

This now happens whenever I run freshclam. I am connecting from the UK via 80.193.210.136 I had only been running this once a day and made surre to let the timeout expire before retrying.

ragusaa commented 7 months ago

Hi, What version of freshclam are you running?

cazbhoy commented 6 months ago

running version ClamAV 0.103.11/27129/

ragusaa commented 6 months ago

If there are other machines on your network downloading signatures, that could cause this issue. I assume you are still having the problem?

cazbhoy commented 6 months ago

Yeah still having the issue. Ive checked but cant see any other systems using this, only have it setup on one server. It had been working previously.

I have stopped the update from running automatically to see if that helps. Ill try updating the database manually and test again. Below is the most recent output:

ClamAV update process started at Tue Jan 9 11:51:14 2024 WARNING: Cool-down expired, ok to try again. daily database available for update (local version: 27129, remote version: 27149) Current database is 20 versions behind. Downloading database patch # 27130... Time: 0.3s, ETA: 0.0s [========================>] 3.37KiB/3.37KiB WARNING: downloadPatch: Can't download daily-27130.cdiff from https://database.clamav.net/daily-27130.cdiff WARNING: Incremental update failed, trying to download daily.cvd Time: 0.1s, ETA: 0.0s [========================>] 3.36KiB/3.36KiB WARNING: Can't download daily.cvd from https://database.clamav.net/daily.cvd WARNING: FreshClam received error code 403 from the ClamAV Content Delivery Network (CDN). This could mean several things:

  1. You are running an out-of-date version of ClamAV / FreshClam. Ensure you are the most updated version by visiting https://www.clamav.net/downloads
  2. Your network is explicitly denied by the FreshClam CDN. In order to rectify this please check that you are: a. Running an up-to-date version of FreshClam b. Running FreshClam no more than once an hour c. If you have checked (a) and (b), please open a ticket at https://github.com/Cisco-Talos/clamav/issues and we will investigate why your network is blocked. WARNING: You are on cool-down until after: 2024-01-10 11:51:20 ERROR: Database update process failed: Forbidden; Blocked by CDN ERROR: Update failed.
ragusaa commented 6 months ago

Thanks for the reply, could you give me the output of 'freshclam --verbose'. That'll give us the CF-Ray ID.

cazbhoy commented 6 months ago

Hi

Verbose output below:

C:\clamav>freshclam.exe --verbose
Connecting via 192.168.41.8
Current working dir is C:\clamav\db\
Loaded freshclam.dat:
  version:    1
  uuid:       fadce9ce-1e7c-47ee-baef-907dddb787cb
  retry-after: 2024-01-12 11:53:15
ClamAV update process started at Mon Jan 15 10:31:46 2024
Current working dir is C:\clamav\db\
DNS Resolver (dnsapi): Querying current.cvd.clamav.net
TTL: 1716
fc_dns_query_update_info: Software version from DNS: 0.103.11
WARNING: Cool-down expired, ok to try again.
Saved freshclam.dat
Current working dir is C:\clamav\db\
check_for_new_database_version: Local copy of daily found: daily.cld.
query_remote_database_version: daily.cvd version from DNS: 27155
daily database available for update (local version: 27129, remote version: 27155)
Current database is 26 versions behind.
Downloading database patch # 27130...
Retrieving https://database.clamav.net/daily-27130.cdiff
Using proxy: 192.168.41.8:80
downloadFile: Download source:      https://database.clamav.net/daily-27130.cdiff
downloadFile: Download destination: .\clamav-f12e8f231bc9fb739d68c3ba2a96bcab.tmp
*   Trying 192.168.41.8:80...
* Connected to 192.168.41.8 (192.168.41.8) port 80
* CONNECT tunnel: HTTP/1.1 negotiated
* allocate connect buffer
* Proxy auth using Basic with user 'aib_clamav'
* Establish HTTP proxy tunnel to database.clamav.net:443
> CONNECT database.clamav.net:443 HTTP/1.1
Host: database.clamav.net:443
Proxy-Authorization: Basic YWliX2NsYW1hdjpEVE9WbGVXNHh6THpoUVc=
User-Agent: ClamAV/0.103.11 (OS: win32, ARCH: x86_64, CPU: x86_64, UUID: fadce9ce-1e7c-47ee-baef-907dddb787cb)
Proxy-Connection: Keep-Alive

< HTTP/1.1 200 Connection established
< Proxy-Agent: Fortinet-Proxy/1.0
<
* CONNECT phase completed
* CONNECT tunnel established, response 200
* ALPN: curl offers http/1.1

Certificate loaded from Windows certificate store: Microsoft Root Certificate Authority
Certificate loaded from Windows certificate store: ISRG Root X1
Certificate loaded from Windows certificate store: Thawte Timestamping CA
Certificate loaded from Windows certificate store: Microsoft Root Authority
Certificate loaded from Windows certificate store: Symantec Enterprise Mobile Root for Microsoft
Certificate loaded from Windows certificate store: Microsoft Root Certificate Authority 2011
Certificate loaded from Windows certificate store: Microsoft Authenticode(tm) Root
Certificate loaded from Windows certificate store: R3
Certificate loaded from Windows certificate store: Microsoft Root Certificate Authority 2010
Certificate loaded from Windows certificate store: Microsoft Timestamp Root
Certificate loaded from Windows certificate store: VeriSign Time Stamping CA
Certificate loaded from Windows certificate store: Microsoft Identity Verification Root Certificate Authority 2020
Certificate loaded from Windows certificate store: Sectigo (UTN Object)
Certificate loaded from Windows certificate store: DigiCert Global Root G2
Certificate loaded from Windows certificate store: DigiCert Trusted Root G4
Certificate loaded from Windows certificate store: DST Root CA X3
Certificate loaded from Windows certificate store: GlobalSign Root CA - R3
Certificate loaded from Windows certificate store: DigiCert Baltimore Root
Certificate loaded from Windows certificate store: Sectigo (AAA)
Certificate loaded from Windows certificate store: GlobalSign Root CA - R1
Certificate loaded from Windows certificate store: Starfield Class 2 Certification Authority
Certificate loaded from Windows certificate store: DigiCert
Certificate loaded from Windows certificate store: Google Trust Services - GlobalSign Root CA-R2
Certificate loaded from Windows certificate store: VeriSign Class 3 Public Primary CA
Certificate loaded from Windows certificate store: DigiCert
Certificate loaded from Windows certificate store: VeriSign
Certificate loaded from Windows certificate store: Sectigo
Certificate loaded from Windows certificate store: thawte
Certificate loaded from Windows certificate store: DigiCert
Certificate loaded from Windows certificate store: Sectigo (AddTrust)
Certificate loaded from Windows certificate store: Microsoft Identity Verification Root Certificate Authority 2020
Certificate loaded from Windows certificate store: SSD ROOT CA
Certificate loaded from Windows certificate store: SSS Root CA

* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-SHA
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
*  start date: May 15 00:00:00 2023 GMT
*  expire date: May 14 23:59:59 2024 GMT
*  subjectAltName: host "database.clamav.net" matched cert's "database.clamav.net"
*  issuer: C=US; ST=California; L=Sunnyvale; O=Fortinet; OU=Certificate Authority; CN=FortiGate CA; emailAddress=support@fortinet.com
*  SSL certificate verify ok.
* using HTTP/1.x
> GET /daily-27130.cdiff HTTP/1.1
Host: database.clamav.net
User-Agent: ClamAV/0.103.11 (OS: win32, ARCH: x86_64, CPU: x86_64, UUID: fadce9ce-1e7c-47ee-baef-907dddb787cb)
Accept: */*
Connection: close

< HTTP/1.1 403 Forbidden
< Connection: close
< Content-Length: 3452
<
Time:    0.4s, ETA:    0.0s [========================>]    3.37KiB/3.37KiB
* Closing connection
Saved freshclam.dat
WARNING: downloadPatch: Can't download daily-27130.cdiff from https://database.clamav.net/daily-27130.cdiff
WARNING: Incremental update failed, trying to download daily.cvd
Retrieving https://database.clamav.net/daily.cvd
Using proxy: 192.168.41.8:80
downloadFile: Download source:      https://database.clamav.net/daily.cvd
downloadFile: Download destination: C:\clamav\db\tmp.37dc2e346b\clamav-d41182b9788262e948cb17752c7b1545.tmp
*   Trying 192.168.41.8:80...
* Connected to 192.168.41.8 (192.168.41.8) port 80
* CONNECT tunnel: HTTP/1.1 negotiated
* allocate connect buffer
* Proxy auth using Basic with user 'aib_clamav'
* Establish HTTP proxy tunnel to database.clamav.net:443
> CONNECT database.clamav.net:443 HTTP/1.1
Host: database.clamav.net:443
Proxy-Authorization: Basic YWliX2NsYW1hdjpEVE9WbGVXNHh6THpoUVc=
User-Agent: ClamAV/0.103.11 (OS: win32, ARCH: x86_64, CPU: x86_64, UUID: fadce9ce-1e7c-47ee-baef-907dddb787cb)
Proxy-Connection: Keep-Alive

< HTTP/1.1 200 Connection established
< Proxy-Agent: Fortinet-Proxy/1.0
<
* CONNECT phase completed
* CONNECT tunnel established, response 200
* ALPN: curl offers http/1.1
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-SHA
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
*  start date: May 15 00:00:00 2023 GMT
*  expire date: May 14 23:59:59 2024 GMT
*  subjectAltName: host "database.clamav.net" matched cert's "database.clamav.net"
*  issuer: C=US; ST=California; L=Sunnyvale; O=Fortinet; OU=Certificate Authority; CN=FortiGate CA; emailAddress=support@fortinet.com
*  SSL certificate verify ok.
* using HTTP/1.x
> GET /daily.cvd HTTP/1.1
Host: database.clamav.net
User-Agent: ClamAV/0.103.11 (OS: win32, ARCH: x86_64, CPU: x86_64, UUID: fadce9ce-1e7c-47ee-baef-907dddb787cb)
Accept: */*
If-Modified-Since: Wed, 20 Dec 2023 09:38:37 GMT
Connection: close

< HTTP/1.1 403 Forbidden
< Connection: close
< Content-Length: 3444
<
Time:    0.2s, ETA:    0.0s [========================>]    3.36KiB/3.36KiB
* Closing connection
Saved freshclam.dat
WARNING: Can't download daily.cvd from https://database.clamav.net/daily.cvd
WARNING: FreshClam received error code 403 from the ClamAV Content Delivery Network (CDN).
This could mean several things:
 1. You are running an out-of-date version of ClamAV / FreshClam.
    Ensure you are the most updated version by visiting https://www.clamav.net/downloads
 2. Your network is explicitly denied by the FreshClam CDN.
    In order to rectify this please check that you are:
   a. Running an up-to-date version of FreshClam
   b. Running FreshClam no more than once an hour
   c. If you have checked (a) and (b), please open a ticket at
      https://github.com/Cisco-Talos/clamav/issues
      and we will investigate why your network is blocked.
WARNING: You are on cool-down until after: 2024-01-16 10:31:50
ERROR: Database update process failed: Forbidden; Blocked by CDN
ERROR: Update failed.

C:\clamav>
ragusaa commented 6 months ago

Thank you.

According to the logs, your IP is being blocked because the version is too old (showing 0.100.3). Looking into how this is possible, since your software version is 0.103.11.

ragusaa commented 6 months ago

I see you are using a proxy. Is there a chance it's not forwarding headers, causing our system to not see a version number? I know you said that this had been working previously, is there a chance something else changed?

pbsolo commented 6 months ago

Identical problem to this one. Had to replace a dying hard drive and reinstalled Ubuntu, followed up by ClamAV from the website. Refused to allow freshclam to complete (at 150mbps down my end?) Waited until Monday having given up and to give the CDN a chance to cool off because I only tried once before from a dynamic IP that has change to 188.30.83.104 over the weekend. Now I get this,.. X----X:~$ sudo service clamav-freshclam stop [sudo] password for one: X----X:~$ sudo freshclam Mon Jan 22 09:24:36 2024 -> ClamAV update process started at Mon Jan 22 09:24:36 2024 Mon Jan 22 09:24:36 2024 -> ^Your ClamAV installation is OUTDATED! Mon Jan 22 09:24:36 2024 -> ^Local version: 0.103.8 Recommended version: 0.103.11 Mon Jan 22 09:24:36 2024 -> DON'T PANIC! Read https://docs.clamav.net/manual/Installing.html Mon Jan 22 09:24:36 2024 -> daily database available for download (remote version: 27161) Mon Jan 22 09:24:52 2024 -> ^Download failed (18) Mon Jan 22 09:24:52 2024 -> ^ Message: Transferred a partial file Mon Jan 22 09:24:52 2024 -> ^Can't download daily.cvd from https://database.clamav.net/daily.cvd Mon Jan 22 09:24:52 2024 -> Trying again in 5 secs... Mon Jan 22 09:24:57 2024 -> daily database available for download (remote version: 27161) Mon Jan 22 09:25:08 2024 -> ^Download failed (18) Mon Jan 22 09:25:08 2024 -> ^ Message: Transferred a partial file Mon Jan 22 09:25:08 2024 -> ^Can't download daily.cvd from https://database.clamav.net/daily.cvd Mon Jan 22 09:25:08 2024 -> Trying again in 5 secs... Mon Jan 22 09:25:13 2024 -> daily database available for download (remote version: 27161) Time: 0.2s, ETA: 0.0s [========================>] 16B/16B Mon Jan 22 09:25:13 2024 -> ^Can't download daily.cvd from https://database.clamav.net/daily.cvd Mon Jan 22 09:25:13 2024 -> ^FreshClam received error code 429 from the ClamAV Content Delivery Network (CDN). Mon Jan 22 09:25:13 2024 -> This means that you have been rate limited by the CDN. Mon Jan 22 09:25:13 2024 -> 1. Run FreshClam no more than once an hour to check for updates. Mon Jan 22 09:25:13 2024 -> FreshClam should check DNS first to see if an update is needed. Mon Jan 22 09:25:13 2024 -> 2. If you have more than 10 hosts on your network attempting to download, Mon Jan 22 09:25:13 2024 -> it is recommended that you set up a private mirror on your network using Mon Jan 22 09:25:13 2024 -> cvdupdate (https://pypi.org/project/cvdupdate/) to save bandwidth on the Mon Jan 22 09:25:13 2024 -> CDN and your own network. Mon Jan 22 09:25:13 2024 -> 3. Please do not open a ticket asking for an exemption from the rate limit, Mon Jan 22 09:25:13 2024 -> it will not be granted. Mon Jan 22 09:25:13 2024 -> ^You are on cool-down until after: 2024-01-22 13:25:13 Mon Jan 22 09:25:13 2024 -> main database available for download (remote version: 62) Time: 0.1s, ETA: 0.0s [========================>] 16B/16B Mon Jan 22 09:25:13 2024 -> ^Can't download main.cvd from https://database.clamav.net/main.cvd Mon Jan 22 09:25:13 2024 -> ^FreshClam received error code 429 from the ClamAV Content Delivery Network (CDN). Mon Jan 22 09:25:13 2024 -> This means that you have been rate limited by the CDN. Mon Jan 22 09:25:13 2024 -> 1. Run FreshClam no more than once an hour to check for updates. Mon Jan 22 09:25:13 2024 -> FreshClam should check DNS first to see if an update is needed. Mon Jan 22 09:25:13 2024 -> 2. If you have more than 10 hosts on your network attempting to download, Mon Jan 22 09:25:13 2024 -> it is recommended that you set up a private mirror on your network using Mon Jan 22 09:25:13 2024 -> cvdupdate (https://pypi.org/project/cvdupdate/) to save bandwidth on the Mon Jan 22 09:25:13 2024 -> CDN and your own network. Mon Jan 22 09:25:13 2024 -> 3. Please do not open a ticket asking for an exemption from the rate limit, Mon Jan 22 09:25:13 2024 -> it will not be granted. Mon Jan 22 09:25:13 2024 -> ^You are on cool-down until after: 2024-01-22 13:25:13 Mon Jan 22 09:25:13 2024 -> bytecode.cvd database is up-to-date (version: 334, sigs: 91, f-level: 90, builder: anvilleg) X----X:~$

Seriously?? The CDN is hanging by the look of it and waiting 5 secs, then again 5 secs, then tells me I've got an error because it's failed three times? Before you ask, only one computer running here at home and not a commercial network.

cazbhoy commented 5 months ago

Not seeing anything blocked by proxy, will try a fresh install do see if that changes anything regarding version numbers

micahsnyder commented 5 months ago

@cazbhoy the download may have been giving up because of the ReceiveTimeout setting in freshclam.conf.

This is my guess because it has a partial download and then retries.

Please check that setting. If it exists, please remove it or else increase it significantly. Then try again after the cool-down expires.

If it doesn't exist, then I'm not sure why you're seeing a partial download before it fails and retries.

cazbhoy commented 5 months ago

No timeout set in freshclam.conf so I added this in and set it quite high but still getting the same issue. Appears to start the download then stops and says I am blocked again.

micahsnyder commented 5 months ago

@cazbhoy That's very strange. I have no idea why it would start the download and then stop if not for the timeout.

By the way, no timeout set means the timeout is disabled. No need to add one.

micahsnyder commented 3 weeks ago

@cazbhoy Did you ever find a solution?