search

Cisco-Talos / clamav

ClamAV - Documentation is here: https://docs.clamav.net
https://www.clamav.net/
GNU General Public License v2.0
4.21k stars 686 forks source link

cli_tgzload: Invalid checksum for file daily.hsb - broken mirror? #455

Closed kucharskim closed 5 months ago

kucharskim commented 2 years ago

Below problem happened once in a while, but now happens more and more often. Nothing changed on my machine.

# freshclam                                                                 
ClamAV update process started at Sat Feb  5 19:02:31 2022
daily database available for update (local version: 26440, remote version: 26444)
Current database is 4 versions behind.
Downloading database patch # 26441...
Time:    0.2s, ETA:    0.0s [========================>]   11.88KiB/11.88KiB
Downloading database patch # 26442...
Time:    0.1s, ETA:    0.0s [========================>]    6.67KiB/6.67KiB
Downloading database patch # 26443...
Time:    0.1s, ETA:    0.0s [========================>]   10.00KiB/10.00KiB
Downloading database patch # 26444...
Time:    0.1s, ETA:    0.0s [========================>]    7.23KiB/7.23KiB
Testing database: '/var/db/clamav/tmp.058621f182/clamav-0a917e3d95c85d7f7444431f5c49aedd.tmp-daily.cld' ...
WARNING: Stderr output from database load : WARNING: [LibClamAV] cli_tgzload: Invalid checksum for file daily.hsb [...] ERROR: Failed to load new database: Malformed database
WARNING: Database load exited with "Test failed"
ERROR: Database test FAILED.
ERROR: Unexpected error when attempting to update daily: Test failed
ERROR: Database update process failed: Test failed
ERROR: Update failed.
# clamconf -n                                                                                                                                                                                                                            
Checking configuration files in /etc

Config file: clamd.conf
-----------------------
ERROR: Please edit the example config file /etc/clamd.conf

Config file: freshclam.conf
---------------------------
LogSyslog = "yes"
DatabaseMirror = "database.clamav.net"
MaxAttempts = "5"
ConnectTimeout = "10"
ReceiveTimeout = "900"

Config file: clamav-milter.conf
-------------------------------
ERROR: Please edit the example config file /etc/clamav-milter.conf

Software settings
-----------------
Version: 0.104.2
Optional features supported: MEMPOOL AUTOIT_EA06 BZIP2 LIBXML2 PCRE2 ICONV JSON RAR 

Database information
--------------------
Database directory: /var/db/clamav
WARNING: freshclam.conf and clamd.conf point to different database directories
main.cvd: version 62, sigs: 6647427, built on Thu Sep 16 12:32:42 2021
bytecode.cvd: version 333, sigs: 92, built on Mon Mar  8 15:21:51 2021
daily.cld: version 26440, sigs: 1973273, built on Tue Feb  1 09:29:16 2022
Total number of signatures: 8620792

Platform information
--------------------
uname: OpenBSD 7.0 GENERIC.MP#296 amd64
OS: OpenBSD, ARCH: amd64, CPU: amd64
zlib version: 1.2.11 (1.2.11), compile flags: a9
platform id: 0x03238e8e0800000000040201

Build information
-----------------
Clang: OpenBSD Clang 13.0.0 (4.2.1)
sizeof(void*) = 8
Engine flevel: 142, dconf: 142
# /etc/freshclam.conf                                                                                                                                                                                                                
LogSyslog yes
ScriptedUpdates yes
DatabaseMirror database.clamav.net
DNSDatabaseInfo current.cvd.clamav.net
MaxAttempts 5
ConnectTimeout 10
ReceiveTimeout 900
Bytecode yes
kucharskim commented 2 years ago 
# freshclam -vvv
Current working dir is /var/db/clamav/
Loaded freshclam.dat:
  version:    1
  uuid:       510cb77e-33ce-426c-bb17-8103a353630b
ClamAV update process started at Sat Feb  5 19:12:20 2022
Current working dir is /var/db/clamav/
Querying current.cvd.clamav.net
TTL: 693
fc_dns_query_update_info: Software version from DNS: 0.103.5
Current working dir is /var/db/clamav/
check_for_new_database_version: Local copy of daily found: daily.cld.
query_remote_database_version: daily.cvd version from DNS: 26444
daily database available for update (local version: 26440, remote version: 26444)
Retrieving https://database.clamav.net/daily-26441.cdiff
downloadFile: Download source:      https://database.clamav.net/daily-26441.cdiff
downloadFile: Download destination: ./clamav-86e4b4f50011abda24799cc1d17b6eac.tmp
*   Trying 2606:4700::6810:db54:443...
* Connected to database.clamav.net (2606:4700::6810:db54) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
*  start date: Jul 15 00:00:00 2021 GMT
*  expire date: Jul 14 23:59:59 2022 GMT
*  subjectAltName: host "database.clamav.net" matched cert's "database.clamav.net"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x851a7e89000)
> GET /daily-26441.cdiff HTTP/2
Host: database.clamav.net
user-agent: ClamAV/0.104.2 (OS: OpenBSD, ARCH: amd64, CPU: amd64, UUID: 510cb77e-33ce-426c-bb17-8103a353630b)
accept: */*
connection: close

* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 200 
< date: Sat, 05 Feb 2022 19:12:20 GMT
< content-type: application/octet-stream
< content-length: 12162
< last-modified: Wed, 02 Feb 2022 09:43:00 GMT
< etag: "61fa5224-2f82"
< expires: Thu, 02 Feb 2023 19:21:05 GMT
< cache-control: public, max-age=31277325
< cf-cache-status: HIT
< age: 34591
< accept-ranges: bytes
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< strict-transport-security: max-age=15552000
< x-content-type-options: nosniff
< server: cloudflare
< cf-ray: 6d8e73227818f2e4-WAW
< 
* Connection #0 to host database.clamav.net left intact
cdiff_apply: Parsed 182 lines and executed 182 commands
Retrieving https://database.clamav.net/daily-26442.cdiff
downloadFile: Download source:      https://database.clamav.net/daily-26442.cdiff
downloadFile: Download destination: ./clamav-8c7389cbba65f62bed7e65c896aa0163.tmp
*   Trying 2606:4700::6810:db54:443...
* Connected to database.clamav.net (2606:4700::6810:db54) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
*  start date: Jul 15 00:00:00 2021 GMT
*  expire date: Jul 14 23:59:59 2022 GMT
*  subjectAltName: host "database.clamav.net" matched cert's "database.clamav.net"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x8517d21d000)
> GET /daily-26442.cdiff HTTP/2
Host: database.clamav.net
user-agent: ClamAV/0.104.2 (OS: OpenBSD, ARCH: amd64, CPU: amd64, UUID: 510cb77e-33ce-426c-bb17-8103a353630b)
accept: */*
connection: close

* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 200 
< date: Sat, 05 Feb 2022 19:12:21 GMT
< content-type: application/octet-stream
< content-length: 6830
< last-modified: Thu, 03 Feb 2022 09:22:00 GMT
< etag: "61fb9eb8-1aae"
< expires: Fri, 03 Feb 2023 19:00:05 GMT
< cache-control: public, max-age=31362464
< cf-cache-status: HIT
< age: 34676
< accept-ranges: bytes
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< strict-transport-security: max-age=15552000
< x-content-type-options: nosniff
< server: cloudflare
< cf-ray: 6d8e7324ac60cc77-WAW
< 
* Connection #0 to host database.clamav.net left intact
cdiff_apply: Parsed 510 lines and executed 510 commands
Retrieving https://database.clamav.net/daily-26443.cdiff
downloadFile: Download source:      https://database.clamav.net/daily-26443.cdiff
downloadFile: Download destination: ./clamav-6fd235ef8b02e0df2de6c24c99eb7bd1.tmp
*   Trying 2606:4700::6810:db54:443...
* Connected to database.clamav.net (2606:4700::6810:db54) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
*  start date: Jul 15 00:00:00 2021 GMT
*  expire date: Jul 14 23:59:59 2022 GMT
*  subjectAltName: host "database.clamav.net" matched cert's "database.clamav.net"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x8517d21d000)
> GET /daily-26443.cdiff HTTP/2
Host: database.clamav.net
user-agent: ClamAV/0.104.2 (OS: OpenBSD, ARCH: amd64, CPU: amd64, UUID: 510cb77e-33ce-426c-bb17-8103a353630b)
accept: */*
connection: close

* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 200 
< date: Sat, 05 Feb 2022 19:12:21 GMT
< content-type: application/octet-stream
< content-length: 10239
< last-modified: Fri, 04 Feb 2022 09:22:00 GMT
< etag: "61fcf038-27ff"
< expires: Sat, 04 Feb 2023 19:00:03 GMT
< cache-control: public, max-age=31448862
< cf-cache-status: HIT
< age: 34671
< accept-ranges: bytes
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< strict-transport-security: max-age=15552000
< x-content-type-options: nosniff
< server: cloudflare
< cf-ray: 6d8e7326cd40501e-WAW
< 
* Connection #0 to host database.clamav.net left intact
cdiff_apply: Parsed 104 lines and executed 104 commands
Retrieving https://database.clamav.net/daily-26444.cdiff
downloadFile: Download source:      https://database.clamav.net/daily-26444.cdiff
downloadFile: Download destination: ./clamav-c07624fb7f85311234618b74778dc010.tmp
*   Trying 2606:4700::6810:db54:443...
* Connected to database.clamav.net (2606:4700::6810:db54) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
*  start date: Jul 15 00:00:00 2021 GMT
*  expire date: Jul 14 23:59:59 2022 GMT
*  subjectAltName: host "database.clamav.net" matched cert's "database.clamav.net"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x8517d21d000)
> GET /daily-26444.cdiff HTTP/2
Host: database.clamav.net
user-agent: ClamAV/0.104.2 (OS: OpenBSD, ARCH: amd64, CPU: amd64, UUID: 510cb77e-33ce-426c-bb17-8103a353630b)
accept: */*
connection: close

* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 200 
< date: Sat, 05 Feb 2022 19:12:21 GMT
< content-type: application/octet-stream
< content-length: 7405
< last-modified: Sat, 05 Feb 2022 09:31:00 GMT
< etag: "61fe43d4-1ced"
< expires: Sun, 05 Feb 2023 19:08:59 GMT
< cache-control: public, max-age=31535798
< cf-cache-status: HIT
< age: 34570
< accept-ranges: bytes
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< strict-transport-security: max-age=15552000
< x-content-type-options: nosniff
< server: cloudflare
< cf-ray: 6d8e73290bca0016-WAW
< 
* Connection #0 to host database.clamav.net left intact
cdiff_apply: Parsed 75 lines and executed 75 commands
updatedb: Running g_cb_download_complete callback...
download_complete_callback: Download complete for database : /var/db/clamav/tmp.343bc7bfec/clamav-e4e06f12d1962b1c24ad5293adf4875e.tmp-daily.cld
download_complete_callback:   fc_context->bTestDatabases   : 1
download_complete_callback:   fc_context->bBytecodeEnabled : 1
Testing database: '/var/db/clamav/tmp.343bc7bfec/clamav-e4e06f12d1962b1c24ad5293adf4875e.tmp-daily.cld' ...
Loading signatures from /var/db/clamav/tmp.343bc7bfec/clamav-e4e06f12d1962b1c24ad5293adf4875e.tmp-daily.cld
WARNING: [LibClamAV] Can't load /var/db/clamav/tmp.343bc7bfec/clamav-e4e06f12d1962b1c24ad5293adf4875e.tmp-daily.cld: Malformed database
ERROR: Failed to load new database: Malformed database
WARNING: Stderr output from database load : WARNING: [LibClamAV] cli_tgzload: Invalid checksum for file daily.hsb [...] ERROR: Failed to load new database: Malformed database
WARNING: Database load exited with "Test failed"
ERROR: Database test FAILED.
updatedb: callback failed: Test failed (8)
ERROR: Unexpected error when attempting to update daily: Test failed
ERROR: Database update process failed: Test failed
ERROR: Update failed.
kucharskim commented 2 years ago 
# for i in 26441 26442 26443 26444
do
  echo "https://database.clamav.net/daily-$i.cdiff"
  curl -sf -o - "https://database.clamav.net/daily-$i.cdiff" | sha256
done
https://database.clamav.net/daily-26441.cdiff
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
https://database.clamav.net/daily-26442.cdiff
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
https://database.clamav.net/daily-26443.cdiff
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
https://database.clamav.net/daily-26444.cdiff
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
kucharskim commented 2 years ago 
# curl -svf -o - https://database.clamav.net/daily-26444.cdiff
*   Trying 2606:4700::6810:da54:443...
* Connected to database.clamav.net (2606:4700::6810:da54) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
*  start date: Jul 15 00:00:00 2021 GMT
*  expire date: Jul 14 23:59:59 2022 GMT
*  subjectAltName: host "database.clamav.net" matched cert's "database.clamav.net"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0xa9c91f75000)
> GET /daily-26444.cdiff HTTP/2
> Host: database.clamav.net
> user-agent: curl/7.81.0
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 403 
< date: Sat, 05 Feb 2022 19:16:03 GMT
< content-type: text/plain; charset=UTF-8
< content-length: 16
< x-frame-options: SAMEORIGIN
< referrer-policy: same-origin
< cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< expires: Thu, 01 Jan 1970 00:00:01 GMT
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< strict-transport-security: max-age=15552000
< x-content-type-options: nosniff
< server: cloudflare
< cf-ray: 6d8e78941f400037-WAW
* The requested URL returned error: 403
* stopped the pause stream!
* Connection #0 to host database.clamav.net left intact
kucharskim commented 2 years ago

That above sha256 is from empty string:

# : | sha256                                                                                                                                                                                                                             
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
kucharskim commented 2 years ago

Anyway, not sure why I am getting HTTP 403 from Cloudflare.

kucharskim commented 2 years ago

Ah, okay. I need to use specific user agent:

# for i in 26441 26442 26443 26444
  do
    echo "https://database.clamav.net/daily-$i.cdiff"
    curl -A 'ClamAV/0.104.2 (OS: OpenBSD, ARCH: amd64, CPU: amd64, UUID: 510cb77e-33ce-426c-bb17-8103a353630b)' \
      -sf -o - "https://database.clamav.net/daily-$i.cdiff" | sha256
  done
https://database.clamav.net/daily-26441.cdiff
1c445fbc537fc0a6373b44bd9c783017b4bd00216894de77dde0cebb16b1de29
https://database.clamav.net/daily-26442.cdiff
8114a0e3bf0efcf453fcf4e896d7a814a44baf0c3f14774e82b42822f739f74d
https://database.clamav.net/daily-26443.cdiff
48ba1c819a697ed344e432b5df2f2d99dd4e1708ae190c00c852555936d639fc
https://database.clamav.net/daily-26444.cdiff
f51af98a29d9160ff44e42facf08faa36dbf8b6ef9c961c36e3c19500568a6ce
kucharskim commented 2 years ago 
# ... curl ... | hexdump -vC | head -n3
https://database.clamav.net/daily-26441.cdiff
00000000  43 6c 61 6d 41 56 2d 44  69 66 66 3a 32 36 34 34  |ClamAV-Diff:2644|
00000010  31 3a 33 33 33 30 37 3a  1f 8b 08 00 00 00 00 00  |1:33307:........|
00000020  02 03 d5 bd 5d b3 1d b7  91 25 fa ee 5f c1 27 87  |....]....%.._.'.|
https://database.clamav.net/daily-26442.cdiff
00000000  43 6c 61 6d 41 56 2d 44  69 66 66 3a 32 36 34 34  |ClamAV-Diff:2644|
00000010  32 3a 32 34 38 39 32 3a  1f 8b 08 00 00 00 00 00  |2:24892:........|
00000020  02 03 dd 9c 4b 73 1d 37  92 b6 f7 fe 15 5a 39 a6  |....Ks.7.....Z9.|
https://database.clamav.net/daily-26443.cdiff
00000000  43 6c 61 6d 41 56 2d 44  69 66 66 3a 32 36 34 34  |ClamAV-Diff:2644|
00000010  33 3a 32 37 32 39 38 3a  1f 8b 08 00 00 00 00 00  |3:27298:........|
00000020  02 03 e5 bd 5d af 25 b9  71 2d f8 ae 5f d1 4f c6  |....].%.q-.._.O.|
https://database.clamav.net/daily-26444.cdiff
00000000  43 6c 61 6d 41 56 2d 44  69 66 66 3a 32 36 34 34  |ClamAV-Diff:2644|
00000010  34 3a 31 38 33 34 30 3a  1f 8b 08 00 00 00 00 00  |4:18340:........|
00000020  02 03 d5 7c 6b af 1d b7  91 ed f7 fc 0a 7f 0a 66  |...|k..........f|
kucharskim commented 2 years ago 
# sha256 < /var/db/clamav/daily.cld                                                                                                                                                                                                      
1464a2e840f8a6d828705e478412fed91fed36d28bec273615db24e41218feab

# hexdump -vC /var/db/clamav/daily.cld | head -n10
00000000  43 6c 61 6d 41 56 2d 56  44 42 3a 30 31 20 46 65  |ClamAV-VDB:01 Fe|
00000010  62 20 32 30 32 32 20 30  34 2d 32 39 20 2d 30 35  |b 2022 04-29 -05|
00000020  30 30 3a 32 36 34 34 30  3a 31 39 37 33 32 37 33  |00:26440:1973273|
00000030  3a 39 30 3a 58 3a 58 3a  72 61 79 6e 6d 61 6e 3a  |:90:X:X:raynman:|
00000040  31 36 34 33 37 30 37 37  35 36 20 20 20 20 20 20  |1643707756      |
00000050  20 20 20 20 20 20 20 20  20 20 20 20 20 20 20 20  |                |
00000060  20 20 20 20 20 20 20 20  20 20 20 20 20 20 20 20  |                |
00000070  20 20 20 20 20 20 20 20  20 20 20 20 20 20 20 20  |                |
00000080  20 20 20 20 20 20 20 20  20 20 20 20 20 20 20 20  |                |
00000090  20 20 20 20 20 20 20 20  20 20 20 20 20 20 20 20  |                |
micahsnyder commented 2 years ago

Anyway, not sure why I am getting HTTP 403 from Cloudflare.

Hi @kucharskim sorry about the lag responding to you. I must have accidentally marked your issue as read on my side and only just discovered it when reviewing the open issues.

As you noticed, you're getting a 403 because we try to require users to use only freshclam or cvdupdate to download from our CDN (Cloudflare) to keep costs down. We no longer use a volunteer-hosted network of mirrors.

It looks like the incremental update process from your daily.cld version 26440 up to version 26444 failed. We had a report of a similar issue from someone going by the username 'Tribune' just this morning on our Discord server. But your two reports, together, are the first I've seen of this specific issue. I haven't seen any mention of it on the users mailing list.

Are you still facing this issue or has it since resolved itself with subsequent updates?

kucharskim commented 2 years ago

In last few days no issues. It started to work a while ago and so far so good.

micahsnyder commented 2 years ago

Glad it's working for now. We'll track this in the future. Please bump this ticket if it happens again.

garrettboone commented 2 years ago

Removing the extra DatabaseMirror line in freshclam.conf helped me as well. Even after deleting daily databases, the freshclam update was failing out. Must have been downloading from the main source but trying to check the hash from a non-working one.

kucharskim commented 2 years ago

This happens again:

# freshclam  
ClamAV update process started at Sun Feb 20 04:02:03 2022
daily database available for update (local version: 26457, remote version: 26458)
Current database is 1 version behind.
Downloading database patch # 26458...
Time:    0.1s, ETA:    0.0s [========================>]    9.13KiB/9.13KiB
Testing database: '/var/db/clamav/tmp.89a287cabc/clamav-4023429d700732bd70555b1dc3baa120.tmp-daily.cld' ...
WARNING: Stderr output from database load : WARNING: [LibClamAV] hm_addhash_str: invalid hash dba1b17d14df13f3ecb1a62d2d3efec [...] ERROR: Failed to load new database: Malformed database
WARNING: Database load exited with "Test failed"
ERROR: Database test FAILED.
ERROR: Unexpected error when attempting to update daily: Test failed
ERROR: Database update process failed: Test failed
ERROR: Update failed.
kucharskim commented 2 years ago

However error is slightly different than I initially reported in this bug report (my previous comment).

# freshclam --version
ClamAV 0.104.2/26457/Fri Feb 18 09:25:22 2022
micahsnyder commented 2 years ago

@kucharskim Are there by chance multiple copies of the daily database in your /var/db/clamav/ directory? Like a .cld and .cvd version?

kucharskim commented 2 years ago 
# ls -lha /var/db/clamav/
total 689228
drwxr-xr-x   2 _clamav  _clamav   512B Feb 20 04:02 .
drwxr-xr-x  11 root     wheel     512B Feb 13 23:21 ..
-rw-r--r--   1 _clamav  _clamav   287K Dec 31 22:16 bytecode.cvd
-rw-r--r--   1 _clamav  _clamav   173M Feb 18 12:22 daily.cld
-rw-r--r--   1 _clamav  _clamav    69B Feb 17 16:32 freshclam.dat
-rw-r--r--   1 _clamav  _clamav   163M Dec 31 22:16 main.cvd
micahsnyder commented 2 years ago

Hm very strange. I am still at a loss.

kucharskim commented 2 years ago

from ktrace:

 74636 freshclam CALL  read(3,0x5e8608b9000,0x4000)
 74636 freshclam GIO   fd 3 read 83 bytes
       "WARNING: [LibClamAV] hm_addhash_str: invalid hash dba1b17d14df13f3\M-decb1a62d2d3efec
       "
 74636 freshclam RET   read 83/0x53
 74636 freshclam CALL  read(3,0x5e8608b9000,0x4000)
 74636 freshclam GIO   fd 3 read 228 bytes
       "WARNING: [LibClamAV] cli_loadhash: Malformed hash string at line 1465497
        WARNING: [LibClamAV] cli_loadhash: Problem parsing database at line 1465497
        WARNING: [LibClamAV] Can't load daily.hsb: Invalid argument passed to function
       "
kucharskim commented 2 years ago

Is there anyway to keep tempfiles around in freshclam so I could have a look at:

Testing database: '/var/db/clamav/tmp.1e5a5f5a89/clamav-f736c814e8821dd082490e6a667218f8.tmp-daily.cld'

that temp daily.cld?

kucharskim commented 2 years ago 
# freshclam -vvvv
Current working dir is /var/db/clamav/
Loaded freshclam.dat:
  version:    1
  uuid:       e56bede1-01f1-40e4-9100-787d0c4fbb91
ClamAV update process started at Sun Feb 20 04:29:47 2022
Current working dir is /var/db/clamav/
Querying current.cvd.clamav.net
TTL: 207
fc_dns_query_update_info: Software version from DNS: 0.103.5
Current working dir is /var/db/clamav/
check_for_new_database_version: Local copy of daily found: daily.cld.
query_remote_database_version: daily.cvd version from DNS: 26458
daily database available for update (local version: 26457, remote version: 26458)
Retrieving https://database.clamav.net/daily-26458.cdiff
downloadFile: Download source:      https://database.clamav.net/daily-26458.cdiff
downloadFile: Download destination: ./clamav-9e743d4cfccd27ff0d54137b82207336.tmp
*   Trying 2606:4700::6810:db54:443...
* Connected to database.clamav.net (2606:4700::6810:db54) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
*  start date: Jul 15 00:00:00 2021 GMT
*  expire date: Jul 14 23:59:59 2022 GMT
*  subjectAltName: host "database.clamav.net" matched cert's "database.clamav.net"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0xbd89aaf9000)
> GET /daily-26458.cdiff HTTP/2
Host: database.clamav.net
user-agent: ClamAV/0.104.2 (OS: OpenBSD, ARCH: amd64, CPU: amd64, UUID: e56bede1-01f1-40e4-9100-787d0c4fbb91)
accept: */*
connection: close

* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 200 
< date: Sun, 20 Feb 2022 04:29:48 GMT
< content-type: application/octet-stream
< content-length: 9349
< last-modified: Sat, 19 Feb 2022 09:28:00 GMT
< etag: "6210b820-2485"
< expires: Mon, 20 Feb 2023 04:26:08 GMT
< cache-control: public, max-age=31535780
< cf-cache-status: HIT
< age: 68209
< accept-ranges: bytes
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< strict-transport-security: max-age=15552000
< x-content-type-options: nosniff
< server: cloudflare
< cf-ray: 6e04fef9dd9ab514-VNO
< 
* Connection #0 to host database.clamav.net left intact
cdiff_apply: Parsed 116 lines and executed 116 commands
updatedb: Running g_cb_download_complete callback...
download_complete_callback: Download complete for database : /var/db/clamav/tmp.0744bf8e03/clamav-5e4dc6b647cb2d11249115d735d69cdb.tmp-daily.cld
download_complete_callback:   fc_context->bTestDatabases   : 1
download_complete_callback:   fc_context->bBytecodeEnabled : 1
Testing database: '/var/db/clamav/tmp.0744bf8e03/clamav-5e4dc6b647cb2d11249115d735d69cdb.tmp-daily.cld' ...
Loading signatures from /var/db/clamav/tmp.0744bf8e03/clamav-5e4dc6b647cb2d11249115d735d69cdb.tmp-daily.cld
WARNING: [LibClamAV] cli_loadhash: Malformed hash string at line 1465497
WARNING: [LibClamAV] cli_loadhash: Problem parsing database at line 1465497
WARNING: [LibClamAV] Can't load daily.hsb: Invalid argument passed to function
WARNING: [LibClamAV] cli_tgzload: Can't load daily.hsb
WARNING: [LibClamAV] Can't load /var/db/clamav/tmp.0744bf8e03/clamav-5e4dc6b647cb2d11249115d735d69cdb.tmp-daily.cld: Malformed database
ERROR: Failed to load new database: Malformed database
WARNING: Stderr output from database load : WARNING: [LibClamAV] hm_addhash_str: invalid hash dba1b17d14df13f3�ecb1a62d2d3efec [...] ERROR: Failed to load new database: Malformed database
WARNING: Database load exited with "Test failed"
ERROR: Database test FAILED.
updatedb: callback failed: Test failed (8)
ERROR: Unexpected error when attempting to update daily: Test failed
ERROR: Database update process failed: Test failed
ERROR: Update failed.
kucharskim commented 2 years ago

Not sure why there is 0xe4 in the middle of that invalid hash

# grep -F 'invalid hash' freshclam.out | hexdump -vC                                                                                                                                                                                     
00000000  57 41 52 4e 49 4e 47 3a  20 53 74 64 65 72 72 20  |WARNING: Stderr |
00000010  6f 75 74 70 75 74 20 66  72 6f 6d 20 64 61 74 61  |output from data|
00000020  62 61 73 65 20 6c 6f 61  64 20 3a 20 57 41 52 4e  |base load : WARN|
00000030  49 4e 47 3a 20 5b 4c 69  62 43 6c 61 6d 41 56 5d  |ING: [LibClamAV]|
00000040  20 68 6d 5f 61 64 64 68  61 73 68 5f 73 74 72 3a  | hm_addhash_str:|
00000050  20 69 6e 76 61 6c 69 64  20 68 61 73 68 20 64 62  | invalid hash db|
00000060  61 31 62 31 37 64 31 34  64 66 31 33 66 33 e4 65  |a1b17d14df13f3.e|
00000070  63 62 31 61 36 32 64 32  64 33 65 66 65 63 20 5b  |cb1a62d2d3efec [|
00000080  2e 2e 2e 5d 20 45 52 52  4f 52 3a 20 46 61 69 6c  |...] ERROR: Fail|
00000090  65 64 20 74 6f 20 6c 6f  61 64 20 6e 65 77 20 64  |ed to load new d|
000000a0  61 74 61 62 61 73 65 3a  20 4d 61 6c 66 6f 72 6d  |atabase: Malform|
000000b0  65 64 20 64 61 74 61 62  61 73 65 0a              |ed database.|
000000bc
micahsnyder commented 2 years ago

There's no option in the config or CLI to keep the the temp-files.

It looks to me like the original hash signature was:

dba1b17d14df13f3decb1a62d2d3efec:44856:Pdf.Dropper.Agent-9574624-0:73

This signature was added in August 2020 and hasn't been moved/removed since, as far as I can tell.

I too do not know how that 0x64 (d) came to be a 0xe4 in your updated database.

kucharskim commented 1 year ago

I am having this issue once in a while still, even I did not report anything new. Today I've looked into this again. I am going to report details again, as clamav was update over the year or so many times on the machine.

Problem is visible as follows:

x1c# ls -lha /var/db/clamav/                                                                                                                                                                                                                 
total 715500
drwxr-xr-x   2 _clamav  _clamav   512B Sep  1 13:27 .
drwxr-xr-x  14 root     wheel     512B Aug 28 08:21 ..
-rw-r--r--   1 _clamav  _clamav   285K Mar  7 23:17 bytecode.cvd
-rw-r--r--   1 _clamav  _clamav   186M Aug 30 08:14 daily.cld
-rw-r--r--   1 _clamav  _clamav    69B Oct 12  2022 freshclam.dat
-rw-r--r--   1 _clamav  _clamav   163M Dec 31  2021 main.cvd

x1c# freshclam                                                                                                                                                                                                                               
ClamAV update process started at Fri Sep  1 13:27:32 2023
daily database available for update (local version: 27016, remote version: 27018)
Current database is 2 versions behind.
Downloading database patch # 27017...
Time:    0.1s, ETA:    0.0s [========================>]    3.16KiB/3.16KiB
Downloading database patch # 27018...
Time:    0.1s, ETA:    0.0s [========================>]    2.66KiB/2.66KiB
Testing database: '/var/db/clamav/tmp.f3b4a893cf/clamav-5c17e1724a713754e6692c405e35dd01.tmp-daily.cld' ...
WARNING: Stderr output from database load : WARNING: [LibClamAV] hm_addhash_str: invalid hash 99d3f86ca69d302b4d82bba2fff3104 [...] ERROR: Failed to load new database: Malformed database
WARNING: Database load exited with "Test failed"
ERROR: Database test FAILED.
ERROR: Unexpected error when attempting to update daily: Test failed
ERROR: Database update process failed: Test failed
ERROR: Update failed.

x1c# echo $?
8

x1c# cat /etc/freshclam.conf                                                                                                                                                                                                                 
# /etc/freshclam.conf
LogSyslog yes
ScriptedUpdates yes
DatabaseMirror database.clamav.net
DNSDatabaseInfo current.cvd.clamav.net
MaxAttempts 5
ConnectTimeout 10
ReceiveTimeout 900
Bytecode yes
kucharskim commented 1 year ago 
x1c# freshclam -V
ClamAV 1.1.2/27016/Wed Aug 30 07:37:04 2023

x1c# freshclam -vvv --debug
Current working dir is /var/db/clamav/
Loaded freshclam.dat:
  version:    1
  uuid:       e56bede1-01f1-40e4-9100-787d0c4fbb91
ClamAV update process started at Fri Sep  1 13:29:25 2023
Current working dir is /var/db/clamav/
Querying current.cvd.clamav.net
TTL: 1150
fc_dns_query_update_info: Software version from DNS: 0.103.10
Current working dir is /var/db/clamav/
check_for_new_database_version: Local copy of daily found: daily.cld.
query_remote_database_version: daily.cvd version from DNS: 27018
daily database available for update (local version: 27016, remote version: 27018)
LibClamAV debug: in cli_untgz()
LibClamAV debug: cli_untgz: Unpacking /var/db/clamav/tmp.82d81c0ca0/clamav-c5a51d509af64aefa22edf2e27937b1c.tmp/COPYING
LibClamAV debug: cli_untgz: Unpacking /var/db/clamav/tmp.82d81c0ca0/clamav-c5a51d509af64aefa22edf2e27937b1c.tmp/daily.info
LibClamAV debug: cli_untgz: Unpacking /var/db/clamav/tmp.82d81c0ca0/clamav-c5a51d509af64aefa22edf2e27937b1c.tmp/daily.cfg
LibClamAV debug: cli_untgz: Unpacking /var/db/clamav/tmp.82d81c0ca0/clamav-c5a51d509af64aefa22edf2e27937b1c.tmp/daily.ign
LibClamAV debug: cli_untgz: Unpacking /var/db/clamav/tmp.82d81c0ca0/clamav-c5a51d509af64aefa22edf2e27937b1c.tmp/daily.ign2
LibClamAV debug: cli_untgz: Unpacking /var/db/clamav/tmp.82d81c0ca0/clamav-c5a51d509af64aefa22edf2e27937b1c.tmp/daily.ftm
LibClamAV debug: cli_untgz: Unpacking /var/db/clamav/tmp.82d81c0ca0/clamav-c5a51d509af64aefa22edf2e27937b1c.tmp/daily.hdb
LibClamAV debug: cli_untgz: Unpacking /var/db/clamav/tmp.82d81c0ca0/clamav-c5a51d509af64aefa22edf2e27937b1c.tmp/daily.hdu
LibClamAV debug: cli_untgz: Unpacking /var/db/clamav/tmp.82d81c0ca0/clamav-c5a51d509af64aefa22edf2e27937b1c.tmp/daily.hsb
LibClamAV debug: cli_untgz: Unpacking /var/db/clamav/tmp.82d81c0ca0/clamav-c5a51d509af64aefa22edf2e27937b1c.tmp/daily.hsu
LibClamAV debug: cli_untgz: Unpacking /var/db/clamav/tmp.82d81c0ca0/clamav-c5a51d509af64aefa22edf2e27937b1c.tmp/daily.mdb
LibClamAV debug: cli_untgz: Unpacking /var/db/clamav/tmp.82d81c0ca0/clamav-c5a51d509af64aefa22edf2e27937b1c.tmp/daily.mdu
LibClamAV debug: cli_untgz: Unpacking /var/db/clamav/tmp.82d81c0ca0/clamav-c5a51d509af64aefa22edf2e27937b1c.tmp/daily.msb
LibClamAV debug: cli_untgz: Unpacking /var/db/clamav/tmp.82d81c0ca0/clamav-c5a51d509af64aefa22edf2e27937b1c.tmp/daily.msu
LibClamAV debug: cli_untgz: Unpacking /var/db/clamav/tmp.82d81c0ca0/clamav-c5a51d509af64aefa22edf2e27937b1c.tmp/daily.ndb
LibClamAV debug: cli_untgz: Unpacking /var/db/clamav/tmp.82d81c0ca0/clamav-c5a51d509af64aefa22edf2e27937b1c.tmp/daily.ndu
LibClamAV debug: cli_untgz: Unpacking /var/db/clamav/tmp.82d81c0ca0/clamav-c5a51d509af64aefa22edf2e27937b1c.tmp/daily.ldb
LibClamAV debug: cli_untgz: Unpacking /var/db/clamav/tmp.82d81c0ca0/clamav-c5a51d509af64aefa22edf2e27937b1c.tmp/daily.ldu
LibClamAV debug: cli_untgz: Unpacking /var/db/clamav/tmp.82d81c0ca0/clamav-c5a51d509af64aefa22edf2e27937b1c.tmp/daily.idb
LibClamAV debug: cli_untgz: Unpacking /var/db/clamav/tmp.82d81c0ca0/clamav-c5a51d509af64aefa22edf2e27937b1c.tmp/daily.fp
LibClamAV debug: cli_untgz: Unpacking /var/db/clamav/tmp.82d81c0ca0/clamav-c5a51d509af64aefa22edf2e27937b1c.tmp/daily.sfp
LibClamAV debug: cli_untgz: Unpacking /var/db/clamav/tmp.82d81c0ca0/clamav-c5a51d509af64aefa22edf2e27937b1c.tmp/daily.pdb
LibClamAV debug: cli_untgz: Unpacking /var/db/clamav/tmp.82d81c0ca0/clamav-c5a51d509af64aefa22edf2e27937b1c.tmp/daily.wdb
LibClamAV debug: cli_untgz: Unpacking /var/db/clamav/tmp.82d81c0ca0/clamav-c5a51d509af64aefa22edf2e27937b1c.tmp/daily.crb
LibClamAV debug: cli_untgz: Unpacking /var/db/clamav/tmp.82d81c0ca0/clamav-c5a51d509af64aefa22edf2e27937b1c.tmp/daily.cdb
LibClamAV debug: in cli_untgz_cleanup()
Retrieving https://database.clamav.net/daily-27017.cdiff
downloadFile: Download source:      https://database.clamav.net/daily-27017.cdiff
downloadFile: Download destination: ./clamav-01d331dc34f7e451c136c4f8a4f381b2.tmp
LibClamAV debug: cdiff_apply() - called with file_descriptor=3, mode=1
LibClamAV debug: cdiff_apply() - current directory is /var/db/clamav/tmp.82d81c0ca0/clamav-c5a51d509af64aefa22edf2e27937b1c.tmp
LibClamAV debug: read_dsig() - dsig length is 350
LibClamAV debug: cdiff_apply() - final dsig length is 342
0x4D 0x59 0x30 0x6C 0x34 0x6E 0x4C 0x35 0x66 0x73 0x32 0x32 0x5A 0x51 0x50 0x70 0x52 0x69 0x38 0x4C 0x2F 0x4F 0x39 0x70 0x57 0x32 0x61 0x51 0x62 0x44 0x7A 0x68 0x78 0x6F 0x63 0x6D 0x33 0x4F 0x33 0x37 0x2F 0x4B 0x4A 0x35 0x75 0x76 0x5A 0x62 0x37 0x46 0x6F 0x42 0x68 0x36 0x69 0x70 0x6C 0x42 0x42 0x4B 0x65 0x70 0x75 0x59 0x5A 0x4F 0x42 0x78 0x74 0x35 0x74 0x43 0x4C 0x73 0x6B 0x64 0x56 0x41 0x58 0x67 0x42 0x4F 0x61 0x4A 0x36 0x64 0x4D 0x6D 0x42 0x74 0x53 0x77 0x4B 0x49 0x48 0x49 0x68 0x32 0x58 0x32 0x78 0x72 0x54 0x68 0x32 0x54 0x61 0x59 0x58 0x35 0x52 0x6C 0x4C 0x33 0x67 0x49 0x57 0x57 0x50 0x35 0x6D 0x58 0x45 0x45 0x6D 0x35 0x44 0x6A 0x42 0x6E 0x75 0x42 0x4C 0x4C 0x32 0x67 0x37 0x6A 0x77 0x7A 0x58 0x49 0x52 0x41 0x2F 0x77 0x35 0x4C 0x69 0x2F 0x78 0x56 0x48 0x42 0x5A 0x56 0x76 0x33 0x2F 0x58 0x53 0x37 0x4B 0x46 0x66 0x77 0x43 0x2F 0x75 0x68 0x73 0x2F 0x4C 0x37 0x52 0x6F 0x62 0x7A 0x34 0x50 0x6C 0x76 0x43 0x77 0x66 0x56 0x35 0x47 0x47 0x7A 0x6D 0x79 0x51 0x4E 0x2F 0x72 0x64 0x4B 0x39 0x56 0x7A 0x65 0x43 0x67 0x47 0x79 0x4E 0x70 0x6D 0x5A 0x74 0x79 0x38 0x77 0x76 0x35 0x4D 0x63 0x31 0x55 0x2F 0x6B 0x47 0x47 0x4D 0x4C 0x32 0x32 0x39 0x30 0x44 0x55 0x57 0x63 0x77 0x41 0x42 0x6E 0x64 0x52 0x5A 0x44 0x57 0x4B 0x74 0x58 0x53 0x6A 0x7A 0x4E 0x57 0x33 0x55 0x46 0x36 0x56 0x38 0x66 0x2B 0x4D 0x54 0x61 0x61 0x53 0x54 0x53 0x70 0x67 0x30 0x50 0x63 0x35 0x43 0x67 0x6C 0x4D 0x48 0x6F 0x59 0x34 0x53 0x4A 0x78 0x48 0x69 0x4F 0x36 0x6E 0x4B 0x49 0x57 0x49 0x2F 0x6C 0x51 0x6C 0x38 0x68 0x53 0x39 0x70 0x77 0x4C 0x47 0x66 0x54 0x69 0x50 0x57 0x4A 0x30 0x79 0x41 0x53 0x56 0x6A 0x46 0x66 0x56 0x45 0x37 0x56 0x39 0x6B 0x35 0x6B 0x31 0x36 0x74 0x77 0x4D 0x66 0x68 0x54 0x6E 0x6B 0x78 0x62 0x66 0x6B 0x68 0x62 
LibClamAV debug: cdiff_apply() - sha256: d167a3ab637efccac5dad614ec75a5719634759807d71834d68319436a761067
LibClamAV debug: cdiff_apply() - cli_versig2() result = 0
LibClamAV debug: cdiff_apply() - header len = 7434, file len = 3240, header offset = 23
LibClamAV debug: cdiff_apply() - current file offset = 23
LibClamAV debug: cmd_close() - finished
LibClamAV debug: cmd_close() - finished
LibClamAV debug: cmd_close() - finished
LibClamAV debug: Expected 7434 decompressed bytes, read 7483 decompressed bytes
* processing: https://database.clamav.net/daily-27017.cdiff
*   Trying 104.16.218.84:443...
* Connected to database.clamav.net (104.16.218.84) port 443
* ALPN: offers h2,http/1.1
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
*  start date: May 15 00:00:00 2023 GMT
*  expire date: May 14 23:59:59 2024 GMT
*  subjectAltName: host "database.clamav.net" matched cert's "database.clamav.net"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* using HTTP/2
* h2 [:method: GET]
* h2 [:scheme: https]
* h2 [:authority: database.clamav.net]
* h2 [:path: /daily-27017.cdiff]
* h2 [user-agent: ClamAV/1.1.2 (OS: OpenBSD, ARCH: amd64, CPU: amd64, UUID: e56bede1-01f1-40e4-9100-787d0c4fbb91)]
* h2 [accept: */*]
* Using Stream ID: 1
> GET /daily-27017.cdiff HTTP/2
Host: database.clamav.net
User-Agent: ClamAV/1.1.2 (OS: OpenBSD, ARCH: amd64, CPU: amd64, UUID: e56bede1-01f1-40e4-9100-787d0c4fbb91)
Accept: */*
Connection: close

< HTTP/2 200 
< date: Fri, 01 Sep 2023 13:29:26 GMT
< content-type: application/octet-stream
< content-length: 3240
< last-modified: Thu, 31 Aug 2023 07:40:00 GMT
< etag: "64f043d0-ca8"
< expires: Fri, 30 Aug 2024 13:21:14 GMT
< cache-control: public, max-age=31449108
< cf-cache-status: HIT
< age: 20473
< accept-ranges: bytes
< set-cookie: __cf_bm=yZxfHwA5chshLDTdgthuEiI5OKVesFWRuvYH7eT1Ftg-1693574966-0-AXhs0fkCAQYcdPOWVNsts5M+g+Mie6WLZKyaJjvQGyZV9+1mm1UK8IHbWITQjCE8nl3xxaO48xI1xkLvjByT9QY=; path=/; expires=Fri, 01-Sep-23 13:59:26 GMT; domain=.clamav.net; HttpOnly; Secure; SameSite=None
< strict-transport-security: max-age=15552000
< x-content-type-options: nosniff
< server: cloudflare
< cf-ray: 7ffddcb59ff73540-WAW
< 
* Connection #0 to host database.clamav.net left intact
Retrieving https://database.clamav.net/daily-27018.cdiff
downloadFile: Download source:      https://database.clamav.net/daily-27018.cdiff
downloadFile: Download destination: ./clamav-ce1067171b90775ea6e9fdad09ae6bfb.tmp
LibClamAV debug: cdiff_apply() - called with file_descriptor=3, mode=1
LibClamAV debug: cdiff_apply() - current directory is /var/db/clamav/tmp.82d81c0ca0/clamav-c5a51d509af64aefa22edf2e27937b1c.tmp
LibClamAV debug: read_dsig() - dsig length is 350
LibClamAV debug: cdiff_apply() - final dsig length is 341
0x52 0x6B 0x39 0x42 0x6E 0x35 0x52 0x37 0x73 0x58 0x65 0x6F 0x37 0x65 0x66 0x36 0x39 0x6B 0x41 0x4D 0x6D 0x4A 0x6A 0x6A 0x35 0x4F 0x58 0x61 0x51 0x31 0x7A 0x77 0x75 0x32 0x35 0x4F 0x37 0x6C 0x46 0x64 0x46 0x59 0x74 0x6F 0x44 0x42 0x72 0x55 0x4F 0x79 0x48 0x56 0x52 0x53 0x6E 0x46 0x62 0x79 0x78 0x61 0x31 0x76 0x59 0x43 0x68 0x70 0x4C 0x59 0x49 0x78 0x44 0x6A 0x78 0x76 0x2B 0x2B 0x4E 0x41 0x54 0x52 0x42 0x59 0x58 0x47 0x38 0x74 0x44 0x74 0x2F 0x41 0x44 0x7A 0x63 0x2F 0x7A 0x53 0x38 0x57 0x79 0x52 0x76 0x59 0x74 0x2B 0x44 0x66 0x61 0x6B 0x73 0x33 0x78 0x6A 0x55 0x42 0x72 0x41 0x43 0x6D 0x4B 0x71 0x7A 0x61 0x45 0x4D 0x66 0x45 0x43 0x75 0x4E 0x49 0x74 0x45 0x70 0x69 0x77 0x79 0x53 0x6C 0x62 0x71 0x53 0x49 0x77 0x62 0x74 0x78 0x35 0x78 0x65 0x48 0x70 0x6B 0x76 0x61 0x2F 0x55 0x61 0x33 0x36 0x56 0x37 0x70 0x79 0x55 0x66 0x58 0x6F 0x66 0x4A 0x58 0x4C 0x2B 0x44 0x43 0x76 0x70 0x68 0x62 0x42 0x56 0x30 0x43 0x51 0x79 0x47 0x34 0x39 0x63 0x6E 0x78 0x31 0x6D 0x70 0x51 0x54 0x69 0x47 0x46 0x4B 0x4E 0x73 0x54 0x61 0x57 0x52 0x5A 0x75 0x7A 0x76 0x4C 0x6D 0x2B 0x7A 0x2B 0x6A 0x38 0x68 0x7A 0x57 0x6D 0x36 0x44 0x6C 0x65 0x73 0x2F 0x52 0x57 0x37 0x4A 0x78 0x4F 0x6A 0x37 0x6D 0x54 0x6D 0x56 0x77 0x71 0x69 0x44 0x44 0x42 0x69 0x79 0x4B 0x49 0x75 0x34 0x35 0x69 0x6F 0x32 0x2B 0x33 0x6A 0x44 0x45 0x4D 0x63 0x2B 0x2B 0x6C 0x70 0x53 0x77 0x4E 0x55 0x61 0x6D 0x46 0x4D 0x2F 0x64 0x33 0x30 0x62 0x56 0x62 0x76 0x61 0x35 0x30 0x46 0x75 0x41 0x35 0x34 0x72 0x73 0x73 0x7A 0x6E 0x42 0x50 0x58 0x4E 0x69 0x7A 0x30 0x2B 0x6B 0x4E 0x66 0x57 0x47 0x4E 0x68 0x44 0x69 0x62 0x2F 0x5A 0x57 0x78 0x36 0x31 0x6F 0x57 0x58 0x52 0x34 0x55 0x6E 0x5A 0x35 0x4B 0x67 0x43 0x75 0x68 0x2B 0x59 0x30 0x44 0x2B 0x69 0x75 0x64 0x79 
LibClamAV debug: cdiff_apply() - sha256: ccad9aae58f5de2724c61c60f7e76b4486a67366e5cbb60f60473277e6fe4a27
LibClamAV debug: cdiff_apply() - cli_versig2() result = 0
LibClamAV debug: cdiff_apply() - header len = 4716, file len = 2723, header offset = 23
LibClamAV debug: cdiff_apply() - current file offset = 23
LibClamAV debug: cmd_close() - finished
LibClamAV debug: cmd_close() - finished
LibClamAV debug: cmd_close() - finished
LibClamAV debug: Expected 4716 decompressed bytes, read 4745 decompressed bytes
* processing: https://database.clamav.net/daily-27018.cdiff
*   Trying 104.16.218.84:443...
* Connected to database.clamav.net (104.16.218.84) port 443
* ALPN: offers h2,http/1.1
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
*  start date: May 15 00:00:00 2023 GMT
*  expire date: May 14 23:59:59 2024 GMT
*  subjectAltName: host "database.clamav.net" matched cert's "database.clamav.net"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* using HTTP/2
* h2 [:method: GET]
* h2 [:scheme: https]
* h2 [:authority: database.clamav.net]
* h2 [:path: /daily-27018.cdiff]
* h2 [user-agent: ClamAV/1.1.2 (OS: OpenBSD, ARCH: amd64, CPU: amd64, UUID: e56bede1-01f1-40e4-9100-787d0c4fbb91)]
* h2 [accept: */*]
* Using Stream ID: 1
> GET /daily-27018.cdiff HTTP/2
Host: database.clamav.net
User-Agent: ClamAV/1.1.2 (OS: OpenBSD, ARCH: amd64, CPU: amd64, UUID: e56bede1-01f1-40e4-9100-787d0c4fbb91)
Accept: */*
Connection: close

< HTTP/2 200 
< date: Fri, 01 Sep 2023 13:29:26 GMT
< content-type: application/octet-stream
< content-length: 2723
< last-modified: Fri, 01 Sep 2023 07:45:00 GMT
< etag: "64f1967c-aa3"
< expires: Sat, 31 Aug 2024 13:26:11 GMT
< cache-control: public, max-age=31535805
< cf-cache-status: HIT
< age: 20419
< accept-ranges: bytes
< set-cookie: __cf_bm=qn5nmgq.wHin6UFPgCiE0.WDzqYfHi.R9rzKUxWMXj0-1693574966-0-Ab1AR+egDDchGOH8ZWB/2fO6lxsGwvuUBBkTCXsQCaYikOrY5VO3d+QWLv0zUX1yNTVm5+f+zPq62X84gMlAHNk=; path=/; expires=Fri, 01-Sep-23 13:59:26 GMT; domain=.clamav.net; HttpOnly; Secure; SameSite=None
< strict-transport-security: max-age=15552000
< x-content-type-options: nosniff
< server: cloudflare
< cf-ray: 7ffddcb76f3b34da-WAW
< 
* Connection #0 to host database.clamav.net left intact
updatedb: Running g_cb_download_complete callback...
download_complete_callback: Download complete for database : /var/db/clamav/tmp.82d81c0ca0/clamav-bc4fdf2704b8987ff0051938567e0e15.tmp-daily.cld
download_complete_callback:   fc_context->bTestDatabases   : 1
download_complete_callback:   fc_context->bBytecodeEnabled : 1
Testing database: '/var/db/clamav/tmp.82d81c0ca0/clamav-bc4fdf2704b8987ff0051938567e0e15.tmp-daily.cld' ...
Loading signatures from /var/db/clamav/tmp.82d81c0ca0/clamav-bc4fdf2704b8987ff0051938567e0e15.tmp-daily.cld
LibClamAV debug: Initializing phishcheck module
LibClamAV debug: Phishcheck: Compiling regex: ^ *(http|https|ftp:(//)?)?[0-9]{1,3}(\.[0-9]{1,3}){3}[/?:]? *$
LibClamAV debug: Phishcheck module initialized
LibClamAV debug: Bytecode initialized in interpreter mode
LibClamAV debug: in cli_cvdload()
LibClamAV debug: in cli_tgzload()
LibClamAV debug: daily.info loaded
LibClamAV debug: in cli_tgzload_cleanup()
LibClamAV debug: in cli_tgzload()
LibClamAV debug: daily.cfg loaded
LibClamAV debug: daily.ign loaded
LibClamAV debug: daily.ign2 loaded
LibClamAV debug: Initializing engine matching structures
LibClamAV debug: cli_loadftm: File type signature for UDF not loaded (required f-level: 191)
LibClamAV debug: Loaded 156 filetype definitions
LibClamAV debug: daily.ftm loaded
LibClamAV debug: daily.hdb loaded
LibClamAV debug: hashtab.c:Growing hashtable 0x6e542e05868, because it has exceeded maxfill, old size: 64
LibClamAV debug: hashtab.c: new capacity: 128
LibClamAV debug: Table 0x6e542e05868 size after grow: 128
LibClamAV debug: daily.hdu loaded
LibClamAV debug: hashtab.c:Growing hashtable 0x6e542e05868, because it has exceeded maxfill, old size: 128
LibClamAV debug: hashtab.c: new capacity: 256
LibClamAV debug: Table 0x6e542e05868 size after grow: 256
LibClamAV debug: hashtab.c:Growing hashtable 0x6e542e05868, because it has exceeded maxfill, old size: 256
LibClamAV debug: hashtab.c: new capacity: 512
LibClamAV debug: Table 0x6e542e05868 size after grow: 512
LibClamAV debug: hashtab.c:Growing hashtable 0x6e542e05868, because it has exceeded maxfill, old size: 512
LibClamAV debug: hashtab.c: new capacity: 1024
LibClamAV debug: Table 0x6e542e05868 size after grow: 1024
LibClamAV debug: hashtab.c:Growing hashtable 0x6e542e05868, because it has exceeded maxfill, old size: 1024
LibClamAV debug: hashtab.c: new capacity: 2048
LibClamAV debug: Table 0x6e542e05868 size after grow: 2048
LibClamAV debug: hashtab.c:Growing hashtable 0x6e542e05868, because it has exceeded maxfill, old size: 2048
LibClamAV debug: hashtab.c: new capacity: 4096
LibClamAV debug: Table 0x6e542e05868 size after grow: 4096
LibClamAV debug: hashtab.c:Growing hashtable 0x6e542e05868, because it has exceeded maxfill, old size: 4096
LibClamAV debug: hashtab.c: new capacity: 8192
LibClamAV debug: Table 0x6e542e05868 size after grow: 8192
LibClamAV debug: hashtab.c:Growing hashtable 0x6e542e05868, because it has exceeded maxfill, old size: 8192
LibClamAV debug: hashtab.c: new capacity: 16384
LibClamAV debug: Table 0x6e542e05868 size after grow: 16384
LibClamAV debug: hashtab.c:Growing hashtable 0x6e542e05868, because it has exceeded maxfill, old size: 16384
LibClamAV debug: hashtab.c: new capacity: 32768
LibClamAV debug: Table 0x6e542e05868 size after grow: 32768
LibClamAV debug: hashtab.c:Growing hashtable 0x6e542e05868, because it has exceeded maxfill, old size: 32768
LibClamAV debug: hashtab.c: new capacity: 65536
LibClamAV debug: Table 0x6e542e05868 size after grow: 65536
LibClamAV debug: hashtab.c:Growing hashtable 0x6e542e05868, because it has exceeded maxfill, old size: 65536
LibClamAV debug: hashtab.c: new capacity: 131072
LibClamAV debug: Table 0x6e542e05868 size after grow: 131072
LibClamAV debug: hashtab.c:Growing hashtable 0x6e542e05868, because it has exceeded maxfill, old size: 131072
LibClamAV debug: hashtab.c: new capacity: 262144
LibClamAV debug: Table 0x6e542e05868 size after grow: 262144
WARNING: [LibClamAV] hm_addhash_str: invalid hash 99d3f86ca69d302b4d8�2bba2fff3104
WARNING: [LibClamAV] cli_loadhash: Malformed hash string at line 1644531
WARNING: [LibClamAV] cli_loadhash: Problem parsing database at line 1644531
WARNING: [LibClamAV] Can't load daily.hsb: Invalid argument passed to function
WARNING: [LibClamAV] cli_tgzload: Can't load daily.hsb
LibClamAV debug: in cli_tgzload_cleanup()
WARNING: [LibClamAV] Can't load /var/db/clamav/tmp.82d81c0ca0/clamav-bc4fdf2704b8987ff0051938567e0e15.tmp-daily.cld: Malformed database
ERROR: Failed to load new database: Malformed database
LibClamAV debug: Cleaning up phishcheck
LibClamAV debug: Freeing phishcheck struct
LibClamAV debug: Phishcheck cleaned up
WARNING: Stderr output from database load : LibClamAV debug: Initialized 1.1.2 engine [...] LibClamAV debug: Phishcheck cleaned up
WARNING: Database load exited with "Test failed"
ERROR: Database test FAILED.
updatedb: callback failed: Test failed (8)
ERROR: Unexpected error when attempting to update daily: Test failed
ERROR: Database update process failed: Test failed
ERROR: Update failed.
ragusaa commented 5 months ago

Hi, This appears to be fairly stable, so I am going to close the issue. Could you re-open if it happens again?

Thanks, Andy