search

Cisco-Talos / clamav

ClamAV - Documentation is here: https://docs.clamav.net
https://www.clamav.net/
GNU General Public License v2.0
4.16k stars 682 forks source link

Freshclam in docker container never downloads cdiff files #1125

Closed tdring closed 7 months ago

tdring commented 7 months ago

Describe the bug

Virus definitions never updated.

During initial container creation, and expecting multiple iterations, i prepolulated the volume with the current database files and work as expected.

On review after 2 weeks of running, and running a container restart, the warning of 'older than 7 days' is presented and there is no update to the main files, on investigation by stopping and manually running freshclam in debug verbose mode the following is output

_Current working dir is /var/lib/clamav/ Loaded freshclam.dat: version: 1 uuid: ed33cd32-50f4-4f27-a68d-695cc481b8e8 ClamAV update process started at Mon Jan 1 14:16:23 2024 Current working dir is /var/lib/clamav/ Querying current.cvd.clamav.net TTL: 627 fc_dns_query_update_info: Software version from DNS: 0.103.11 Current working dir is /var/lib/clamav/ check_for_new_database_version: Local copy of daily found: daily.cvd. query_remote_database_version: daily.cvd version from DNS: 27141 daily database available for update (local version: 27128, remote version: 27141) Current database is 13 versions behind. Downloading database patch # 27129... LibClamAV debug: MD5(.tar.gz) = 179bdb3f940495b66fcdafc4ce7000e1 LibClamAV debug: cli_versig: Decoded signature: 179bdb3f940495b66fcdafc4ce7000e1 LibClamAV debug: cli_versig: Digital signature is correct. LibClamAV debug: in cli_untgz() LibClamAV debug: cli_untgz: Unpacking /var/lib/clamav/tmp.41040798e8/clamav-3e9cc3deb7e7f7c23ed3bf5be5be5ad3.tmp/COPYING LibClamAV debug: cli_untgz: Unpacking /var/lib/clamav/tmp.41040798e8/clamav-3e9cc3deb7e7f7c23ed3bf5be5be5ad3.tmp/daily.info LibClamAV debug: cli_untgz: Unpacking /var/lib/clamav/tmp.41040798e8/clamav-3e9cc3deb7e7f7c23ed3bf5be5be5ad3.tmp/daily.cfg LibClamAV debug: cli_untgz: Unpacking /var/lib/clamav/tmp.41040798e8/clamav-3e9cc3deb7e7f7c23ed3bf5be5be5ad3.tmp/daily.ign LibClamAV debug: cli_untgz: Unpacking /var/lib/clamav/tmp.41040798e8/clamav-3e9cc3deb7e7f7c23ed3bf5be5be5ad3.tmp/daily.ign2 LibClamAV debug: cli_untgz: Unpacking /var/lib/clamav/tmp.41040798e8/clamav-3e9cc3deb7e7f7c23ed3bf5be5be5ad3.tmp/daily.ftm LibClamAV debug: cli_untgz: Unpacking /var/lib/clamav/tmp.41040798e8/clamav-3e9cc3deb7e7f7c23ed3bf5be5be5ad3.tmp/daily.hdb LibClamAV debug: cli_untgz: Unpacking /var/lib/clamav/tmp.41040798e8/clamav-3e9cc3deb7e7f7c23ed3bf5be5be5ad3.tmp/daily.hdu LibClamAV debug: cli_untgz: Unpacking /var/lib/clamav/tmp.41040798e8/clamav-3e9cc3deb7e7f7c23ed3bf5be5be5ad3.tmp/daily.hsb LibClamAV debug: cli_untgz: Unpacking /var/lib/clamav/tmp.41040798e8/clamav-3e9cc3deb7e7f7c23ed3bf5be5be5ad3.tmp/daily.hsu LibClamAV debug: cli_untgz: Unpacking /var/lib/clamav/tmp.41040798e8/clamav-3e9cc3deb7e7f7c23ed3bf5be5be5ad3.tmp/daily.mdb LibClamAV debug: cli_untgz: Unpacking /var/lib/clamav/tmp.41040798e8/clamav-3e9cc3deb7e7f7c23ed3bf5be5be5ad3.tmp/daily.mdu LibClamAV debug: cli_untgz: Unpacking /var/lib/clamav/tmp.41040798e8/clamav-3e9cc3deb7e7f7c23ed3bf5be5be5ad3.tmp/daily.msb LibClamAV debug: cli_untgz: Unpacking /var/lib/clamav/tmp.41040798e8/clamav-3e9cc3deb7e7f7c23ed3bf5be5be5ad3.tmp/daily.msu LibClamAV debug: cli_untgz: Unpacking /var/lib/clamav/tmp.41040798e8/clamav-3e9cc3deb7e7f7c23ed3bf5be5be5ad3.tmp/daily.ndb LibClamAV debug: cli_untgz: Unpacking /var/lib/clamav/tmp.41040798e8/clamav-3e9cc3deb7e7f7c23ed3bf5be5be5ad3.tmp/daily.ndu LibClamAV debug: cli_untgz: Unpacking /var/lib/clamav/tmp.41040798e8/clamav-3e9cc3deb7e7f7c23ed3bf5be5be5ad3.tmp/daily.ldb LibClamAV debug: cli_untgz: Unpacking /var/lib/clamav/tmp.41040798e8/clamav-3e9cc3deb7e7f7c23ed3bf5be5be5ad3.tmp/daily.ldu LibClamAV debug: cli_untgz: Unpacking /var/lib/clamav/tmp.41040798e8/clamav-3e9cc3deb7e7f7c23ed3bf5be5be5ad3.tmp/daily.idb LibClamAV debug: cli_untgz: Unpacking /var/lib/clamav/tmp.41040798e8/clamav-3e9cc3deb7e7f7c23ed3bf5be5be5ad3.tmp/daily.fp LibClamAV debug: cli_untgz: Unpacking /var/lib/clamav/tmp.41040798e8/clamav-3e9cc3deb7e7f7c23ed3bf5be5be5ad3.tmp/daily.sfp LibClamAV debug: cli_untgz: Unpacking /var/lib/clamav/tmp.41040798e8/clamav-3e9cc3deb7e7f7c23ed3bf5be5be5ad3.tmp/daily.pdb LibClamAV debug: cli_untgz: Unpacking /var/lib/clamav/tmp.41040798e8/clamav-3e9cc3deb7e7f7c23ed3bf5be5be5ad3.tmp/daily.wdb LibClamAV debug: cli_untgz: Unpacking /var/lib/clamav/tmp.41040798e8/clamav-3e9cc3deb7e7f7c23ed3bf5be5be5ad3.tmp/daily.crb LibClamAV debug: cli_untgz: Unpacking /var/lib/clamav/tmp.41040798e8/clamav-3e9cc3deb7e7f7c23ed3bf5be5be5ad3.tmp/daily.cdb LibClamAV debug: in cli_untgzcleanup() Retrieving https://database.clamav.net/daily-27129.cdiff downloadFile: Download source: https://database.clamav.net/daily-27129.cdiff downloadFile: Download destination: ./clamav-772c19fefa793defeff104ddf79b768d.tmp

And here it will stay forever with no files downloaded or updated

How to reproduce the problem

Run the latest (stable or base) image in docker (running on synology DSM at the moment)

Config file: clamd.conf

LogFile = "/var/log/clamav/clamd.log" LogTime = "yes" PidFile = "/tmp/clamd.pid" LocalSocket = "/tmp/clamd.sock" TCPSocket = "3310" User = "clamav"

Config file: freshclam.conf

PidFile = "/tmp/freshclam.pid" UpdateLogFile = "/var/log/clamav/freshclam.log" DatabaseMirror = "database.clamav.net" MaxAttempts = "5" ConnectTimeout = "60" ReceiveTimeout = "300"

Config file: clamav-milter.conf

LogFile = "/var/log/clamav/milter.log" LogTime = "yes" PidFile = "/tmp/clamav-milter.pid" User = "clamav" ClamdSocket = "unix:/tmp/clamd.sock", "unix:/tmp/clamd.sock", "unix:/tmp/clamd.sock", "unix:/tmp/clamd.sock", "unix:/tmp/clamd.sock", "unix:/tmp/clamd.sock" MilterSocket = "inet:7357"

Software settings

Version: 1.2.1 Optional features supported: MEMPOOL AUTOIT_EA06 BZIP2 LIBXML2 PCRE2 ICONV JSON RAR

Database information

Database directory: /var/lib/clamav daily.cvd: version 27128, sigs: 2049126, built on Tue Dec 19 09:36:48 2023 main.cvd: version 62, sigs: 6647427, built on Thu Sep 16 12:32:42 2021 bytecode.cvd: version 334, sigs: 91, built on Wed Feb 22 21:33:21 2023 Total number of signatures: 8696644

Platform information

uname: Linux 3.10.105 #25426 SMP Tue May 12 04:40:16 CST 2020 x86_64 OS: Linux, ARCH: x86_64, CPU: x86_64 zlib version: 1.3 (1.3), compile flags: a9 platform id: 0x0a21bfbf08000000000d0201

Build information

GNU C: 13.2.1 20231014 (13.2.1) sizeof(void*) = 8 Engine flevel: 191, dconf: 191

Attachments

None

micahsnyder commented 7 months ago

It sounds like the freshclam service is not running in your container. The docker image will run it by default when you start the container, though you can disable it with the CLAMAV_NO_FRESHCLAMD environment variable. See: https://docs.clamav.net/manual/Installing/Docker.html#controlling-the-container

On my own system, I tried started the clamav/clamav:latest image, which at present bundles a database about 25 days out of date. On initial startup, the freshclam process run and updated it using the cdiff database patching process which you can see at the top of the log:


❯ docker run clamav/clamav:latest
Starting Freshclamd
Starting ClamAV
Socket for clamd not found yet, retrying (0/1800) ...ClamAV update process started at Thu Jan  4 18:04:19 2024
daily database available for update (local version: 27119, remote version: 27144)
LibClamAV Warning: **************************************************
LibClamAV Warning: ***  The virus database is older than 7 days!  ***
LibClamAV Warning: ***   Please update it as soon as possible.    ***
LibClamAV Warning: **************************************************
Socket for clamd not found yet, retrying (4/1800) ...Testing database: '/var/lib/clamav/tmp.bb6b2c0abd/clamav-71521859fe326e5a0570be1c909d33b4.tmp-daily.cld' ...
Socket for clamd not found yet, retrying (10/1800) ...Database test passed.
daily.cld updated (version: 27144, sigs: 2050280, f-level: 90, builder: raynman)
main.cvd database is up-to-date (version: 62, sigs: 6647427, f-level: 90, builder: sigmgr)
bytecode.cvd database is up-to-date (version: 334, sigs: 91, f-level: 90, builder: anvilleg)
WARNING: Clamd was NOT notified: Can't connect to clamd through /tmp/clamd.sock: No such file or directory
Socket for clamd not found yet, retrying (14/1800) ...Thu Jan  4 18:04:33 2024 -> Limits: Global time limit set to 120000 milliseconds.
Thu Jan  4 18:04:33 2024 -> Limits: Global size limit set to 419430400 bytes.
Thu Jan  4 18:04:33 2024 -> Limits: File size limit set to 104857600 bytes.
Thu Jan  4 18:04:33 2024 -> Limits: Recursion level limit set to 17.
Thu Jan  4 18:04:33 2024 -> Limits: Files limit set to 10000.
Thu Jan  4 18:04:33 2024 -> Limits: MaxEmbeddedPE limit set to 41943040 bytes.
Thu Jan  4 18:04:33 2024 -> Limits: MaxHTMLNormalize limit set to 41943040 bytes.
Thu Jan  4 18:04:33 2024 -> Limits: MaxHTMLNoTags limit set to 8388608 bytes.
Thu Jan  4 18:04:33 2024 -> Limits: MaxScriptNormalize limit set to 20971520 bytes.
Thu Jan  4 18:04:33 2024 -> Limits: MaxZipTypeRcg limit set to 1048576 bytes.
Thu Jan  4 18:04:33 2024 -> Limits: MaxPartitions limit set to 50.
Thu Jan  4 18:04:33 2024 -> Limits: MaxIconsPE limit set to 100.
Thu Jan  4 18:04:33 2024 -> Limits: MaxRecHWP3 limit set to 16.
Thu Jan  4 18:04:33 2024 -> Limits: PCREMatchLimit limit set to 100000.
Thu Jan  4 18:04:33 2024 -> Limits: PCRERecMatchLimit limit set to 2000.
Thu Jan  4 18:04:33 2024 -> Limits: PCREMaxFileSize limit set to 104857600.
Thu Jan  4 18:04:33 2024 -> Archive support enabled.
Thu Jan  4 18:04:33 2024 -> AlertExceedsMax heuristic detection disabled.
Thu Jan  4 18:04:33 2024 -> Heuristic alerts enabled.
Thu Jan  4 18:04:33 2024 -> Portable Executable support enabled.
Thu Jan  4 18:04:33 2024 -> ELF support enabled.
Thu Jan  4 18:04:33 2024 -> Mail files support enabled.
Thu Jan  4 18:04:33 2024 -> OLE2 support enabled.
Thu Jan  4 18:04:33 2024 -> PDF support enabled.
Thu Jan  4 18:04:33 2024 -> SWF support enabled.
Thu Jan  4 18:04:33 2024 -> HTML support enabled.
Thu Jan  4 18:04:33 2024 -> XMLDOCS support enabled.
Thu Jan  4 18:04:33 2024 -> HWP3 support enabled.
Thu Jan  4 18:04:33 2024 -> Self checking every 600 seconds.
Thu Jan  4 18:04:33 2024 -> Set stacksize to 1048576
socket found, clamd started.

I was concerned if the freshclam database update process would be working right for me, since you're having issues, so I hopped into the running container and checked. It appears to be running, and would presumably do another update in about 24 hours, since it's using --checks=1:

❯ docker ps
CONTAINER ID   IMAGE                  COMMAND   CREATED          STATUS                             PORTS                NAMES
0666ef5aa54e   clamav/clamav:latest   "/init"   28 seconds ago   Up 28 seconds (health: starting)   3310/tcp, 7357/tcp   amazing_villani

~ via 🐍 v3.8.10
❯ docker exec -it amazing_villani /bin/ash
/ # ps ef | grep clam
   12 clamav    0:01 freshclam --checks=1 --daemon --foreground --stdout --user=clamav
   13 clamav    0:14 clamd --foreground
   47 root      0:00 grep clam
/ #

Your freshclam.conf settings look a little different than mine, but I don't see anything particularly problematic, unless you have a very slow internet connection. Mine is:

Config file: freshclam.conf
---------------------------
PidFile = "/tmp/freshclam.pid"
UpdateLogFile = "/var/log/clamav/freshclam.log"
DatabaseMirror = "database.clamav.net"

This extra setting in yours may cause problems if you have a slower internet connection:

ReceiveTimeout = "300"

I see that your UpdateLogFile option is set, so I think the next steps are:

  1. check if freshclam is running, like I did.
  2. inspect /var/log/clamav/freshclam.log to see if there are any error messages that would explain why it is failing to update.

Hope this helps, Micah

tdring commented 7 months ago

Hi Micah,

Unfortunately this was my first thought, but freshclam is running, but never actually downloads anything either on inital startup, if (re)started manually within the container or while running normally... it just sits there as described, indicating it is downloading but never finishing.

I have attempted to run in debug and verbose but it never displays any error message or anything helpful and the log file has the same info as presented on the cli

Regards

micahsnyder commented 7 months ago

Is it possible that you're running out of RAM on in the container, or running out of disk space?

tdring commented 7 months ago

Not short on diskspace there is plenty in there, And still a fair amount of RAM available as far as I can see.

However using the same images and setup on another (virtual) host it all works ok, pointing more to issues with the current setup being environment specific so in fairness out of the scope for here.

tdring commented 7 months ago

On further investigation, and testing this is a bug in curl and older kernel images (synology is fairly behind) specifically 3.x check https://info.linuxserver.io/issues/2023-12-30-synology/ for more info I was able to fix this in a current container with apk add c-ares -u --repository=https://dl-cdn.alpinelinux.org/alpine/edge/main but a longerterm fix will be to update my core systems