Cisco-Talos / clamav

ClamAV - Documentation is here: https://docs.clamav.net
https://www.clamav.net/
GNU General Public License v2.0
4.39k stars 705 forks source link

Socket for clamd not found yet! Failed to start clamd #1326

Closed kamran1860 closed 1 month ago

kamran1860 commented 3 months ago

Describe the bug

I am trying to upgrade from old clamav version 0.104.3 to 1.3.1 inside k8 cluster and facing following problem:

Socket for clamd not found yet! Failed to start clamd

Due to this problem, Pod restarted every 30 min. I did already went through issues #757, #760 and #957 but still can't solve this problem. Can you please help on solving of this issue? Thanks in advance!

How to reproduce the problem

My Docker File: FROM clamav/clamav:1.3 ENTRYPOINT ["/init-unprivileged"]

Configuration (result of clamconf -n):

Checking configuration files in /etc/clamav

Config file: clamd.conf
-----------------------
LogFile = "/var/log/clamav/clamd.log"
LogTime = "yes"
LogClean = "yes"
LogVerbose = "yes"
ExtendedDetectionInfo = "yes"
PidFile = "/run/clamav/clamd.pid"
FailIfCvdOlderThan = "7"
LocalSocket = "/run/clamav/clamd.sock"
LocalSocketGroup = "clamav"
LocalSocketMode = "660"
TCPSocket = "3310"
SelfCheck = "3600"
ConcurrentDatabaseReload disabled
Foreground = "yes"
GenerateMetadataJson = "yes"
User = "clamav"

Config file: freshclam.conf
---------------------------
LogTime = "yes"
LogVerbose = "yes"
PidFile = "/run/clamav/freshclam.pid"
UpdateLogFile = "/var/log/clamav/freshclam.log"
Checks = "1"
DatabaseMirror = "database.clamav.net"
TestDatabases disabled
HTTPProxyServer = "my-proxy"
HTTPProxyPort = "3128"

Config file: clamav-milter.conf
-------------------------------
LogFile = "/var/log/clamav/milter.log"
LogTime = "yes"
PidFile = "/tmp/clamav-milter.pid"
User = "clamav"
ClamdSocket = "unix:/tmp/clamd.sock", "unix:/tmp/clamd.sock", "unix:/tmp/clamd.sock", "unix:/tmp/clamd.sock", "unix:/tmp/clamd.sock", "unix:/tmp/clamd.sock"
MilterSocket = "inet:7357"

Software settings
-----------------
Version: 1.3.1
Optional features supported: MEMPOOL AUTOIT_EA06 BZIP2 LIBXML2 PCRE2 ICONV JSON RAR 

Database information
--------------------
Database directory: /var/lib/clamav
main.cvd: version 62, sigs: 6647427, built on Thu Sep 16 12:32:42 2021
bytecode.cvd: version 335, sigs: 86, built on Tue Feb 27 15:37:24 2024
daily.cld: version 27357, sigs: 2065104, built on Sun Aug  4 08:34:19 2024
Total number of signatures: 8712617

Platform information
--------------------
uname: Linux 5.15.0-107-generic #117~20.04.1-Ubuntu SMP Tue Apr 30 10:35:57 UTC 2024 x86_64
OS: Linux, ARCH: x86_64, CPU: x86_64
zlib version: 1.3.1 (1.3.1), compile flags: a9
platform id: 0x0a21c9c908000000000d0201

Build information
-----------------
GNU C: 13.2.1 20240309 (13.2.1)
sizeof(void*) = 8
Engine flevel: 201, dconf: 201

And Short Logging (See attachment for full log)

Starting Freshclamd
Starting ClamAV

Socket for clamd not found yet, retrying (0/1800) ...Connecting via my-proxy
Mon Aug  5 11:13:44 2024 -> Current working dir is /var/lib/clamav/
Mon Aug  5 11:13:44 2024 -> Loaded freshclam.dat:
Mon Aug  5 11:13:44 2024 ->   version:    1
Mon Aug  5 11:13:44 2024 ->   uuid:       63429af9-21b6-49a0-abdb-4ee176841804
Mon Aug  5 11:13:44 2024 -> ClamAV update process started at Mon Aug  5 11:13:44 2024
Mon Aug  5 11:13:44 2024 -> Current working dir is /var/lib/clamav/
Mon Aug  5 11:13:44 2024 -> Querying current.cvd.clamav.net
WARNING: Mon Aug  5 11:13:44 2024 -> Can't query current.cvd.clamav.net
WARNING: Mon Aug  5 11:13:44 2024 -> Invalid DNS reply. Falling back to HTTP mode.
Mon Aug  5 11:13:44 2024 -> Current working dir is /var/lib/clamav/
Mon Aug  5 11:13:44 2024 -> check_for_new_database_version: Local copy of daily found: daily.cld.
Mon Aug  5 11:13:44 2024 -> Trying to retrieve CVD header from https://database.clamav.net/daily.cvd
Mon Aug  5 11:13:44 2024 -> Using proxy: my-proxy:3128

Socket for clamd not found yet, retrying (1/1800) ...
Socket for clamd not found yet, retrying (2/1800) ...* Host my-proxy:3128 was resolved.
* IPv6: (none)
* IPv4: xx.xx.xx.xx
*   Trying xx.xx.xx.xx:3128...
* Connected to my-proxy (xx.xx.xx.xx) port 3128
* CONNECT tunnel: HTTP/1.1 negotiated
* allocate connect buffer
* Establish HTTP proxy tunnel to database.clamav.net:443
> CONNECT database.clamav.net:443 HTTP/1.1
Host: database.clamav.net:443
User-Agent: ClamAV/1.3.1 (OS: Linux, ARCH: x86_64, CPU: x86_64, UUID: 63429af9-21b6-49a0-abdb-4ee176841804)
Proxy-Connection: Keep-Alive

< HTTP/1.1 200 Connection established
...

Attachments

clamconf.log clamav.log

pfallasro commented 1 month ago

I'm also facing this issue, I can run the container fine locally but it won't run in my EKS cluster.

Are there any network restrictions for kubernetes environments?

micahsnyder commented 1 month ago

@pfallasro @kamran1860 Could it be that you aren't assigning enough RAM for your pods? I see some clamd startup log messages but I don't see any error message from clamd in the log.

I see in the original report that ConcurrentDatabaseReload and TestDatabases are off. That does reduce the amount of RAM needed, though the requirement is still fairly high.

micahsnyder commented 1 month ago

Side note: I see in the original report that GenerateMetadataJson is enabled. I'm curious why this is.

kamran1860 commented 1 month ago

Hi micahsnyder, thanks for your reply! regarding Ram usage, i use 2Gi of memory for starting of pod with limit of 4Gi in worse case. Due to the specification of clamav, it must be enough and i verified it also using kubectl top pod and can see that memory usage with grow up to 1465Mi and doesn't change during this 30 minutes. So it seems not to be the reason for this problem. Regarding GenerateMetadataJson, we activate it due to the requirements of logging/monitoring concept. The main problem seems to be "Socket for clamd not found yet, retrying (1800/1800) ... " which i don't understand! Can you please give some more detail about this error and what may be the reason? Thanks in advance!

micahsnyder commented 1 month ago

"Socket for clamd not found yet, retrying (1800/1800) ...

When clamd finishes loading it opens up a socket to listen for scan requests. The message indicates that clamd never did this.

Looking at your config closer, I think I see the issue. You have configured both the LocalSocket and TCPSocket. The docker image is set up to use the TCP socket and I bet the LocalSocket from your config is used instead:

LocalSocket = "/run/clamav/clamd.sock"
LocalSocketGroup = "clamav"
LocalSocketMode = "660"
TCPSocket = "3310"

Try removing the LocalSocket options from your config and try again.

Regarding GenerateMetadataJson, we activate it due to the requirements of logging/monitoring concept.

This still doesn't make much sense to me. GenerateMetadataJson is probably poorly named. It does not result in any sort of JSON output for scan results. It records metadata about the file being scanned and stores it in a JSON structure that may be dumped to disk for analysts to inspect the structure of the file. It is mostly for researchers and developers. As far as I am aware, it is not useful production scanning environments with the standard signature set.

micahsnyder commented 1 month ago

I take it back. It does need the LocalSocket option but it needs it to be /tmp/clamd.sock, and not /run/clamav/clamd.sock.

The script which outputs that message has this logic:

    if [ "${CLAMAV_NO_CLAMD:-false}" != "true" ]; then
        echo "Starting ClamAV"
        if [ -S "/tmp/clamd.sock" ]; then
            unlink "/tmp/clamd.sock"
        fi
        clamd --foreground &
        while [ ! -S "/tmp/clamd.sock" ]; do
            if [ "${_timeout:=0}" -gt "${CLAMD_STARTUP_TIMEOUT:=1800}" ]; then
                echo
                echo "Failed to start clamd"
                exit 1
            fi
            printf "\r%s" "Socket for clamd not found yet, retrying (${_timeout}/${CLAMD_STARTUP_TIMEOUT}) ..."
            sleep 1
            _timeout="$((_timeout + 1))"
        done
        echo "socket found, clamd started."
    fi

So just edit your clamd.conf and set: LocalSocket /tmp/clamd.sock

kamran1860 commented 1 month ago

Thanks a lot! that helped and now it works properly. just a question regarding json logging. is there any possibility to generate clamav logs in json format?

micahsnyder commented 1 month ago

I'm glad to hear it's working for you now. Sorry it took a bit for me to respond and for us to figure this out.

I'd love to add an option to output scan results in JSON, and include additional metadata about the detection. Sadly, it's not something we're planning to work on and we're thin on resources at present.