search

jsuto / piler

Email archiving application
https://www.mailpiler.org/
Other
50 stars 8 forks source link

clamd service running but: "Can't open file or directory ERROR" #127

Closed vmario89 closed 1 month ago

vmario89 commented 1 month ago

Hi, i installed piler from recent master branch and compiled it with clamd support and enabled verbosity to 5 in config

./configure --prefix=/ --localstatedir=/var --enable-memcached --enable-clamd --with-database=mysql

However, in /var/log/mail.log piler-smtp says that clamd cannot open the files

12:05:00 ✔ root@piler:/var/log# grep "ERROR" mail.log 
Jul 31 11:06:41 piler piler[3986]: 2/PS5PMOM9LFZOUDZ3: CLAMD DEBUG: 83 /mnt/storagespace/piler/tmp/2/PS5PMOM9LFZOUDZ3: Can't open file or directory ERROR
Jul 31 11:06:41 piler piler[3986]: 2/DDG5CSFD032S2YYP: CLAMD DEBUG: 83 /mnt/storagespace/piler/tmp/2/DDG5CSFD032S2YYP: Can't open file or directory ERROR
Jul 31 11:07:14 piler piler[3985]: 1/762L7I4WSWW546D0: CLAMD DEBUG: 83 /mnt/storagespace/piler/tmp/1/762L7I4WSWW546D0: Can't open file or directory ERROR
Jul 31 11:11:33 piler piler[3985]: 1/ORGWSBQGSUIIKCP5: CLAMD DEBUG: 83 /mnt/storagespace/piler/tmp/1/ORGWSBQGSUIIKCP5: Can't open file or directory ERROR

piler-smtp is running fine without warnings or errors in journalctl.

piler service is configured to use the following setting in piler.conf: clamd_socket=/var/run/clamav/clamd.ctl

this socket is existing with following permissions:

srw-rw-rw-  1 clamav clamav    0 Jul 28 12:57 clamd.ctl=
journalctl -u clamav-daemon.service
Jul 31 11:06:41 piler clamd[658]: Wed Jul 31 11:06:41 2024 -> SelfCheck: Database status OK.
Jul 31 11:06:41 piler clamd[658]: Wed Jul 31 11:06:41 2024 -> ~/mnt/storagespace/piler/tmp/2/PS5PMOM9LFZOUDZ3: Can't open file or directory ERROR
Jul 31 11:06:41 piler clamd[658]: Wed Jul 31 11:06:41 2024 -> ~/mnt/storagespace/piler/tmp/2/DDG5CSFD032S2YYP: Can't open file or directory ERROR
Jul 31 11:07:14 piler clamd[658]: Wed Jul 31 11:07:14 2024 -> ~/mnt/storagespace/piler/tmp/1/762L7I4WSWW546D0: Can't open file or directory ERROR
Jul 31 11:11:33 piler clamd[658]: Wed Jul 31 11:11:33 2024 -> ~/mnt/storagespace/piler/tmp/1/ORGWSBQGSUIIKCP5: Can't open file or directory ERROR
Jul 31 11:57:43 piler clamd[658]: Wed Jul 31 11:57:43 2024 -> Reading databases from /var/lib/clamav
Jul 31 11:58:22 piler clamd[658]: Wed Jul 31 11:58:22 2024 -> Database correctly reloaded (8696571 signatures)
Jul 31 11:58:22 piler clamd[658]: Wed Jul 31 11:58:22 2024 -> Activating the newly loaded database...
Jul 31 12:00:09 piler clamd[658]: Wed Jul 31 12:00:09 2024 -> ~/mnt/storagespace/piler/tmp/1/FOPAJ1AEVKLZW2ME: Can't open file or directory ERROR
Jul 31 12:09:43 piler clamd[658]: Wed Jul 31 12:09:43 2024 -> SelfCheck: Database status OK.
Jul 31 12:09:43 piler clamd[658]: Wed Jul 31 12:09:43 2024 -> ~/mnt/storagespace/piler/tmp/2/VDK9A5E948O8234Q: Can't open file or directory ERROR
journalctl -u clamav-freshclam.service
Jul 31 11:57:43 piler freshclam[681]: Wed Jul 31 11:57:43 2024 -> daily.cld updated (version: 27353, sigs: 2065022, f-level: 90, builder: raynman)
Jul 31 11:57:43 piler freshclam[681]: Wed Jul 31 11:57:43 2024 -> main.cvd database is up-to-date (version: 62, sigs: 6647427, f-level: 90, builder: sigmgr)
Jul 31 11:57:43 piler freshclam[681]: Wed Jul 31 11:57:43 2024 -> bytecode.cvd database is up-to-date (version: 335, sigs: 86, f-level: 90, builder: raynman)
Jul 31 11:57:43 piler freshclam[681]: Wed Jul 31 11:57:43 2024 -> Clamd successfully notified about the update.

looks good so far, except the mentioned issue

Anyone with ideas what cause the issue? The permissions of tmp dir are

drwx--x--x  5 piler piler       4096 Jul 31 12:09 /var/piler/tmp/
#note: /var/piler is a symlink for /mnt/storagespace/piler/
#ls -alF in the dir shows zero byte files:
total 20
drwx--x--x  5 piler piler 4096 Jul 31 12:09 ./
drwxr-xr-x 10 piler piler 4096 Jul 28 13:41 ../
drwx--x--x  2 piler piler 4096 Jul 31 10:36 0/
drwx--x--x  2 piler piler 4096 Jul 31 12:00 1/
-rw-r-----  1 piler piler    0 Jul 28 12:40 1QUHVGPUDM7Q3DQQ
-rw-r-----  1 piler piler    0 Jul 26 09:52 1Y0JD7OJ5IS7CUAZ
drwx--x--x  2 piler piler 4096 Jul 31 12:09 2/
-rw-r-----  1 piler piler    0 Jul 26 10:01 45ARI3XF59LCXM5V
-rw-r-----  1 piler piler    0 Jul 28 11:10 9LI9LA3QHTTN11FZ
-rw-r-----  1 piler piler    0 Jul 26 10:10 9RNE2KJL59UW601J
-rw-r-----  1 piler piler    0 Jul 26 10:21 BZ8DO8WPQ9DPQZSY
-rw-r-----  1 piler piler    0 Jul 28 13:42 DFSJVJBHSTPUK5WW
-rw-r-----  1 piler piler    0 Jul 26 10:21 E4XTEEP3H3ALW96I
-rw-r-----  1 piler piler    0 Jul 26 09:25 F9520TT6T9MF88N6
-rw-r-----  1 piler piler    0 Jul 28 10:55 KDK1IM8EXIWF26MP
-rw-r-----  1 piler piler    0 Jul 28 11:15 LQ0QQEK3UUB50MJF
-rw-r-----  1 piler piler    0 Jul 28 12:40 M4XJ9W2HXVQZ1XDC
-rw-r-----  1 piler piler    0 Jul 28 12:21 M801OMD9SIDVOVA8
-rw-r-----  1 piler piler    0 Jul 28 12:21 NN0JIKOHYG0SY541
-rw-r-----  1 piler piler    0 Jul 26 09:32 QZ907X40JOPYNC6N
-rw-r-----  1 piler piler    0 Jul 28 10:42 RG7K4LO7B8G4JH4D
-rw-r-----  1 piler piler    0 Jul 26 10:15 Y9NRISDH3KFYUEEN
-rw-r-----  1 piler piler    0 Jul 28 10:42 YZMN2KOOZLEONPIR
jsuto commented 1 month ago

I think the problem is that /var/piler/tmp has 700 permissions, so clamd can't access the files. Those zero length files worry me.

vmario89 commented 1 month ago

Hi, i tested a lot with this yesterday and i was not able to fix it. i tried the following things:

set ACLs for clamav user (setfacl --recursive --modify u:clamav:rwX,d:u:clamav:rwX /var/piler/tmp/)

nothing worked for this

then i checked out that a lof of users have the same problem with clamd. they tried to run clamd as root then but this should not be done.

i found out that the command "clamdscan" successfully works when passing with parameter "--fdpass" or "--stream". So the following works: 01:37:58 ✘ root@piler:/var/run/clamav# clamdscan -fdpass /var/piler/tmp/

is there any chance to compile src/clamd.c with this --fdpass parameter? the piler.conf has only settings for the socket but not for any further flags to modify the antivirus check routine

jsuto commented 1 month ago

Piler doesn't use the clamdscan command. Instead it connects to clamd unix socket and passes the full path after the SCAN command.

I can see that you use a non-standard path for the piler files. Verify the permissions on the dirs so that clamav user can list the contents of /mnt/storagespace/piler/tmp/2. I think that the problem is that 0,1,2 dirs in tmp are created with 700 permissions. Try setting them to 711, and no need to put clamav user to piler's group.

vmario89 commented 1 month ago

i tried different places for tmp like /var/piler/tmp or /tmp/ or that external /mnt/storage/var/piler/tmp/ - nothing works

the permissions are fine and they also mus be fine, because piler itself re-created all the dirs like 0/ 1/ 2/ with its own permissions. i did not change. Thats, what they look like:

drwx--x--x  5 piler piler       4096 Aug  6 16:18 tmp/

16:19:05 ✔ root@piler:/mnt/storagespace/piler/tmp# ll
total 20
drwx--x--x  5 piler piler 4096 Aug  6 16:19 ./
drwxr-xr-x 10 piler piler 4096 Aug  1 01:48 ../
drwx--x--x  2 piler piler 4096 Aug  6 16:19 0/
drwx--x--x  2 piler piler 4096 Aug  6 16:19 1/
drwx--x--x  2 piler piler 4096 Aug  6 16:13 2/

16:19:16 ✔ root@piler:/mnt/storagespace/piler/tmp/0# ll
total 8
drwx--x--x 2 piler piler 4096 Aug  6 16:19 ./
drwx--x--x 5 piler piler 4096 Aug  6 16:19 ../
16:19:16 ✔ root@piler:/mnt/storagespace/piler/tmp/0#

i know that piler connects to clamd socket but i thought the command it would process over socket still would be clamdscan

what command will piler build up exactly with clamd to scan a file? 

[ClamDScan](https://docs.clamav.net/manual/Usage/Scanning.html#clamdscan)

clamdscan is a clamd client, which greatly simplifies the task of scanning files with clamd. It sends commands to the clamd daemon across the socket specified in clamd.conf and generates a scan report after all requested scanning has been completed by the daemon.

Thus, to run clamdscan, you must have an instance of clamd already running as well.

Please keep in mind, that as a simple scanning client, clamdscan cannot change scanning and engine configurations. These are tied to the clamd instance and the configuration you set up in clamd.conf. Therefore, while clamdscan will accept many of the same commands as its sister tool clamscan, it will simply ignore most of them as (by design) no mechanism exists to make ClamAV engine configuration changes over the clamd socket.

Again, running clamdscan, once you have a working clamd instance, is simple:

clamdscan [*options*] [*file/directory/-*]
vmario89 commented 1 month ago

interestingly clamdscan fails, clamscan is okay - but clamdscan is the one listening on the socket:

16:25:54 ✔ root@piler:/mnt/storagespace/piler/tmp/0# clamscan --verbose /var/piler/tmp/

----------- SCAN SUMMARY -----------
Known viruses: 8696769
Engine version: 0.103.11
Scanned directories: 1
Scanned files: 0
Infected files: 0
Data scanned: 0.00 MB
Data read: 0.00 MB (ratio 0.00:1)
Time: 39.426 sec (0 m 39 s)
Start Date: 2024:08:06 16:26:12
End Date:   2024:08:06 16:26:52
16:26:52 ✔ root@piler:/mnt/storagespace/piler/tmp/0# clamdscan --verbose /var/piler/tmp/
/mnt/storagespace/piler/tmp: File path check failure: Permission denied. ERROR

----------- SCAN SUMMARY -----------
Infected files: 0
Total errors: 1
Time: 0.001 sec (0 m 0 s)
Start Date: 2024:08:06 16:27:04
End Date:   2024:08:06 16:27:04

clamconf | grep -iE 'User|Socket'
LocalSocket = "/var/run/clamav/clamd.ctl"
LocalSocketGroup = "clamav"
LocalSocketMode = "666"
FixStaleSocket = "yes"
TCPSocket disabled
User = "clamav"
HTTPProxyUsername disabled
HTTPUserAgent disabled
jsuto commented 1 month ago

No, it doesn't use clamdscan, so it's no use when you try it from the command line.

Piler sends the 'SCAN /path/to/file' command. It even syslogs it at debug verbosity. If 0/ 1/ and 2/ dirs have 711 permissions, then it should work.

vmario89 commented 1 month ago

okay found some other documentation, which mentions that SCAN command with is used in pilers source code: https://linux.die.net/man/8/clamd

i still tried that manually to check what happens:

fails, when clamav daemon is running: clamd SCAN /mnt/storagespace/piler/www/<SOMEFILE>

ERROR: /var/log/clamav/clamav.log is locked by another process
ERROR: Can't initialize the internal logger

when stopping clamav and trying again ...

systemctl stop clamav-daemon.service
systemctl disable clamav-daemon.service

.... it works: clamd SCAN /mnt/storagespace/piler/tmp/<SOMEFILE>

check the logs:

less /var/log/clamav/clamav.log

Aug  6 16:44:40 piler piler[202127]: CLAMD ERR: connect to /var/run/clamav/clamd.ctl
Aug  6 16:44:40 piler piler[202127]: 1/LE0QORJBNWOJA3HH: done virus scanning

but when i disable the service, piler says it cannot connect to socket. if i leave clamav enabled, the error occures.

so the thing is: performing a scan manually with disabled service, it works; why piler cannot do?

jsuto commented 1 month ago

Give me some time until I setup a VM, and deploy piler with clamav. I'll keep you posted.

jsuto commented 1 month ago

I've deployed piler with clamd support, and run into the same problem. I tweaked the directory and file permissions, but no matter what I applied, clamd produced an access error problem. Yet unsure why.

jsuto commented 1 month ago

I compiled a test program to pass a file's path to the clamd_scan() function, and it seems that it's not the permissions or the ownership of the file that matters, rather the path. I think the clamd.conf settings prohibit to anything not in $HOME or in /tmp. I'll open an issue in the clamav project to confirm this, and hopefully provide a fix.

jsuto commented 1 month ago

The reported issue: https://github.com/Cisco-Talos/clamav/issues/1328 Update: no response from the clamav developers after a week. I suggest to not use clamav with piler, instead do the malware scanning on your MX servers.

vmario89 commented 3 weeks ago

you closed the issue and commited https://github.com/jsuto/piler/commit/8cf96dce08e6eee6157142804a7879a3aece9eb9 - so i guess you are going to completely remove clamav Support? If yes, we need to also remove statements in https://github.com/jsuto/piler/blob/master/configure.in

jsuto commented 3 weeks ago

Well, I'm somewhat hesitant to do that, but it might be the best course of action. Give me some time to think about it.