goshansp commented 3 years ago

issue

clamonacc from packaged rpm 103.2 up to 104/dev under load (i.e. when beeing deployed by ansible) during start fails to OnAccessIncludePath existing directory claiming it does not exist. this failure is not detected as the process doesn't terminate. this bug is possibly causing clamav on-access to not perform under any Linux for < 5.1 kernel.

ClamInotif: could not watch path - no such file or directory

next steps / brainstorming / todo

[x] blocked by or related to clamonacc fatal error https://github.com/Cisco-Talos/clamav/issues/184
[ ] reduce watchpoints to /usr /home /tmp
[ ] start clamonacc without systemd
[ ] compile dev/0.104 on centos8 and explore reproduction (fio?)
[ ] reproduction steps

proposed fixes to clamonacc

terminate clamonacc upon ERROR: ClamInotif: could not watch path '/home', No such file or directory
???

observations

manual systemctl restart clamav-clamonacc.service will succeed 100% watch /home (cannot reproduce)
molecule verify will only work ~ 2 out of 11 times when /usr is watched
said test succeeds under fedora33+4, fails centos7+8 (most likely because kernel <5.1 using inotify)
if /usr is not watched we pass eicar test reliably (much less fileio)
setenforce 0 # no effect
ulimit -n # 100000
happens with --stream and --fdpass
max_user_watches = 500k

symptomes

Jun 29 20:08:24 centos7 clamonacc: ERROR: ClamInotif: could not watch path '/home', No such file or directory
Jun 29 23:18:55 centos8 clamonacc[14434]: ERROR: ClamInotif: could not watch path '/home', No such file or directory
Jun 30 10:40:47 centos8 clamonacc[23644]: ERROR: ClamInotif: could not watch path '/tmp', No such file or directory

/etc/clamd.d/clamd.conf

LocalSocket /run/clamd.scan/clamd.sock

TemporaryDirectory /tmp/clamav

OnAccessExcludeUname clamscan
OnAccessExtraScanning yes

OnAccessIncludePath /boot
OnAccessIncludePath /etc
OnAccessIncludePath /home
OnAccessIncludePath /media
OnAccessIncludePath /mnt
OnAccessIncludePath /opt
OnAccessIncludePath /root
OnAccessIncludePath /tmp
OnAccessIncludePath /usr
OnAccessIncludePath /var

# onaccess_exclude_default_paths
OnAccessExcludePath /var/lib/rsyslog
OnAccessExcludePath /var/lib/clamav-unofficial-sigs
OnAccessExcludePath /var/log
OnAccessExcludePath /var/spool/quarantine

/etc/systemd/system/clamav-clamonacc.service

[Unit]
Description = ClamAV On-Access Notifier
After = clamav-clamd.service syslog.target network.target
Requires = clamav-clamd.service

[Service]
Type = simple
ExecStart = /usr/sbin/clamonacc -F --config-file=/etc/clamd.d/clamd.conf --move=/var/spool/quarantine --fdpass --verbose

# workaround for https://gitlab.com/goshansp/clamav_onaccess/-/issues/5
ExecStop = /bin/kill -s SIGKILL $MAINPID
SuccessExitStatus = SIGKILL SIGTERM

[Install]
WantedBy = multi-user.target

Please let me know if there is anything to be tested.

goshansp commented 3 years ago

this issue does persist after fixing #184 but no reproduction steps have been found. it happens rarely on local infra and is more frequent on shared (cloud) systems. it's more frequent on small 1-cpu systems and can me mitigated by not including /var. on larger systems with plenty of cpu it seems to become less common. also on a idle system it happens rarely but after hammering it happens again more frequent. a major mitigation step is adding ExecStartPre=/bin/sleep 32 or higher to clamonacc.service ... any hints pointing to reproduction on preventing clamonacc to initialize all includes appreciated!

frank-fegert commented 2 years ago

@goshansp I've had this issue too, but since i've changed the systemd unit file for clamonacc to

Type=forking

and removed the -F or --foreground flag from the ExecStart line the issue hasn't occured anymore.

Maybe the cause is a bad interaction between clamonacc and systemd and the way the latter handles the standard FDs?

Babber commented 2 years ago

@frank-fegert, I tried your suggestion, but unfortunately I still experience the same issue with v0.105.0. After every boot, it drops my /tmp folder with ERROR: ClamInotif: could not watch path '/tmp', No such file or directory

Cisco-Talos / clamav

clamonacc / ClamInotif: could not watch path #186