1.0.7 runs out of memory (Ubuntu 24.04.1 LTS)

[MarkyMarkDE]

ERROR: Can't allocate memory

----------- SCAN SUMMARY ----------- Known viruses: 2051532 Engine version: 1.0.7 Scanned directories: 0 Scanned files: 0 Infected files: 0 Data scanned: 0.00 MB Data read: 0.00 MB (ratio 0.00:1) Time: 6.301 sec (0 m 6 s) Start Date: 2024:09:23 03:44:01 End Date: 2024:09:23 03:44:07

with this Bug, clamscan / clamd din't work. Please fix it soon.

same Error for clamd

[micahsnyder]

@MarkyMarkDE I can't reproduce the issue with just running clamscan from 1.0.7 with latest signatures.

I should note that when I run clamscan, I see 8701623 "known viruses":

❯ ~/clams/1.0.7/bin/clamscan -d ~/.cvdupdate/database BLAH
Loading:    19s, ETA:   0s [========================>]    8.70M/8.70M sigs
Compiling:   2s, ETA:   0s [========================>]       41/41 tasks

BLAH: No such file or directory
WARNING: BLAH: Can't access file

----------- SCAN SUMMARY -----------
Known viruses: 8701623
Engine version: 1.0.7
Scanned directories: 0
Scanned files: 0
Infected files: 0
Data scanned: 0.00 MB
Data read: 0.00 MB (ratio 0.00:1)
Time: 22.334 sec (0 m 22 s)
Start Date: 2024:09:23 11:35:55
End Date:   2024:09:23 11:36:18

If you used any special arguments for clamscan, or are using any third party signature databases please add more details.

Without any more clues, I'll have to close this as "can't reproduce".

[MarkyMarkDE]

@MarkyMarkDE I can't reproduce the issue with just running clamscan from 1.0.7 with latest signatures.

I should note that when I run clamscan, I see 8701623 "known viruses":

❯ ~/clams/1.0.7/bin/clamscan -d ~/.cvdupdate/database BLAH
Loading:    19s, ETA:   0s [========================>]    8.70M/8.70M sigs
Compiling:   2s, ETA:   0s [========================>]       41/41 tasks

BLAH: No such file or directory
WARNING: BLAH: Can't access file

----------- SCAN SUMMARY -----------
Known viruses: 8701623
Engine version: 1.0.7
Scanned directories: 0
Scanned files: 0
Infected files: 0
Data scanned: 0.00 MB
Data read: 0.00 MB (ratio 0.00:1)
Time: 22.334 sec (0 m 22 s)
Start Date: 2024:09:23 11:35:55
End Date:   2024:09:23 11:36:18

If you used any special arguments for clamscan, or are using any third party signature databases please add more details.

Without any more clues, I'll have to close this as "can't reproduce".

have a look at this, please, hope this help:

$ clamconf -n Checking configuration files in /etc/clamav

clamd.conf not found

Config file: freshclam.conf

LogFileMaxSize = "4294967295" LogTime = "yes" UpdateLogFile = "/var/log/clamav/freshclam.log" Checks = "24" DatabaseMirror = "database.clamav.net" MaxAttempts = "12" ConnectTimeout = "60" ReceiveTimeout = "300"

clamav-milter.conf not found

Software settings

Version: 1.0.7 Optional features supported: MEMPOOL AUTOIT_EA06 BZIP2 LIBXML2 PCRE2 ICONV JSON

Database information

Database directory: /var/lib/clamav WARNING: freshclam.conf and clamd.conf point to different database directories [3rd Party] whitelist.wdb: 7 sigs bytecode.cvd: version 335, sigs: 86, built on Tue Feb 27 16:37:24 2024 main.cvd: version 62, sigs: 6647427, built on Thu Sep 16 14:32:42 2021 daily.cld: version 27407, sigs: 2066994, built on Mon Sep 23 10:31:24 2024 Total number of signatures: 8714514

Platform information

uname: Linux 6.8.0-45-generic #45-Ubuntu SMP PREEMPT_DYNAMIC Fri Aug 30 12:02:04 UTC 2024 x86_64 OS: Linux, ARCH: x86_64, CPU: x86_64 Full OS version: No LSB modules are available. Ubuntu 24.04.1 LTS zlib version: 1.3 (1.3), compile flags: a9 platform id: 0x0a21a7a708000000000d0200

Build information

GNU C: 13.2.0 (13.2.0) sizeof(void*) = 8 Engine flevel: 167, dconf: 167

what should i now do?

[micahsnyder]

@MarkyMarkDE It seems like this might be the problem: https://github.com/Cisco-Talos/clamav/issues/771

Your whitelist.wdb file probably has one or more invalid signatures.

I'm not defending the new behavior / bad error message. We do need to improve the signature loading code so it fails gracefully and explains what is wrong with the signature (hence that ticket is still open).

[MarkyMarkDE]

Indeed, you are so right! We cannot defiantly know what's the problem in case, the error says simply '"can't allocate memory"' but not why. This didn't really help to find and solve bugs.

And this info:

Database directory: /var/lib/clamav WARNING: freshclam.conf and clamd.conf point to different database directories

from clamconf -n log don't match. I don't have any clamd.conf file and I have also checked the Script in /ect/init.d/clamav-deamon and have a look

DATABASEDIR="/var/lib/clamav"

Where are now the different lib-dirs? In my freshclam.conf I use since day one the same lib-dir -> /var/lib/clamav

My whitelist should now the Problem? Maybe but I don't know, this has already worked before 1.0.7

'#' '#' Phishing-Whitelist whitelist.wdb '#' newcomer01@newcomer01-MS-7A40 '#' ClamAV - 1.0.7 (27407) - 23.09.2024, 17:01:01 +0200 '#' /var/lib/clamav_cronjob/clamav_whitelistmake.sh '#' Stand 01.01.2023 (0.103.6) '#'

'#' Ausnahmeregeln fuer Sparkasse und Sparkasse Langen-Seligenstadt '#' X:.+(facebook|twitter|instagram|youtube|play.google|apps.apple).com([\/?].)?:.+mailing.(sparkasse|sls-direkt).de([\/?].)?:0- X:.+(sparkasse|sls-direkt).de([\/?].)?:.+mailing.(sparkasse|sls-direkt).de([\/?].)?:0-

'#' Ausnahmeregel fuer Amazon und Amazon-Subdomains '#' X:.+.amazon.(at|ca|co.uk|co.jp|de|fr)([/?].)?:.+.amazon.com([/?].)?:0-

What should be wrong now?

[micahsnyder]

Comment lines should start with a #. Yours seems to have a ' around the #.

#
# Phishing-Whitelist whitelist.wdb
# newcomer01@newcomer01-MS-7A40
# ClamAV - 1.0.7 (27407) - 23.09.2024, 17:01:01 +0200
# /var/lib/clamav_cronjob/clamav_whitelistmake.sh
#  Stand 01.01.2023 (0.103.6)
#

# Ausnahmeregeln fuer Sparkasse und Sparkasse Langen-Seligenstadt
# X:.+(facebook|twitter|instagram|youtube|play\.google|apps\.apple)\.com([\/?].*)?:.+mailing\.(sparkasse|sls\-direkt)\.de([\/?].*)?:0-
X:.+(sparkasse|sls\-direkt)\.de([\/?].*)?:.+mailing\.(sparkasse|sls\-direkt)\.de([\/?].*)?:0-

# Ausnahmeregel fuer Amazon und Amazon-Subdomains
# X:.+\.amazon\.(at|ca|co\.uk|co\.jp|de|fr)([/?].*)?:.+\.amazon\.com([/?].*)?:0-

Try this?

[MarkyMarkDE]

Comment lines should start with a #. Yours seems to have a ' around the #.

#
# Phishing-Whitelist whitelist.wdb
# newcomer01@newcomer01-MS-7A40
# ClamAV - 1.0.7 (27407) - 23.09.2024, 17:01:01 +0200
# /var/lib/clamav_cronjob/clamav_whitelistmake.sh
#  Stand 01.01.2023 (0.103.6)
#

# Ausnahmeregeln fuer Sparkasse und Sparkasse Langen-Seligenstadt
# X:.+(facebook|twitter|instagram|youtube|play\.google|apps\.apple)\.com([\/?].*)?:.+mailing\.(sparkasse|sls\-direkt)\.de([\/?].*)?:0-
X:.+(sparkasse|sls\-direkt)\.de([\/?].*)?:.+mailing\.(sparkasse|sls\-direkt)\.de([\/?].*)?:0-

# Ausnahmeregel fuer Amazon und Amazon-Subdomains
# X:.+\.amazon\.(at|ca|co\.uk|co\.jp|de|fr)([/?].*)?:.+\.amazon\.com([/?].*)?:0-

Try this?

This is looks mine original code looks like. I have had trouble with github with the #-sign 😉

[micahsnyder]

If you want to learn more about githum markdown syntax, check out https://docs.github.com/github/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax

Looking at your active WDB signature a little closer:

X:.+(sparkasse|sls\-direkt)\.de([\/?].*)?:.+mailing\.(sparkasse|sls\-direkt)\.de([\/?].*)?:0-

I think the issue is the extra \

It looks like this works for me: X:.+(sparkasse|sls-direkt)\.de([/?].*)?:.+mailing\.(sparkasse|sls-direkt)\.de([/?].*)?:0-

But we should also use :17- at the end, instead of :0-: X:.+(sparkasse|sls-direkt)\.de([/?].*)?:.+mailing\.(sparkasse|sls-direkt)\.de([/?].*)?:17-

I'm also not great with regex. I found our online docs useful to compare with the examples: https://docs.clamav.net/manual/Signatures/PhishSigs.html#examples-of-wdb-signatures

[MarkyMarkDE]

If you want to learn more about githum markdown syntax, check out https://docs.github.com/github/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax

Looking at your active WDB signature a little closer:

X:.+(sparkasse|sls\-direkt)\.de([\/?].*)?:.+mailing\.(sparkasse|sls\-direkt)\.de([\/?].*)?:0-

I think the issue is the extra \

It looks like this works for me: X:.+(sparkasse|sls-direkt)\.de([/?].*)?:.+mailing\.(sparkasse|sls-direkt)\.de([/?].*)?:0-

But we should also use :17- at the end, instead of :0-: X:.+(sparkasse|sls-direkt)\.de([/?].*)?:.+mailing\.(sparkasse|sls-direkt)\.de([/?].*)?:17-

I'm also not great with regex. I found our online docs useful to compare with the examples: https://docs.clamav.net/manual/Signatures/PhishSigs.html#examples-of-wdb-signatures

Thank you for your elpful assist!

Have reworked my regex like follow:

# # Phishing-Whitelist whitelist.wdb # newcomer01@newcomer01-MS-7A40 # ClamAV - 1.0.7 (27408) - 24.09.2024, 20:01:01 +0200 # /var/lib/clamav_cronjob/clamav_whitelistmake.sh # Stand 01.01.2023 (0.103.6) #

# Ausnahmeregeln fuer Sparkasse und Sparkasse Langen-Seligenstadt

X:.+(facebook|twitter|instagram|youtube|play\.google|apps\.apple)\.com([\/?].*)?:.+mailing\.(sparkasse|sls-direkt)\.de([\/?].*)?:17-
X:.+(sparkasse|sls-direkt)\.de([/?].*)?:.+mailing\.(sparkasse|sls-direkt)\.de([/?].*)?:17-

# Ausnahmeregel fuer Amazon und Amazon-Subdomains

X:.+\.amazon\.(at|ca|co\.uk|co\.jp|de|fr)([/?].*)?:.+\.amazon\.com([/?].*)?:17-

You see two things:

1. removed the "\-" from sls-direkt -> I thought you always had to escape meta characters - regex101.com does not report this as an error (with escaped \-)
2. changed all the trailing ends from :0- to :17- hopefully it help.

In every case, a huge thank you for your assistance and help me to investigate this issue.

[micahsnyder]

I'm glad we got to the bottom of this. I'll close this issue now and leave the other one open to track the poor quality error messages when loading invalid WDB signatures.

[MarkyMarkDE]

I'm glad we got to the bottom of this. I'll close this issue now and leave the other one open to track the poor quality error messages when loading invalid WDB signatures.

No let this ticket open or reopen for now, your changes did not solve my problem, have copied your whole text and got the same Error in clamscan.log

[MarkyMarkDE]

YES, the Problem are caused by one or more lines of this regexes:

X:.+(facebook|twitter|instagram|youtube|play\.google|apps\.apple)\.com([\/?].*)?:.+mailing\.(sparkasse|sls-direkt)\.de([\/?].*)?:17-
X:.+(sparkasse|sls-direkt)\.de([/?].*)?:.+mailing\.(sparkasse|sls-direkt)\.de([/?].*)?:17-

[micahsnyder]

YES, the Problem are caused by one or more lines of this regexes:

X:.+(facebook|twitter|instagram|youtube|play\.google|apps\.apple)\.com([\/?].*)?:.+mailing\.(sparkasse|sls-direkt)\.de([\/?].*)?:17-
X:.+(sparkasse|sls-direkt)\.de([/?].*)?:.+mailing\.(sparkasse|sls-direkt)\.de([/?].*)?:17-

@MarkyMarkDE the top regex still has the extra \ in ([\/?].*)?

Remove those and it will work:

X:.+(facebook|twitter|instagram|youtube|play\.google|apps\.apple)\.com([/?].*)?:.+mailing\.(sparkasse|sls-direkt)\.de([/?].*)?:17-
X:.+(sparkasse|sls-direkt)\.de([/?].*)?:.+mailing\.(sparkasse|sls-direkt)\.de([/?].*)?:17-

[MarkyMarkDE]

yes, you are right!

But this behavior of ClamAV is against all regex rules, please have a look:

Here with escape-sequent:

Here without:

But so far, it worked now and that's the best what we can do for now, but we need really additional infos in the clamav documentation which explains such differences and that's the "right way" causes internal failures.

To escape the char "-" isn't a must have (also not on regex101.com) but that you SHOULD NEVER ESCAPE "([\/?].*)?" is really important. All developers work in such regex-cases with this or similar tools and make all "perfect" and ClamAV has trouble with ... :wink:

[MarkyMarkDE]

@micahsnyder Indeed, the error has been fixed. The regex issue was my fault, the default setting on regex101.com is PCRE2 (PHP >=7.3) when i change it to ECMAScript (JavaScript) @micahsnyder 's code is right and valid. I ran into the trap, I'm really huge sorry!

[micahsnyder]

@MarkyMarkDE It's no problem. I I'm a little surprised and disappointed by what you found with regex101.com using PCRE2.

We use two libraries for regex in clamav. For phishing signatures (WDB and PDB signatures), ClamAV uses a very basic regex library that is bundled in. I've been wanting to upgrade it for years. For logical pattern matching signatures (LDB signatures) ClamAV uses PCRE2 to enable regex matching. I suppose we may run into compatibility issues with the WDB signatures if we swap to use PCRE2 or another library for the WDB and PDB signatures. :-/

[MarkyMarkDE]

@MarkyMarkDE It's no problem. I I'm a little surprised and disappointed by what you found with regex101.com using PCRE2.

We use two libraries for regex in clamav. For phishing signatures (WDB and PDB signatures), ClamAV uses a very basic regex library that is bundled in. I've been wanting to upgrade it for years. For logical pattern matching signatures (LDB signatures) ClamAV uses PCRE2 to enable regex matching. I suppose we may run into compatibility issues with the WDB signatures if we swap to use PCRE2 or another library for the WDB and PDB signatures. :-/

I was surprised too, that the PHP PCRE2 is complete differed from ECMA JavaScript one (different escaping) and indeed i work mostly with PCRE2 for my PHP Webproject.