Confusing .wdb error message for invalid signature in clamav 1.0

[LSchuepbach]

Describe the bug

ClamAV refuses to load .wdb signatures with such a format: X:urldefense\.com:.+

Here's an excerpt of the error message:

LibClamAV debug: Loading regex_list
LibClamAV Warning: cli_realloc(): File or section is too large to scan (0 bytes).                      For your safety, ClamAV limits how much memory an operation can allocate to 1073741824 bytes
LibClamAV Error: Can't load /root/test.wdb: Can't allocate memory
LibClamAV Error: cli_loaddbdir: error loading database /root/test.wdb
ERROR: Can't allocate memory

Using such a format helps a lot to deal with the FPs coming when a correspondent uses a URL-rewriting phishing protection. urldefense.com is just one example in a long list of such services that I don't want to get hits on.

How to reproduce the problem

I used today's docker's unstable_base.

Create a file test.wdb with the content mentioned above.

Using X:urldefense\.com:.{1,30}, ClamAV will not complain while loading, but the SpoofedDomain will still hit.

It works without problem under latest_base (0.105.1)

[micahsnyder]

@LSchuepbach

I looked at it harder and realized that the provided WDB signature is actually invalid and fails to load in 0.105 (and prior) as well, but it fails silently. Fixes during 1.0 development that patched WDB signature load bugs now cause it to fail loudly.

Unfortunately the error message is completely unhelpful and doesn't explain why the signature is invalid. So the reported issue really should still be looked at, but mostly to improve the error handling to detect the issue and explain why the signature is invalid. That's not as urgent since it's not really a new bug in 1.0 after all.

My understanding right now is a little muddled. Our documentation claims that the second URL in the signature should have the last 3 characters in the expression as an exact match, as stated here: https://docs.clamav.net/manual/Signatures/PhishSigs.html#wdb-format However, for similar clicktime protection sites, I have had luck in testing with a format like this:

X:(.+\.)?safelinks.protection.outlook.com:(.+\.)?.*\.(at|be|ca|ch|co\.uk|de|es|fr|ie|in|it|nl|ph|pl|com|com\.(au|cn|hk|my|sg))([/?].*)?
X:(.+\.)?linkprotect.cudasvc.com:(.+\.)?.*\.(at|be|ca|ch|co\.uk|de|es|fr|ie|in|it|nl|ph|pl|com|com\.(au|cn|hk|my|sg))([/?].*)?

So for urldefense.com links, maybe something like this will work for you:

X:(.+\.)?urldefense\.com:(.+\.)?.*\.(at|be|ca|ch|co\.uk|de|es|fr|ie|in|it|nl|ph|pl|com|com\.(au|cn|hk|my|sg))([/?].*)?

I have a ticket in to add these into our daily.cvd and will add in the urldefense one as well. I will bring it up with our threat defense team again to see if they're happy with that.

[LSchuepbach]

Thank you for looking into it.

Strange as it worked as expected on 105.0 (we use it in production... a failure is quickly visible).

Adding this list of TLDs is just pushing the ball a little bit down the road as more and more new TLDs are being used also by big companies. I wouldn't really prefer to not have to provide anything for the second part. Maybe a new format could be used that isn't X or M if that makes it easier for you.

Good idea to add urldefense to daily.cvd, but our list is quite longer. Let me know if you want more clicktime protection URLs that we collected.

[micahsnyder]

Adding this list of TLDs is just pushing the ball a little bit down the road as more and more new TLDs are being used also by big companies. I wouldn't really prefer to not have to provide anything for the second part. Maybe a new format could be used that isn't X or M if that makes it easier for you.

I agree. That does seem like a reasonable solution.

Edit: worth noting though that we only need to have TLD's in the WDB in the second part that are included in entries in the .pdb database, as those are the domains being watched for phishing.

[micahsnyder]

Let me know if you want more clicktime protection URLs that we collected.

@LSchuepbach sorry I somehow missed this offer. Yes if you have additional clicktime protection URLs, I'd love to see your list.

[micahsnyder]

I have created a separate issue for the feature request to allow just by real-URL domain. Since we do have a workable solution for now, and we're overwhelmed with other tasks, we do not plan to work on it. So I marked it as "help wanted".

Will keep this issue open because the error message is really bad and should be fixed to explain what's wrong with the signature.

[LSchuepbach]

Here's the list that we collected:

.global.blackspider.com
secure-web.cisco.com
linkprotect.cudasvc.com
deref-gmx.net
urlsand.esvalabs.com
protect2.fireeye.com
atpscan.global.hornetsecurity.com
protect-eu.mimecast.com
safelinks.protection.outlook.com
urldefense.proofpoint.com
protection.retarus.com
.protection.sophos.com
clicktime.symantec.com
linklock.titanhq.com
*-ctp.trendmicro.com
scanmail.trustwave.com
urldefense.com
url-proctection.com
.urlsand.com
antiphishing.vadesecure.com

Thank you for opening that other issue. I hope someone will have the time at some point to work on it.

EDITED TO ADD url-protection.com

[SecT0uch]

I'm facing this error now I passed to 1.0.

I agree an other format would be a good idea, listing all the TLD in the regex doesn't sound optimal.

[mhammedyildirim]

hi folks, is there any progress on this problem? because i am facing this error now. here's some examples in my .wdb file. first one:

X:.+\.amazon\.(at|ca|co\.uk|co\.jp|de|fr)([/?].*)?:.+\.amazon\.com([/?].*)?:17-

second one:

X:.+\.awstrack.me([/?].*)?:.*([/?].*)?:17-

first one is ok, there is no problem but if i want to use second one i am getting an error like this:

Jul 17 14:41:15 freshclam[3203884]: WARNING: Stderr output from database load : WARNING: [LibClamAV] Can't load /var/lib/clamav/tmp.a6f85b23b/clamav-14a9a107cb0f8fbbcda532702f024c68.tmp-custom-signatures-phishing.wdb: Can't allocate memory [...] ERROR: Failed to load new database: Can't allocate memory
Jul 17 14:41:15 freshclam[3203884]: WARNING: Database load exited with "Test failed"
Jul 17 14:41:15 freshclam[3203884]: ERROR: Database test FAILED.

---

[root]# clamconf -n
Checking configuration files in /etc

Config file: clamd.conf
-----------------------
LogClean = "yes"
LogSyslog = "yes"
LogFacility = "LOG_MAIL"
LogVerbose = "yes"
ExtendedDetectionInfo = "yes"
LocalSocket = "/run/clamd.scan/clamd.sock"
TCPSocket = "3310"
TCPAddr = "127.0.0.1"
MaxThreads = "50"
ReadTimeout = "300"
User = "clamscan"
AlertEncryptedArchive = "yes"

Config file: freshclam.conf
---------------------------
DatabaseMirror = xxxxxxxxxxxxxxxxxxxxxxx
DatabaseCustomURL = xxxxxxxxxxxxxxxxxxxx

clamav-milter.conf not found

Software settings
-----------------
Version: 1.0.6
Optional features supported: MEMPOOL AUTOIT_EA06 BZIP2 LIBXML2 PCRE2 ICONV JSON 

Database information
--------------------
Database directory: /var/lib/clamav
[3rd Party] securiteinfo.ign2: 219 sigs
[3rd Party] custom-signatures-filetype.ftm: 6 sigs
[3rd Party] custom-signatures-phishing.pdb: 6 sigs
[3rd Party] securiteinfoascii.hdb: 152167 sigs
[3rd Party] malwarepatrol.db: 0 sig 
[3rd Party] securiteinfoandroid.hdb: 124318 sigs
[3rd Party] javascript.ndb: 53379 sigs
[3rd Party] custom-signatures-whitelist.ign2: 15 sigs
[3rd Party] custom-signatures-meta.cdb: 12 sigs
[3rd Party] custom-signatures-md5.hdb: 13 sigs
[3rd Party] safebrowsing.gdb: 6002939 sigs
daily.cld: version 27339, sigs: 2064430, built on Wed Jul 17 11:36:14 2024
[3rd Party] securiteinfohtml.hdb: 104652 sigs
main.cvd: version 62, sigs: 6647427, built on Thu Sep 16 15:32:42 2021
[3rd Party] custom-signatures-text.ndb: 263 sigs
bytecode.cvd: version 335, sigs: 86, built on Tue Feb 27 18:37:24 2024
[3rd Party] custom-signatures-phishing-dev.wdb: 7 sigs
[3rd Party] securiteinfo.hdb: 225381 sigs
Total number of signatures: 15375320

Platform information
--------------------
uname: Linux 5.15.102-1-pve #1 SMP PVE 5.15.102-1 (2023-03-14T13:48Z) x86_64
OS: Linux, ARCH: x86_64, CPU: x86_64
zlib version: 1.2.11 (1.2.11), compile flags: a9
platform id: 0x0a21a6a60800000000080500

Build information
-----------------
GNU C: 8.5.0 20210514 (Red Hat 8.5.0-20) (8.5.0)
sizeof(void*) = 8
Engine flevel: 166, dconf: 166

@micahsnyder