curl: (22) The requested URL returned error: 404 Not Found

[jobst]

System: CentOS 7.X, clamav-0.101.3 clamav-unofficial-sigs.sh -V Version: v6.1.1 (2019-09-02) Required Configuration Version: v76

I get this error every night curl: (22) The requested URL returned error: 404 Not Found

when running the updates from cron: /bin/bash /usr/local/bin/clamav-unofficial-sigs.sh > /dev/null

Running it from the command line /bin/bash /usr/local/bin/clamav-unofficial-sigs.sh

does not produce any errors.

How can I debug this problem to find out why CURL is failing?

[nrosier]

Looks like the directory structure of the Yara-Rules site has changed. All lower case now and rules seem to have been added. files are defined in master.conf so this file will have to be updated (or overrule with user.conf).

[jobst]

When I upgraded to the latest version I downloaded the master.conf from github and made changes to my original one. I just checked and both match.

BUT I can SEE there are discrepancies in the directory structure of the actual files and what is in the master.conf: ` [root /var/lib/clamav-unofficial-sigs] #>ls -al total 44 drwxr-x--- 11 clamav clamav 4096 Nov 28 2017 . drwxr-xr-x 51 root root 4096 Nov 13 15:59 .. drwxr-xr-x 2 clamav clamav 4096 Jan 22 10:11 configs drwxr-xr-x 2 clamav clamav 4096 Nov 28 2017 dbs-add drwxr-xr-x 2 clamav clamav 4096 Nov 28 2017 dbs-lmd drwxr-xr-x 2 clamav clamav 4096 Nov 28 2017 dbs-mbl drwxr-xr-x 2 clamav clamav 4096 Nov 28 2017 dbs-si drwxr-xr-x 2 clamav clamav 4096 Jan 22 10:11 dbs-ss drwxr-xr-x 2 clamav clamav 4096 Jan 8 03:10 dbs-yara drwx------ 2 clamav clamav 4096 Jan 22 10:11 gpg-key drwxr-xr-x 2 clamav clamav 4096 Jan 22 10:12 pid

`

and:

`

[root /var/lib/clamav-unofficial-sigs] #>ls -al dbs-yara/ total 180 drwxr-xr-x 2 clamav clamav 4096 Jan 8 03:10 . drwxr-x--- 11 clamav clamav 4096 Nov 28 2017 .. -rw-r--r-- 1 clamav clamav 47013 Jan 21 2019 antidebug_antivm.yar -rw-r--r-- 1 clamav clamav 465 Jan 8 03:10 CVE-2010-0805.yar -rw-r--r-- 1 clamav clamav 823 Jan 8 03:10 CVE-2010-0887.yar -rw-r--r-- 1 clamav clamav 442 Jan 8 03:10 CVE-2010-1297.yar -rw-r--r-- 1 clamav clamav 341 Jan 8 03:10 CVE-2013-0074.yar -rw-r--r-- 1 clamav clamav 903 Jan 8 03:10 CVE-2013-0422.yar -rw-r--r-- 1 clamav clamav 775 Jan 8 03:10 CVE-2015-5119.yar -rw-r--r-- 1 clamav clamav 10889 Jan 8 03:09 EK_Angler.yar -rw-r--r-- 1 clamav clamav 14659 Jan 8 03:09 EK_Blackhole.yar -rw-r--r-- 1 clamav clamav 3401 Jan 8 03:09 EK_BleedingLife.yar -rw-r--r-- 1 clamav clamav 1349 Jan 8 03:10 EK_Crimepack.yar -rw-r--r-- 1 clamav clamav 4688 Jan 8 03:10 EK_Eleonore.yar -rw-r--r-- 1 clamav clamav 8268 Jan 8 03:10 EK_Fragus.yar -rw-r--r-- 1 clamav clamav 16842 Jan 8 03:10 EK_Phoenix.yar -rw-r--r-- 1 clamav clamav 1860 Jan 8 03:10 EK_Sakura.yar -rw-r--r-- 1 clamav clamav 8488 Jan 8 03:10 EK_ZeroAcces.yar -rw-r--r-- 1 clamav clamav 1435 Jan 8 03:10 EK_Zerox88.yar -rw-r--r-- 1 clamav clamav 800 Jan 8 03:10 EK_Zeus.yar

`

while the definition in master.conf is like this (as found on github) I have taken out the comments to shorten the output

` declare -a yararulesproject_dbs=( Exploit-Kits/EK_Angler.yar|LOW # Angler Exploit Kit Redirector Exploit-Kits/EK_Blackhole.yar|LOW # BlackHole2 Exploit Kit Detection Exploit-Kits/EK_BleedingLife.yar|LOW # BleedingLife2 Exploit Kit Detection Exploit-Kits/EK_Crimepack.yar|LOW # CrimePack Exploit Kit Detection Exploit-Kits/EK_Eleonore.yar|LOW # Eleonore Exploit Kit Detection Exploit-Kits/EK_Fragus.yar|LOW # Fragus Exploit Kit Detection Exploit-Kits/EK_Phoenix.yar|LOW # Phoenix Exploit Kit Detection Exploit-Kits/EK_Sakura.yar|LOW # Sakura Exploit Kit Detection Exploit-Kits/EK_ZeroAcces.yar|LOW # ZeroAccess Exploit Kit Detection Exploit-Kits/EK_Zerox88.yar|LOW # 0x88 Exploit Kit Detection Exploit-Kits/EK_Zeus.yar|LOW # Zeus Exploit Kit Detection Malicious_Documents/maldoc_somerules.yar|HIGH # documents with malicious code Malicious_Documents/Maldoc_Hidden_PE_file.yar|HIGH # Detect a hidden PE file inside a sequence of numbers (comma separated) Packers/packer.yar|MEDIUM # well-known sofware packers CVE_Rules/CVE-2010-0805.yar|MEDIUM # CVE 2010 0805 CVE_Rules/CVE-2010-0887.yar|MEDIUM # CVE 2010 0887 CVE_Rules/CVE-2010-1297.yar|MEDIUM # CVE 2010 1297 CVE_Rules/CVE-2013-0074.yar|MEDIUM # CVE 2013 0074 CVE_Rules/CVE-2013-0422.yar|MEDIUM # CVE 2013 0422 CVE_Rules/CVE-2015-5119.yar|MEDIUM # CVE 2015 5119 Packers/Javascript_exploit_and_obfuscation.yar|HIGH # JavaScript Obfuscation Detection ) #END yararulesproject DATABASES

`

I am not 100% sure what the directory structure should be below /var/lib/clamav-unofficial-sigs

Any docs? Could you highlight/elaborate?

thanks

[nrosier]

I've put this in my user.conf. It might need some tuning and not sure all rules are needed, but it now updates without errors:

declare -a yararulesproject_dbs=(
### Yara Rules https://github.com/Yara-Rules/rules
#
# Some rules are now in sub-directories. To reference a file in a sub-directory
# use subdir/file
# LOW
exploit_kits/EK_Eleonore.yar|LOW
exploit_kits/EK_Fragus.yar|LOW
exploit_kits/EK_Zeus.yar|LOW
exploit_kits/EK_Zerox88.yar|LOW
exploit_kits/EK_Sakura.yar|LOW
exploit_kits/EK_Angler.yar|LOW
exploit_kits/EK_Blackhole.yar|LOW
exploit_kits/EK_Phoenix.yar|LOW
exploit_kits/EK_Crimepack.yar|LOW
exploit_kits/EK_BleedingLife.yar|LOW
exploit_kits/EK_ZeroAcces.yar|LOW
antidebug_antivm/antidebug_antivm.yar|LOW
# MEDIUM
packers/packer_compiler_signatures.yar|MEDIUM
packers/packer.yar|MEDIUM
packers/JJencode.yar|MEDIUM
email/email_Ukraine_BE_powerattack.yar|MEDIUM
email/EMAIL_Cryptowall.yar|MEDIUM
email/attachment.yar|MEDIUM
email/image.yar|MEDIUM
email/scam.yar|MEDIUM
email/urls.yar|MEDIUM
email/bank_rule.yar|MEDIUM
cve_rules/CVE-2010-0887.yar|MEDIUM
cve_rules/CVE-2015-2545.yar|MEDIUM
cve_rules/CVE-2012-0158.yar|MEDIUM
cve_rules/CVE-2013-0074.yar|MEDIUM
cve_rules/CVE-2015-2426.yar|MEDIUM
cve_rules/CVE-2015-1701.yar|MEDIUM
cve_rules/CVE-2017-11882.yar|MEDIUM
cve_rules/CVE-2018-20250.yar|MEDIUM
cve_rules/CVE-2010-1297.yar|MEDIUM
cve_rules/CVE-2016-5195.yar|MEDIUM
cve_rules/CVE-2010-0805.yar|MEDIUM
cve_rules/CVE-2013-0422.yar|MEDIUM
cve_rules/CVE-2018-4878.yar|MEDIUM
cve_rules/CVE-2015-5119.yar|MEDIUM
# HIGH
maldocs/Maldoc_DDE.yar|HIGH
maldocs/maldoc_somerules.yar|HIGH
maldocs/Maldoc_Suspicious_OLE_target.yar|HIGH
maldocs/Maldoc_APT_OLE_JSRat.yar|HIGH
maldocs/Maldoc_APT19_CVE-2017-1099.yar|HIGH
maldocs/Maldoc_Dridex.yar|HIGH
maldocs/Maldoc_VBA_macro_code.yar|HIGH
maldocs/Maldoc_Contains_VBE_File.yar|HIGH
maldocs/Maldoc_malrtf_ole2link.yar|HIGH
maldocs/Maldoc_PowerPointMouse.yar|HIGH
maldocs/Maldoc_Hidden_PE_file.yar|HIGH
maldocs/Maldoc_Word_2007_XML_Flat_OPC.yar|HIGH
maldocs/Maldoc_CVE_2017_11882.yar|HIGH
maldocs/Maldoc_CVE-2017-0199.yar|HIGH
maldocs/Maldoc_APT10_MenuPass.yar|HIGH
maldocs/Maldoc_CVE_2017_8759.yar|HIGH
maldocs/Maldoc_UserForm.yar|HIGH
maldocs/Maldoc_MIME_ActiveMime_b64.yar|HIGH
maldocs/Maldoc_PDF.yar|HIGH
packers/Javascript_exploit_and_obfuscation.yar|HIGH
crypto/crypto_signatures.yar|HIGH
) #END yararulesproject DATABASES

[nrosier]

3 rules fail to load in ClamAV (i.e. give BAD status):

• antidebug_antivm.yar
• packer.yar
• packer_compiler_signatures.yar 1st 2 were included in the original master.conf, last one I added as it is new.

[jobst]

Oh! I knew about antidebug* but not about the two packer rules, excluded them.

I actually just understood something about the directories in the master.conf file (user.conf). The directories are not LOCAL, but they are PROVIDER directories, it confused the hell out of me as ALL files LOCALLY are in ONE directory. Also there are two copies of each file, one in /var/lib/clamav-unofficial-sigs the other in /var/lib/clamav. Then it daunted on me that one is the DOWNLOAD directory, file here are copied to the clamd database.

If I could post an image of Patrick Stewart's facepalm I would do so now.

[extremeshok]

Will push code updates today with the fixes

[extremeshok]

commit a783a7c4da6ed7d6a83c149354d22a69cccc69f4

[extremeshok]

working/fixed in the dev