search

extremeshok / clamav-unofficial-sigs

ClamAV Unofficial Signatures Updater maintained by eXtremeSHOK.com
https://eXtremeSHOK.com
Other
517 stars 117 forks source link

URlhaus Malware Patrol LinuxMalwareDetec InterServer, never updated #398

Open ghost opened 3 years ago

ghost commented 3 years ago

--------------------- ClamUnofficial-update Begin ------------------------

jul 23 12:00:02 Preparing Databases jul 23 12:00:03 Removing unused file: /var/lib/clamav/spam_marketing.ndb jul 23 12:00:04 vie jul 23 12:00:04 CEST 2021 - Pausing database file updates for 294 seconds... jul 23 12:04:59 vie jul 23 12:04:59 CEST 2021 - Pause complete, checking for new database files... jul 23 12:04:59 Sanesecurity Database & GPG Signature File Updates jul 23 12:04:59 Checking for Sanesecurity updates... jul 23 12:05:01 Sanesecurity mirror site used: jessie.fonant.com 95.217.37.104 jul 23 12:05:02 Testing updated Sanesecurity database file: blurl.ndb jul 23 12:05:02 Clamscan reports Sanesecurity blurl.ndb database integrity tested good jul 23 12:05:03 Successfully updated Sanesecurity production database file: blurl.ndb jul 23 12:05:03 Testing updated Sanesecurity database file: junk.ndb jul 23 12:05:04 Clamscan reports Sanesecurity junk.ndb database integrity tested good jul 23 12:05:04 Successfully updated Sanesecurity production database file: junk.ndb jul 23 12:05:04 Testing updated Sanesecurity database file: jurlbl.ndb jul 23 12:05:04 Clamscan reports Sanesecurity jurlbl.ndb database integrity tested good jul 23 12:05:04 Successfully updated Sanesecurity production database file: jurlbl.ndb jul 23 12:05:04 Testing updated Sanesecurity database file: rogue.hdb jul 23 12:05:04 Clamscan reports Sanesecurity rogue.hdb database integrity tested good jul 23 12:05:04 Successfully updated Sanesecurity production database file: rogue.hdb jul 23 12:05:05 Testing updated Sanesecurity database file: jurlbla.ndb jul 23 12:05:05 Clamscan reports Sanesecurity jurlbla.ndb database integrity tested good jul 23 12:05:05 Successfully updated Sanesecurity production database file: jurlbla.ndb jul 23 12:05:05 Testing updated Sanesecurity database file: phishtank.ndb jul 23 12:05:05 Clamscan reports Sanesecurity phishtank.ndb database integrity tested good jul 23 12:05:05 Successfully updated Sanesecurity production database file: phishtank.ndb jul 23 12:05:05 Testing updated Sanesecurity database file: porcupine.hsb jul 23 12:05:05 Clamscan reports Sanesecurity porcupine.hsb database integrity tested good jul 23 12:05:05 Successfully updated Sanesecurity production database file: porcupine.hsb jul 23 12:05:05 Testing updated Sanesecurity database file: porcupine.ndb jul 23 12:05:06 Clamscan reports Sanesecurity porcupine.ndb database integrity tested good jul 23 12:05:06 Successfully updated Sanesecurity production database file: porcupine.ndb jul 23 12:05:06 LinuxMalwareDetect Database File Updates jul 23 12:05:06 Checking for LinuxMalwareDetect updates... jul 23 12:05:08 No LinuxMalwareDetect database file updates jul 23 12:05:08 interserver Database File Updates jul 23 12:05:08 Checking for interserver updates... jul 23 12:05:08 Checking for updated interServer database file: whitelist.fp jul 23 12:05:11 No updated interServer whitelist.fp database file jul 23 12:05:11 Checking for updated interServer database file: interserver256.hdb jul 23 12:05:17 No updated interServer interserver256.hdb database file jul 23 12:05:17 Checking for updated interServer database file: interservertopline.db jul 23 12:05:24 No updated interServer interservertopline.db database file jul 23 12:05:24 No interServer database file updates jul 23 12:05:24 Removing disabled Malware Expert Database files jul 23 12:05:24 MalwarePatrol Database File Updates jul 23 12:05:24 Checking for MalwarePatrol updates... jul 23 12:05:24 Checking for updated MalwarePatrol database file: malwarepatrol.db jul 23 12:05:31 No updated MalwarePatrol malwarepatrol.db database file jul 23 12:05:31 No MalwarePatrol database file updates jul 23 12:05:31 URLhaus Database File Updates jul 23 12:05:31 Checking for urlhaus updates... jul 23 12:05:31 Checking for updated urlhaus database file: jul 23 12:05:38 WARNING: Failed connection to https://urlhaus.abuse.ch/downloads - SKIPPED urlhaus update jul 23 12:05:38 No updated urlhaus database file jul 23 12:05:38 No urlhaus database file updates jul 23 12:05:38 Removing disabled yararulesproject Database files jul 23 12:05:39 Update(s) detected, reloading ClamAV databases jul 23 12:05:39 ClamAV databases reloading jul 23 12:05:39 Issue tracker : https://github.com/extremeshok/clamav-unofficial-sigs/issues jul 23 12:05:51 Powered By https://eXtremeSHOK.com

---------------------- ClamUnofficial-update End -------------------------

URlhaus not is: https://urlhaus.abuse.ch/downloads; Is: https://urlhaus.abuse.ch/downloads/urlhaus.ndb

perplexityjeff commented 3 years ago

Hi @cotelo, could you please specify what version you are running?

ghost commented 3 years ago

Hello.

ClamUnofficial 7.2.5 config_version="97" Ubuntu 16.04

Thank you very much.

perplexityjeff commented 3 years ago

Hi @cotelo, the error just displays the URL host part of the master config not the full URL. The full URL the script tries is the URL you mentioned.

Could you try and wget or curl 'https://urlhaus.abuse.ch/downloads/urlhaus.ndb' to a temporary place to check if you are able to download the file without the use of the script? There is a good chance that there is a connection issue that is not coming from the script.

Also could you run the update script with --force.

Let me know how it all goes.

Thank you

jengels commented 2 years ago

Hi @perplexityjeff, I am also experiencing this problem on:

and on:

I've checked that the urlhaus database is accessible and can be successfully downloaded: wget "https://urlhaus.abuse.ch/downloads/urlhaus.ndb" --2021-10-08 13:10:42-- https://urlhaus.abuse.ch/downloads/urlhaus.ndb Resolving urlhaus.abuse.ch (urlhaus.abuse.ch)... 151.101.114.49 Connecting to urlhaus.abuse.ch (urlhaus.abuse.ch)|151.101.114.49|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 1258385 (1.2M) Saving to: ‘urlhaus.ndb’ ... 2021-10-08 13:10:42 (12.8 MB/s) - ‘urlhaus.ndb’ saved [1258385/1258385]

But my log looks like: grep WARNING /var/log/clamav-unofficial-sigs/clamav-unofficial-sigs.log | tail -n5 Oct 08 11:19:57 WARNING: Failed connection to https://urlhaus.abuse.ch/downloads - SKIPPED urlhaus urlhaus.ndb update Oct 08 12:31:57 WARNING: Failed connection to https://urlhaus.abuse.ch/downloads - SKIPPED urlhaus urlhaus.ndb update Oct 08 12:38:44 WARNING: Failed connection to https://urlhaus.abuse.ch/downloads - SKIPPED urlhaus urlhaus.ndb update Oct 08 12:48:14 WARNING: Failed connection to https://urlhaus.abuse.ch/downloads - SKIPPED urlhaus urlhaus.ndb update Oct 08 12:53:38 WARNING: Failed connection to https://urlhaus.abuse.ch/downloads - SKIPPED urlhaus urlhaus.ndb update

I think I've also found a possible(?) bug (variable "$work_dir_urlhaust" instead of "$work_dir_urlhaus") in the script "clamav-unofficial-sigs.sh":

grep -n work_dir_urlhaust clamav-unofficial-sigs.sh 1923:if [ -z "$work_dir_urlhaust" ] ; then

But even after correcting this "possible bug" the script still fails to download the urlhaus.ndb (even after removing everything inside the caching directory): rm -rf /var/lib/clamav-unofficial-sigs/*

In the production environment I'm still running an older version: CentOS 7 + clamav-unofficial-sigs-7.0.1-5.el7.noarch + config_version="91". There everything seems to be OK. The Database exists and gets updated as expected:

ls -l /var/lib/clamav/urlhaus.ndb -rw-r--r-- 1 clamupdate clamupdate 1252317 Oct 8 13:15 /var/lib/clamav/urlhaus.ndb

grep urlhaus /var/log/clamav-unofficial-sigs/clamav-unofficial-sigs.log | tail -n10 Oct 08 12:15:18 Checking for urlhaus updates... Oct 08 12:15:18 Checking for updated urlhaus database file: urlhaus.ndb Oct 08 12:15:18 Testing updated urlhaus database file: urlhaus.ndb Oct 08 12:15:18 Clamscan reports urlhaus urlhaus.ndb database integrity tested good Oct 08 12:15:18 Successfully updated urlhaus production database file: urlhaus.ndb Oct 08 13:16:41 Checking for urlhaus updates... Oct 08 13:16:41 Checking for updated urlhaus database file: urlhaus.ndb Oct 08 13:16:41 Testing updated urlhaus database file: urlhaus.ndb Oct 08 13:16:41 Clamscan reports urlhaus urlhaus.ndb database integrity tested good Oct 08 13:16:41 Successfully updated urlhaus production database file: urlhaus.ndb

So this seems to be a problem which only affects the newer version(s)?

perplexityjeff commented 2 years ago

@jengels In previous versions there was support for urlhaus but because of a typo it was not used. I don't know when that was introduced, but I had the same issues as you guys.

See the issue here https://github.com/extremeshok/clamav-unofficial-sigs/issues/385, https://github.com/extremeshok/clamav-unofficial-sigs/pull/386

I attempted that to fix here https://github.com/extremeshok/clamav-unofficial-sigs/pull/390

Related https://github.com/extremeshok/clamav-unofficial-sigs/pull/400

Currently it is merged into 'dev' and the original developer still needs to give the 'go' and push the fix as an actual update.

I hope that if you change the script that you are able to fix it at least until an official update is available.

I am myself currently looking into https://github.com/rseichter/fangfrisch for our production environment instead of this script because sadly it takes a while for these bugs to get fixed. I have full respect for the original developer of this script and understand that it is not his full time priority but it does at least for us users take some time for these bugs to get fixed.

jengels commented 2 years ago

@perplexityjeff, thanks for the feedback. The problem got fixed by applying patches #390 and #386 locally. Hopefully the patches will be included in the next release...

perplexityjeff commented 2 years ago

@jengels No problem at all.