clamscan broken with Malet 1.6

[gbot]

Hi,

Since installing Maldet 1.6 (on Ubuntu 14.04), whenever I run a clamscan I get the following errors, server CPU and memory usage spike way up, and if I don't kill the process quickly, the server crashes.

Appears to be an issue with var/lib/clamav/rfxn.yara containing duplicate rules.

My clamscan version is:

ClamAV 0.99.2/23297/Sat Apr 15 08:54:36 2017

When I run a foreground clamscan, such as clamscan -ir /srv/users/username/apps/appname/public, I get:

LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 9897 duplicate identifier "dump_sales_quote_payment"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11502 duplicate identifier "dump_sales_order"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11506 duplicate identifier "md5_64651cede2467fdeb1b3b7e6ff3f81cb"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11510 duplicate identifier "md5_6bf4910b01aa4f296e590b75a3d25642"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11526 duplicate identifier "eval_post"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11532 duplicate identifier "spam_mailer"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11538 duplicate identifier "md5_0105d05660329704bdb0ecd3fd3a473b"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11546 duplicate identifier "md5_0b1bfb0bdc7e017baccd05c6af6943ea"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11552 duplicate identifier "md5_2495b460f28f45b40d92da406be15627"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11556 duplicate identifier "md5_2c37d90dd2c9c743c273cb955dd83ef6"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11560 duplicate identifier "md5_3ccdd51fe616c08daafd601589182d38"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11564 duplicate identifier "md5_4b69af81b89ba444204680d506a8e0a1"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11569 duplicate identifier "md5_71a7c769e644d8cf3cf32419239212c7"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11578 duplicate identifier "md5_825a3b2a6abbe6abcdeda64a73416b3d"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11584 duplicate identifier "md5_87cf8209494eedd936b28ff620e28780"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11600 duplicate identifier "md5_c647e85ad77fd9971ba709a08566935d"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11604 duplicate identifier "md5_fb9e35bf367a106d18eb6aa0fe406437"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11608 duplicate identifier "md5_8e5f7f6523891a5dcefcbb1a79e5bbe9"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11612 duplicate identifier "eval_base64_decode_a"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11615 duplicate identifier "obfuscated_eval"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11626 duplicate identifier "md5_ab63230ee24a988a4a9245c2456e4874"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11629 duplicate identifier "md5_b579bff90970ec58862ea8c26014d643"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11635 duplicate identifier "md5_d30b23d1224438518d18e90c218d7c8b"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11639 duplicate identifier "md5_24f2df1b9d49cfb02d8954b08dba471f"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11641 duplicate identifier "base64_hidden_in_image"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11645 duplicate identifier "hide_data_in_jpeg"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11649 duplicate identifier "hidden_file_upload_in_503"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11655 duplicate identifier "md5_fd141197c89d27b30821f3de8627ac38"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11661 duplicate identifier "visbot"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11663 duplicate identifier "md5_39ca2651740c2cef91eb82161575348b"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11671 duplicate identifier "md5_4c4b3d4ba5bce7191a5138efa2468679"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11677 duplicate identifier "md5_6eb201737a6ef3c4880ae0b8983398a9"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11681 duplicate identifier "md5_d201d61510f7889f1a47257d52b15fa2"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11685 duplicate identifier "md5_06e3ed58854daeacf1ed82c56a883b04"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11689 duplicate identifier "md5_28690a72362e021f65bb74eecc54255e"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11691 duplicate identifier "overwrite_globals_hack"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11696 duplicate identifier "md5_4adef02197f50b9cc6918aa06132b2f6"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11701 duplicate identifier "obfuscated_globals"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11707 duplicate identifier "ld_preload_backdoor"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11711 duplicate identifier "fake_magentoupdate_site"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11715 duplicate identifier "md5_b3ee7ea209d2ff0d920dfb870bad8ce5"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11721 duplicate identifier "md5_e03b5df1fa070675da8b6340ff4a67c2"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11725 duplicate identifier "md5_023a80d10d10d911989e115b477e42b5"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11731 duplicate identifier "md5_4aa900ddd4f1848a15c61a9b7acd5035"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11735 duplicate identifier "md5_f797dd5d8e13fe5c8898dbe3beb3cc5b"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11921 duplicate identifier "onepage_or_checkout"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11930 duplicate identifier "sinlesspleasure_com"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11934 duplicate identifier "amasty_biz"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11938 duplicate identifier "amasty_biz_js"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11942 duplicate identifier "returntosender"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11946 duplicate identifier "ip_5uu8_com"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11950 duplicate identifier "cloudfusion_me"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11954 duplicate identifier "grelos_v"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11967 duplicate identifier "hacked_domains"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11971 duplicate identifier "mage_cdn_link"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11975 duplicate identifier "credit_card_regex"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11979 duplicate identifier "jquery_code_su"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11983 duplicate identifier "jquery_code_su_multi"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11987 duplicate identifier "Trafficanalyzer_js"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11991 duplicate identifier "atob_js"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 11995 duplicate identifier "gate_php_js"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 12001 duplicate identifier "googieplay_js"
LibClamAV Error: yyerror(): /var/lib/clamav/rfxn.yara line 12004 duplicate identifier "md5_cdn_js_link_js"
LibClamAV Error: cli_loadyara: failed to parse rules file /var/lib/clamav/rfxn.yara, error count 63

[gbot]

I've only tested on one server so far, but it appears that running maldet -u has resolved this:

maldet -u
Linux Malware Detect v1.6
            (C) 2002-2017, R-fx Networks <proj@rfxn.com>
            (C) 2017, Ryan MacDonald <ryan@rfxn.com>
This program may be freely redistributed under the terms of the GNU GPL v2

maldet(23035): {sigup} performing signature update check...
maldet(23035): {sigup} local signature set is version 2017041129590
maldet(23035): {sigup} new signature set (2017041410039) available
maldet(23035): {sigup} downloading https://cdn.rfxn.com/downloads/maldet-sigpack.tgz
maldet(23035): {sigup} downloading https://cdn.rfxn.com/downloads/maldet-cleanv2.tgz
maldet(23035): {sigup} verified md5sum of maldet-sigpack.tgz
maldet(23035): {sigup} unpacked and installed maldet-sigpack.tgz
maldet(23035): {sigup} verified md5sum of maldet-clean.tgz
maldet(23035): {sigup} unpacked and installed maldet-clean.tgz
maldet(23035): {sigup} signature set update completed
maldet(23035): {sigup} 12451 signatures (9721 MD5 | 1951 HEX | 779 YARA | 0 USER)

Slightly off topic here, but does running a maldet -a scan also scan with the clamscan definitions? i.e. is running maldet like running clamscan as well?

Thanks!

[milosdjakonovic]

@gbot good off topic question, I was wandering the same and does clamscan benefits from maldet rules automatically when maldet is installed?

[rfxn]

@gbot @milosdjakonovic This is a two way relationship; when LMD is installed it places the LMD compatible ClamAV rules inside the ClamAV signature path. As such, when LMD uses clamscan to perform scanning, it is taking advantage of all ClamAV signatures, inclusive of LMD. This is also true of manual scans with clamscan, it will take advantage of all its normal rules as well as the LMD rules, since they are located in the ClamAV signature path.

The duplicate signature issue was corrected some time back and was an upstream issue with the YARA rules and how they are published by their creators. I have since added additional checking on the rules when they are downloaded into the LMD repo to ensure the duplicates never happen again.

Thanks!

[gbot]

@rfxn Thanks very much for the explanation! I was sometimes repeating scans of the same path with maldet and then clamscan -- great to know that's not necessary.