search

rfxn / linux-malware-detect

Linux Malware Detection (LMD)
http://www.rfxn.com/projects/linux-malware-detect/
GNU General Public License v2.0
1.2k stars 234 forks source link

Updating signature and other assets fails #200

Closed waja closed 7 years ago

waja commented 7 years ago 
maldet(6576): {sigup} local signature set is version 2017012825142
maldet(6576): {sigup} could not download signature data from server, please try again later.

With c55e7d3a6f4ff386bf45a0308de9d3751d6df503 you introduced https based URLs. Anyway ... there might some issues with this.

Debian 7.x:

$ LANG=C; wget https://cdn.rfxn.com/downloads/maldet.current.ver.beta
--2017-02-02 21:29:09--  https://cdn.rfxn.com/downloads/maldet.current.ver.beta
Resolving cdn.rfxn.com (cdn.rfxn.com)... 104.28.23.81, 104.28.22.81, 2400:cb00:2048:1::681c:1751, ...
Connecting to cdn.rfxn.com (cdn.rfxn.com)|104.28.23.81|:443... connected.
GnuTLS: A TLS fatal alert has been received.
Unable to establish SSL connection.
$ LANG=C; wget http://cdn.rfxn.com/downloads/maldet.current.ver.beta
--2017-02-02 21:29:31--  http://cdn.rfxn.com/downloads/maldet.current.ver.beta
Resolving cdn.rfxn.com (cdn.rfxn.com)... 104.28.23.81, 104.28.22.81, 2400:cb00:2048:1::681c:1751, ...
Connecting to cdn.rfxn.com (cdn.rfxn.com)|104.28.23.81|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4
Saving to: `maldet.current.ver.beta'

100%[===================================================================================================>] 4           --.-K/s   in 0s

2017-02-02 21:29:31 (247 KB/s) - `maldet.current.ver.beta' saved [4/4]

Debian 6.x:

$ LANG=C; wget https://cdn.rfxn.com/downloads/maldet.current.ver.beta
--2017-02-02 21:28:37--  https://cdn.rfxn.com/downloads/maldet.current.ver.beta
Resolving cdn.rfxn.com... 104.28.23.81, 104.28.22.81, 2400:cb00:2048:1::681c:1751, ...
Connecting to cdn.rfxn.com|104.28.23.81|:443... connected.
OpenSSL: error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error
Unable to establish SSL connection.

From what I've found on the search engines there might be some issue in redirected https connections, but nothing specific. Other distributions and (Debian) derivates might be affected. On Debian 8.x this works as expected.

Many thanks, Jan.

rfxn commented 7 years ago

I will investigate this, I suspect the issue is the SSL/TLS version on connection to Cloudflare CDN.

Thanks

waja commented 7 years ago

Looking into https://www.ssllabs.com/ssltest/analyze.html?d=cdn.rfxn.com&hideResults=on seems to have TLS1.0-TLS1.2.

rfxn commented 7 years ago

I got deb6/7/8 virtualbox instances I'll kick off tonight and poke at this further. As always, thanks for the contribution in pointing this out!

rfxn commented 7 years ago

There doesn't appear to be an easy fix with wget itself on these older distro releases, the issue is with wget itself and the cipher suites enforced on the CDN simply not being available on older versions of wget.

An option might be to preference curl which works fine on deb7, then fallback to wget as a last resort. I am liking this better than the alternative of turning off the CDN.

rfxn commented 7 years ago

In the interest of closing out v1.6, I've for now switched CDN's to something with a little looser cipher suites that works on deb 6/7 TLSv1.

I will revisit this as a TODO item in a later iterative update to switch to CURL.

waja commented 7 years ago

@rfxn Unfortunately it seems not to work right now for Debian 6 and 7

~$ grep PRETTY_NAME /etc/os-release
PRETTY_NAME="Debian GNU/Linux 7 (wheezy)"
~$ LANG=C /usr/bin/wget --referer=LMD:1.6 -e http_proxy= -e https_proxy= -T5 -t3 https://cdn.rfxn.com/downloads/maldet.sigs.ver -O /tmp/.hver19815
--2017-03-19 16:31:31--  https://cdn.rfxn.com/downloads/maldet.sigs.ver
Resolving cdn.rfxn.com (cdn.rfxn.com)... 104.28.22.81, 104.28.23.81, 2400:cb00:2048:1::681c:1651, ...
Connecting to cdn.rfxn.com (cdn.rfxn.com)|104.28.22.81|:443... connected.
GnuTLS: A TLS fatal alert has been received.
Unable to establish SSL connection.
~$ cat /etc/debian_version
6.0.10
~$ LANG=C /usr/bin/wget --referer=LMD:1.6 -e http_proxy= -e https_proxy= -T5 -t3 https://cdn.rfxn.com/downloads/maldet.sigs.ver -O /tmp/.hver19815
--2017-03-19 16:35:20--  https://cdn.rfxn.com/downloads/maldet.sigs.ver
Resolving cdn.rfxn.com... 104.28.23.81, 104.28.22.81, 2400:cb00:2048:1::681c:1751, ...
Connecting to cdn.rfxn.com|104.28.23.81|:443... connected.
OpenSSL: error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error
Unable to establish SSL connection.
waja commented 7 years ago

While downloading with curl on Debian 7 works:

~$ grep PRETTY_NAME /etc/os-release
PRETTY_NAME="Debian GNU/Linux 7 (wheezy)"
~$ LANG=C curl https://cdn.rfxn.com/downloads/maldet.sigs.ver --tlsv1 -o /tmp/.hver19815
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    14  100    14    0     0     24      0 --:--:-- --:--:-- --:--:--    24
~$ cat /tmp/.hver19815
2017022510908

On Debian 6 it doesn't:

~$ cat /etc/debian_version
6.0.10
~$ LANG=C curl https://cdn.rfxn.com/downloads/maldet.sigs.ver --tlsv1 -o /tmp/.hver19815
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (35) error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
MichelClement commented 7 years ago

Info: on Debian GNU/Linux 8 (jessie) is works if you install the package 'ca-certificates'.

waja commented 7 years ago

Info: on Debian GNU/Linux 8 (jessie) is works if you install the package 'ca-certificates'.

But that issue has another source:

~$ LANG=C /usr/bin/wget --referer=LMD:1.6 -e http_proxy= -e https_proxy= -T5 -t3 https://cdn.rfxn.com/downloads/maldet.sigs.ver -O /tmp/.hver19815
converted 'https://cdn.rfxn.com/downloads/maldet.sigs.ver' (ANSI_X3.4-1968) -> 'https://cdn.rfxn.com/downloads/maldet.sigs.ver' (UTF-8)
--2017-03-27 12:28:01--  https://cdn.rfxn.com/downloads/maldet.sigs.ver
Resolving cdn.rfxn.com (cdn.rfxn.com)... 45.33.122.151
Connecting to cdn.rfxn.com (cdn.rfxn.com)|45.33.122.151|:443... connected.
ERROR: The certificate of 'cdn.rfxn.com' is not trusted.
ERROR: The certificate of 'cdn.rfxn.com' hasn't got a known issuer.
kero242 commented 7 years ago

I installed ca-certificates but same error with Debian 7. So i removed temporarily the s from https in internals.conf.

PSiAU commented 7 years ago

Thanks for the workaround @kero242 . I got bit by this too.

psvec commented 7 years ago

I replaced in /usr/local/maldetect/internals/internals.conf wget=which wget 2> /dev/null to wget=which curl 2> /dev/null and than in /usr/local/maldetect/internals/functions in all wget calls you should delete options related to wget and leave just option to curl. You should delete "--referer="$lmd_referer" $wgetop" and change "-O to -o"

PSiAU commented 7 years ago

The https to http change in internals.conf got reverted back, presumably due to maldet detecting a modified file :( Short of making the file immutable, could we just have https reverted back to http until this is resolved pretty please?

rfxn commented 7 years ago

As an aside @PSiAU , @taylorskalyo @psvec and @kero242, the following sysconfig files are sourced into LMD after internals.conf. As such, you can redefine ANY variable within these files and they will be preserved across updates/upgrades.

maldet:

if [ -f "/etc/sysconfig/maldet" ]; then
        syscnf=/etc/sysconfig/maldet
elif [ -f "/etc/default/maldet" ]; then
        syscnf=/etc/default/maldet
fi
.....
if [ -f "$syscnf" ]; then
        source $syscnf
fi
rfxn commented 7 years ago

As noted previously, this issue inherently is the TLS versioning that Cloudflare is enforcing on the 'cdn.rfxn.com' domain, which looks to be at free tier v1.2 or newer. That is an incredibly strict standard to enforce where even semi-modern servers would not meet it.

https://support.cloudflare.com/hc/en-us/articles/200933580-What-cipher-suites-does-Cloudflare-use-for-SSL-

As of commit d1ae53417fa6c2e4a51750009c53b177e3dde52b , there is now support for curl as default with wget as secondary. This, combined with disabling cloudflare as the CDN seems to allow Debian 6 and 7 to operate without issue out-of-the-box.

waja commented 7 years ago

As of commit d1ae534 , there is now support for curl as default with wget as secondary. This, combined with disabling cloudflare as the CDN seems to allow Debian 6 and 7 to operate without issue out-of-the-box.

Do you think it would be a good idea to release this soon as minor release of 1.6? Or maybe at least a RC version of it for easier testing.

rfxn commented 7 years ago

@waja @PSiAU 1.6-1.rc1 RC is up, testing would be a huge help! https://github.com/rfxn/linux-malware-detect/releases/tag/1.6-1.rc1

Thanks

waja commented 7 years ago

1.6-1.rc1 RC is up, testing would be a huge help!

This is why I asked for a new tarball. :)

Looks fine on Debian 6 with curl installed:

maldet(19509): {sigup} performing signature update check...
maldet(19509): {sigup} could not determine signature version
maldet(19509): {sigup} signature files missing or corrupted, forcing update...
maldet(19509): {sigup} new signature set (2017051530038) available
maldet(19509): {sigup} downloading https://cdn.rfxn.com/downloads/maldet-sigpack.tgz
maldet(19509): {sigup} downloading https://cdn.rfxn.com/downloads/maldet-cleanv2.tgz
maldet(19509): {sigup} verified md5sum of maldet-sigpack.tgz
maldet(19509): {sigup} unpacked and installed maldet-sigpack.tgz
maldet(19509): {sigup} verified md5sum of maldet-clean.tgz
maldet(19509): {sigup} unpacked and installed maldet-clean.tgz
maldet(19509): {sigup} signature set update completed
maldet(19509): {sigup} 16561 signatures (13831 MD5 | 1951 HEX | 779 YARA | 0 USER)

Many thanks for fixing this. Very appreciated!

rfxn commented 7 years ago

1.6.1 is now up live, think that takes care of this issue. Closing but anyone feel free to reopen if issues persist.

Thanks!

philippetrov commented 7 years ago

`root@srv2:/ # maldet -u Linux Malware Detect v1.6.1 (C) 2002-2017, R-fx Networks proj@rfxn.com (C) 2017, Ryan MacDonald ryan@rfxn.com This program may be freely redistributed under the terms of the GNU GPL v2

maldet(1303): {sigup} performing signature update check... maldet(1303): {sigup} could not determine signature version

root@srv2:/ # uname -v FreeBSD 11.0-RELEASE-p2 #0: Mon Oct 24 06:55:27 UTC 2016 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC`

byatis commented 6 years ago

On CentOS release 5.11 (Final): Linux Malware Detect v1.6.2 (C) 2002-2017, R-fx Networks proj@rfxn.com (C) 2017, Ryan MacDonald ryan@rfxn.com This program may be freely redistributed under the terms of the GNU GPL v2

maldet(7174): {sigup} performing signature update check... maldet(7174): {sigup} local signature set is version 2017070716978 maldet(7174): {sigup} could not download https://cdn.rfxn.com/downloads/maldet.sigs.ver, please try again later. maldet(7174): {sigup} could not download signature data from server, please try again later.

if I replace https to http from internals.conf it works.