search

opnsense / plugins

OPNsense plugin collection
https://opnsense.org/
BSD 2-Clause "Simplified" License
845 stars 639 forks source link

Maltrail plugin just stopped detecting anything #1470

Closed Taomyn closed 5 years ago

Taomyn commented 5 years ago

I've had Maltrail running pretty well from 5th August to 12th August, but since then it's made zero detections.

The service is running, there is nothing in its error log. I restarted the firewall and still the same. I'm using the two test examples from the Maltrail readme:

nslookup morphed.ru.
ping 136.161.101.53

Neither gets picked up.

I'm on OPNsense v19.7.2 and the plugin is v1.0 - Maltrail is monitoring the WAN interface.

Please forum thread for further on this: https://forum.opnsense.org/index.php?topic=13823

Taomyn commented 5 years ago

FYI, I retested after upgrading to 19.7.3 then leaving it over night and trying the two tests above - same thing, nothing gets logged.

mimugmail commented 5 years ago

Hm, also doesn't work on 19.7.1, need to investigate ...

Taomyn commented 5 years ago

Any idea when this might get fixed? Happy to help test any patches.

mimugmail commented 5 years ago

Downgraded to 19.7, nothing found yet. Maybe a limitation of maltrail itself. Have to contact the author.

Taomyn commented 5 years ago

As this is only on the forum and to save the author multiple hops to see the errors, I'll add it here as well:

root@bart:~ # /usr/local/etc/rc.d/opnsense-maltrailsensor stop
Stopping maltrailsensor.
Waiting for PIDS: 41882.
root@bart:~ # python2.7 /usr/local/share/maltrail/sensor.py
Maltrail (sensor) #v0.13.26

[i] using configuration file '/usr/local/share/maltrail/maltrail.conf'
[i] using '/var/log/maltrail' for log storage
[?] at least 384MB of free memory required
[i] using '/root/.maltrail/trails.csv' for trail storage
[i] updating trails (this might take a while)...
 [o] 'https://data.netlab.360.com/feeds/dga/chinad.txt'
 [o] 'https://data.netlab.360.com/feeds/dga/conficker.txt'
 [o] 'https://data.netlab.360.com/feeds/dga/cryptolocker.txt'
 [o] 'https://data.netlab.360.com/feeds/dga/gameover.txt'
 [o] 'https://data.netlab.360.com/feeds/dga/locky.txt'
 [o] 'https://data.netlab.360.com/feeds/dga/necurs.txt'
 [o] 'https://data.netlab.360.com/feeds/dga/tofsee.txt'
 [o] 'https://data.netlab.360.com/feeds/dga/virut.txt'
 [o] 'https://www.abuseipdb.com/statistics'
 [o] 'https://reputation.alienvault.com/reputation.generic'
 [o] 'https://cybercrime-tracker.net/ccam.php'
 [o] 'https://www.badips.com/get/list/any/2?age=7d'
 [o] 'https://osint.bambenekconsulting.com/feeds/c2-dommasterlist-high.txt'
 [o] 'https://osint.bambenekconsulting.com/feeds/dga-feed.txt'
 [o] 'https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/bitcoin_nodes_1d.ipset'
 [o] 'https://raw.githubusercontent.com/stamparm/blackbook/master/blackbook.csv'
 [o] 'https://lists.blocklist.de/lists/all.txt'
 [o] 'https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/botscout_1d.ipset'
 [o] 'http://danger.rulez.sk/projects/bruteforceblocker/blist.php'
 [o] 'https://raw.githubusercontent.com/fox-it/cobaltstrike-extraneous-space/master/cobaltstrike-servers.csv'
 [o] 'https://www.cruzit.com/xxwbl2txt.php'
 [o] 'https://cybercrime-tracker.net/all.php'
 [o] 'https://dataplane.org/*.txt'
 [o] 'https://isc.sans.edu/feeds/suspiciousdomains_Low.txt'
 [o] 'https://feeds.dshield.org/top10-2.txt'
 [o] 'https://rules.emergingthreats.net/open/suricata/rules/botcc.rules'
 [o] 'https://rules.emergingthreats.net/open/suricata/rules/compromised-ips.txt'
 [o] 'https://rules.emergingthreats.net/open/suricata/rules/emerging-dns.rules'
 [o] 'https://feodotracker.abuse.ch/blocklist/?download=domainblocklist'
 [o] 'https://feodotracker.abuse.ch/blocklist/?download=ipblocklist'
 [o] 'https://blocklist.greensnow.co/greensnow.txt'
 [o] 'https://raw.githubusercontent.com/Neo23x0/signature-base/master/iocs/otx-c2-iocs.txt'
 [o] 'https://raw.githubusercontent.com/gwillem/magento-malware-scanner/master/rules/burner-domains.txt'
 [o] 'https://malc0de.com/bl/ZONES'
 [o] 'https://www.malwaredomainlist.com/hostslist/hosts.txt'
 [o] 'http://malwaredomains.lehigh.edu/files/domains.txt'
 [o] 'https://www.maxmind.com/en/high-risk-ip-sample-list'
 [o] 'https://raw.githubusercontent.com/Hestat/minerchk/master/hostslist.txt'
 [o] 'https://www.nothink.org/blacklist/blacklist_malware_irc.txt'
 [o] 'https://openphish.com/feed.txt'
 [o] 'https://palevotracker.abuse.ch/blocklists.php?download=combinedblocklist'
 [o] 'https://cybercrime-tracker.net/ccpmgate.php'
 [o] 'https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/proxylists_1d.ipset'
 [o] 'https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/proxyrss_1d.ipset'
 [o] 'https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/proxyspy_1d.ipset'
 [o] 'https://ransomwaretracker.abuse.ch/downloads/RW_DOMBL.txt'
 [o] 'https://ransomwaretracker.abuse.ch/downloads/RW_IPBL.txt'
 [o] 'https://ransomwaretracker.abuse.ch/downloads/RW_URLBL.txt'
 [o] 'https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/ri_web_proxies_30d.ipset'
 [o] 'https://report.cs.rutgers.edu/DROP/attackers'
 [o] 'https://sblam.com/blacklist.txt'
 [o] 'https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/socks_proxy_7d.ipset'
 [o] 'https://sslbl.abuse.ch/blacklist/sslipblacklist.csv'
 [o] 'https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/sslproxies_1d.ipset'
 [o] 'https://www.talosintelligence.com/feeds/ip-filter.blf'
 [o] 'https://check.torproject.org/cgi-bin/TorBulkExitList.py?ip=1.1.1.1'
 [o] 'https://torstatus.blutmagie.de/ip_list_all.php/Tor_ip_list_ALL.csv'
 [o] 'https://github.com/JR0driguezB/malware_configs'
 [o] 'https://urlhaus.abuse.ch/downloads/text/'
 [o] 'http://www.urlvir.com/export-hosts/'
 [o] 'http://www.voipbl.org/update/'
 [o] 'http://vxvault.net/URL_List.php'
 [o] 'https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist'
 [o] 'https://zeustracker.abuse.ch/blocklist.php?download=badips'
 [o] 'https://zeustracker.abuse.ch/monitor.php?filter=all'
 [o] 'https://zeustracker.abuse.ch/blocklist.php?download=compromised'
 [o] '(static)'
 [o] '(custom)'
[x] something went wrong during remote data retrieval ('(custom)')
[i] update finished
[i] trails stored to '/root/.maltrail/trails.csv'
[i] updating ipcat database...
[i] opening interface 'pppoe0'
[i] setting capture filter 'udp or icmp or (tcp and (tcp[tcpflags] == tcp-syn or port 80 or port 1080 or port 3128 or port 8000 or port 8080 or port 8118))'
[i] preparing capture buffer...
[i] creating 3 more processes (out of total 4)
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/threading.py", line 1071, in run
    self.finished.wait(self.interval)
  File "/usr/local/lib/python2.7/threading.py", line 614, in wait
    self.__cond.wait(timeout)
  File "/usr/local/lib/python2.7/threading.py", line 349, in wait
    endtime = _time() + timeout
TypeError: unsupported operand type(s) for +: 'float' and 'str'

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/threading.py", line 1071, in run
    self.finished.wait(self.interval)
  File "/usr/local/lib/python2.7/threading.py", line 614, in wait
    self.__cond.wait(timeout)
  File "/usr/local/lib/python2.7/threading.py", line 349, in wait
    endtime = _time() + timeout
TypeError: unsupported operand type(s) for +: 'float' and 'str'

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/threading.py", line 1071, in run
    self.finished.wait(self.interval)
  File "/usr/local/lib/python2.7/threading.py", line 614, in wait
    self.__cond.wait(timeout)
  File "/usr/local/lib/python2.7/threading.py", line 349, in wait
    endtime = _time() + timeout
TypeError: unsupported operand type(s) for +: 'float' and 'str'

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/threading.py", line 1071, in run
    self.finished.wait(self.interval)
  File "/usr/local/lib/python2.7/threading.py", line 614, in wait
    self.__cond.wait(timeout)
  File "/usr/local/lib/python2.7/threading.py", line 349, in wait
    endtime = _time() + timeout
TypeError: unsupported operand type(s) for +: 'float' and 'str'

[?] please install 'schedtool' for better CPU scheduling
[o] running...
mimugmail commented 5 years ago

The traces also occur when listening on LAN?

stamparm commented 5 years ago

Can you please give me the content of used maltrail.conf (/usr/local/share/maltrail/maltrail.conf), particularly option UPDATE_PERIOD? It seems to me that it has some strange non-integer value

Taomyn commented 5 years ago

@mimugmail same errors

@stamparm it seems to be empty

# [Server]
HTTP_ADDRESS 192.168.1.1
HTTP_PORT 8338
USE_SSL false

DISABLE_LOCAL_LOG_STORAGE false

SENSOR_NAME $HOSTNAME
CUSTOM_TRAILS_DIR /usr/local/maltrail/trails/custom/
PROCESS_COUNT $CPU_CORES
DISABLE_CPU_AFFINITY false
USE_FEED_UPDATES true
DISABLED_FEEDS turris, ciarmy, policeman, myip
UPDATE_PERIOD 
USE_SERVER_UPDATE_TRAILS false
USE_HEURISTICS true
CHECK_MISSING_HOST false
CHECK_HOST_DOMAINS false
SHOW_DEBUG false
LOG_DIR /var/log/maltrail
MONITOR_INTERFACE pppoe0
CAPTURE_BUFFER 10%
CAPTURE_FILTER udp or icmp or (tcp and (tcp[tcpflags] == tcp-syn or port 80 or port 1080 or port 3128 or port 8000 or port 8080 or port 8118))
USERS

The GUI does have a value set:

Annotation 2019-09-20 112651

When I manually set it to 86400 like the GUI I get this (I switched back to WAN):

Maltrail (sensor) #v0.13.26

[i] using configuration file '/usr/local/share/maltrail/maltrail.conf'
[i] using '/var/log/maltrail' for log storage
[?] at least 384MB of free memory required
[i] using '/root/.maltrail/trails.csv' for trail storage (last modification: 'Thu, 15 Aug 2019 18:49:17 GMT')
[i] updating trails (this might take a while)...
 [o] 'https://data.netlab.360.com/feeds/dga/chinad.txt'
 [o] 'https://data.netlab.360.com/feeds/dga/conficker.txt'
 [o] 'https://data.netlab.360.com/feeds/dga/cryptolocker.txt'
 [o] 'https://data.netlab.360.com/feeds/dga/gameover.txt'
 [o] 'https://data.netlab.360.com/feeds/dga/locky.txt'
 [o] 'https://data.netlab.360.com/feeds/dga/necurs.txt'
 [o] 'https://data.netlab.360.com/feeds/dga/tofsee.txt'
 [o] 'https://data.netlab.360.com/feeds/dga/virut.txt'
 [o] 'https://www.abuseipdb.com/statistics'
 [o] 'https://reputation.alienvault.com/reputation.generic'
 [o] 'https://cybercrime-tracker.net/ccam.php'
 [o] 'https://www.badips.com/get/list/any/2?age=7d'
 [o] 'https://osint.bambenekconsulting.com/feeds/c2-dommasterlist-high.txt'
 [o] 'https://osint.bambenekconsulting.com/feeds/dga-feed.txt'
 [o] 'https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/bitcoin_nodes_1d.ipset'
 [o] 'https://raw.githubusercontent.com/stamparm/blackbook/master/blackbook.csv'
 [o] 'https://lists.blocklist.de/lists/all.txt'
 [o] 'https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/botscout_1d.ipset'
 [o] 'http://danger.rulez.sk/projects/bruteforceblocker/blist.php'
 [o] 'https://raw.githubusercontent.com/fox-it/cobaltstrike-extraneous-space/master/cobaltstrike-servers.csv'
 [o] 'https://www.cruzit.com/xxwbl2txt.php'
 [o] 'https://cybercrime-tracker.net/all.php'
 [o] 'https://dataplane.org/*.txt'
 [o] 'https://isc.sans.edu/feeds/suspiciousdomains_Low.txt'
 [o] 'https://feeds.dshield.org/top10-2.txt'
 [o] 'https://rules.emergingthreats.net/open/suricata/rules/botcc.rules'
 [o] 'https://rules.emergingthreats.net/open/suricata/rules/compromised-ips.txt'
 [o] 'https://rules.emergingthreats.net/open/suricata/rules/emerging-dns.rules'
 [o] 'https://feodotracker.abuse.ch/blocklist/?download=domainblocklist'
 [o] 'https://feodotracker.abuse.ch/blocklist/?download=ipblocklist'
 [o] 'https://blocklist.greensnow.co/greensnow.txt'
 [o] 'https://raw.githubusercontent.com/Neo23x0/signature-base/master/iocs/otx-c2-iocs.txt'
 [o] 'https://raw.githubusercontent.com/gwillem/magento-malware-scanner/master/rules/burner-domains.txt'
 [o] 'https://malc0de.com/bl/ZONES'
 [o] 'https://www.malwaredomainlist.com/hostslist/hosts.txt'
 [o] 'http://malwaredomains.lehigh.edu/files/domains.txt'
 [o] 'https://www.maxmind.com/en/high-risk-ip-sample-list'
 [o] 'https://raw.githubusercontent.com/Hestat/minerchk/master/hostslist.txt'
 [o] 'https://www.nothink.org/blacklist/blacklist_malware_irc.txt'
 [o] 'https://openphish.com/feed.txt'
 [o] 'https://palevotracker.abuse.ch/blocklists.php?download=combinedblocklist'
 [o] 'https://cybercrime-tracker.net/ccpmgate.php'
 [o] 'https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/proxylists_1d.ipset'
 [o] 'https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/proxyrss_1d.ipset'
 [o] 'https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/proxyspy_1d.ipset'
 [o] 'https://ransomwaretracker.abuse.ch/downloads/RW_DOMBL.txt'
 [o] 'https://ransomwaretracker.abuse.ch/downloads/RW_IPBL.txt'
 [o] 'https://ransomwaretracker.abuse.ch/downloads/RW_URLBL.txt'
 [o] 'https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/ri_web_proxies_30d.ipset'
 [o] 'https://report.cs.rutgers.edu/DROP/attackers'
 [o] 'https://sblam.com/blacklist.txt'
 [o] 'https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/socks_proxy_7d.ipset'
 [o] 'https://sslbl.abuse.ch/blacklist/sslipblacklist.csv'
 [o] 'https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/sslproxies_1d.ipset'
 [o] 'https://www.talosintelligence.com/feeds/ip-filter.blf'
 [o] 'https://check.torproject.org/cgi-bin/TorBulkExitList.py?ip=1.1.1.1'
 [o] 'https://torstatus.blutmagie.de/ip_list_all.php/Tor_ip_list_ALL.csv'
 [o] 'https://github.com/JR0driguezB/malware_configs'
 [o] 'https://urlhaus.abuse.ch/downloads/text/'
 [o] 'http://www.urlvir.com/export-hosts/'
 [o] 'http://www.voipbl.org/update/'
 [o] 'http://vxvault.net/URL_List.php'
 [o] 'https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist'
 [o] 'https://zeustracker.abuse.ch/blocklist.php?download=badips'
 [o] 'https://zeustracker.abuse.ch/monitor.php?filter=all'
 [o] 'https://zeustracker.abuse.ch/blocklist.php?download=compromised'
 [o] '(static)'
 [o] '(custom)'
[x] something went wrong during remote data retrieval ('(custom)')

[x] stopping (Ctrl-C pressed)
[i] finished
mimugmail commented 5 years ago

Damn typo :( https://github.com/opnsense/plugins/blob/master/security/maltrail/src/opnsense/service/templates/OPNsense/Maltrail/maltrail.conf#L33

Will fix this.

But does this fix the pppoe problem?

stamparm commented 5 years ago

Note sure. I've commented the later steps here

Taomyn commented 5 years ago

So how can I test this because I changed both /usr/local/share/maltrail/maltrail.conf and the template as per the fix, restarted the service and still nothing is recorded when I use the two tests. Nothing is getting logged in /var/log/maltrail except the error.log which only seems to log the service exits.

Taomyn commented 5 years ago

Ok, I'm confused, how do I restart this correctly so that it rebuilds the .conf because I changed from WAN to LAN, went to the system diagnostics and restarted the maltrail service, and the .conf still says pppoe?

mimugmail commented 5 years ago

When you apply the patch you also need to restart configd service configd restart

Taomyn commented 5 years ago

Did that, the generated file still remains the same

stamparm commented 5 years ago

Can somebody please send me a sample PCAP for a capture from such interface to miroslav@sqlmap.org? I really do need something to work on.

Taomyn commented 5 years ago

I'll ask how to do that if I that's even possible in OPNsense, once I know the patch is effective - at the moment I don't even know if either WAN or LAN is working because the config doesn't change when I switch the interfaces through the GUI

mimugmail commented 5 years ago

@stamparm I'll send you some, needed some time to get access to my home device with pppoe enabled. Can also offer root access if needed.

Taomyn commented 5 years ago

Ok, so I was able to reboot the firewall in lieu of a proper way to restart whatever service is necessary, and it rebuilt the conf file with the LAN interfaces. I then ran the tests and got hits.

I then changed it back to WAN (pppoe) and disabled/enabled the sensor (can we have some restart buttons please), and I saw the config had changed back to "pppoe". I ran the tests and once again nothing is being picked up - I deleted the new log file produced by the previous hits so I could definitely tell if it detected anything. The file has not been recreated.

At least the fix for the update period being blank works.

stamparm commented 5 years ago

@mimugmail okie dokie. First PCAP, then "failback" root if required :)

mimugmail commented 5 years ago

@Taomyn you can manually patch sensor.py: https://github.com/stamparm/maltrail/commit/b06bd4ba69ad3bb58a668862e59c619dfaa66296

Or wait some weeks for maltrail 0.15

MikhailKasimov commented 5 years ago

@Taomyn you can manually patch sensor.py: stamparm/maltrail@b06bd4b

Or wait some weeks for maltrail 0.15

https://github.com/stamparm/maltrail/commit/f921732b8ff9124adf96f9e1e12202486c3eb7de

Taomyn commented 5 years ago

After realising that downloading the new sensor.py file did not work verywell, I manually patched just the four lines into the current version. After that and restarting the sensor, a few minutes later I started getting hits from various systems.

Many thanks to both of you.

mimugmail commented 5 years ago

This one can be closed, 0.15 shipped with 19.7.5 fixes it

mimugmail commented 5 years ago

Close?