Minemeld on Alpine linux

[mterron]

I've been trying to get Minemeld running on a different distro without much success. I have compiled the latest version without problems, but can't get it to run. I see that the published Docker container based on Ubuntu have similar problems (same errors when running). This is the output of a run on a clean built system, any ideas? It'd be really useful if there was a guide explaining how all the parts of minemeld (mm-run, mm-traced and the web-ui) are put together in addition to the ansible script. Also, is mm-run the engine and mm-traced the API or ?? Can we have consistent naming so that the engine becomes mm-engine and the api mm-api, etc?

~ # mm-run /etc/minemeld/
2017-07-01T06:57:45 (522)launcher.main INFO: Starting mm-run.py version 0.9.40
2017-07-01T06:57:45 (522)launcher.main INFO: mm-run.py arguments: Namespace(config='/etc/minemeld/', multiprocessing=0, nodes_per_chassis=15.0, verbose=False)
2017-07-01T06:57:55 (522)config._load_config_from_dir INFO: Switching to candidate config
2017-07-01T06:57:55 (522)config._load_config_from_dir INFO: Changes in config: []
2017-07-01T06:57:55 (522)config._destroy_old_nodes INFO: Destroyed nodes: []
2017-07-01T06:57:55 (522)launcher.main INFO: mm-run.py config: _Config(nodes={'spamhaus_EDROP': {'output': True, 'config': {'indicator': {'regex': '^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}/[0-9]{1,2}'}, 'source_name': 'spamhaus.EDROP', 'age_out': {'default': None, 'sudden_death': True, 'interval': 677}, 'url': 'https://www.spamhaus.org/drop/edrop.txt', 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '^;.*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'dshield_blocklist': {'output': True, 'config': {'indicator': {'regex': '^([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})\\t([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})', 'transform': '\\1-\\2'}, 'source_name': 'dshield.block', 'age_out': {'default': None, 'sudden_death': True, 'interval': 257}, 'url': 'https://www.dshield.org/block.txt', 'fields': {'dshield_name': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t([^\\t]+)', 'transform': '\\1'}, 'dshield_country': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t[^\\t]+\\t([A-Z]+)', 'transform': '\\1'}, 'dshield_nattacks': {'regex': '^.*\\t.*\\t[0-9]+\\t([0-9]+)', 'transform': '\\1'}, 'dshield_email': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t[^\\t]+\\t[A-Z]+\\t(\\S+)', 'transform': '\\1'}}, 'interval': 619, 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '[#S].*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'inboundaggregator': {'inputs': ['spamhaus_DROP', 'spamhaus_EDROP', 'dshield_blocklist', 'wlWhiteListIPv4'], 'config': {'whitelist_prefixes': ['wl'], 'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ["type == 'IPv4'", "direction == 'inbound'"], 'name': 'accept inbound IPv4', 'actions': ['accept']}, {'conditions': ["type == 'IPv4'", 'direction == null'], 'name': 'accept generic IPv4', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.ipop.AggregateIPv4FT', 'output': True}, 'inboundfeedhc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence > 75', "share_level == 'green'"], 'name': 'accept confidence > 75 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}, 'spamhaus_DROP': {'output': True, 'config': {'indicator': {'regex': '^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}/[0-9]{1,2}'}, 'source_name': 'spamhaus.DROP', 'age_out': {'default': None, 'sudden_death': True, 'interval': 677}, 'url': 'https://www.spamhaus.org/drop/drop.txt', 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '^;.*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'wlWhiteListIPv4': {'inputs': [], 'config': {'attributes': {'confidence': 100, 'share_level': 'red'}, 'interval': 3600, 'age_out': {'default': None, 'sudden_death': True, 'interval': 67}}, 'class': 'minemeld.ft.local.YamlIPv4FT', 'output': True}, 'inboundfeedlc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence < 50', "share_level == 'green'"], 'name': 'accept confidence < 50 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}, 'inboundfeedmc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence >= 50', 'confidence < 75', "share_level == 'green'"], 'name': 'accept confidence 50-75 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}}, fabric={'config': {'priority': -2, 'num_connections': 50}, 'class': 'AMQP'}, mgmtbus={'slave': {}, 'master': {}, 'transport': {'config': {'priority': 2, 'num_connections': 10}, 'class': 'AMQP'}}, changes=[])
2017-07-01T06:57:55 (522)launcher.main INFO: multiprocessing: #cores: 2
2017-07-01T06:57:55 (522)launcher.main INFO: multiprocessing: max #chassis: 2
2017-07-01T06:57:55 (522)launcher.main INFO: Number of chassis: 1
2017-07-01T06:57:55 (527)loader.load INFO: Loading minemeld_nodes:minemeld.ft.http.HttpFT
2017-07-01T06:57:56 (527)base.read_checkpoint ERROR: spamhaus_EDROP - Error reading last checkpoint
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/minemeld_core-0.9.40-py2.7-linux-x86_64.egg/minemeld/ft/base.py", line 255, in read_checkpoint
    with open(self.name+'.chkp', 'r') as f:
IOError: [Errno 2] No such file or directory: 'spamhaus_EDROP.chkp'
2017-07-01T06:57:56 (527)base.state INFO: spamhaus_EDROP - transitioning to state 1
2017-07-01T06:57:56 (527)loader.load INFO: Loading minemeld_nodes:minemeld.ft.http.HttpFT
2017-07-01T06:57:56 (527)base.read_checkpoint ERROR: dshield_blocklist - Error reading last checkpoint
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/minemeld_core-0.9.40-py2.7-linux-x86_64.egg/minemeld/ft/base.py", line 255, in read_checkpoint
    with open(self.name+'.chkp', 'r') as f:
IOError: [Errno 2] No such file or directory: 'dshield_blocklist.chkp'
2017-07-01T06:57:56 (527)base.state INFO: dshield_blocklist - transitioning to state 1
2017-07-01T06:57:56 (527)loader.load INFO: Loading minemeld_nodes:minemeld.ft.redis.RedisSet
2017-07-01T06:57:56 (527)base.connect INFO: inboundfeedlc - requesting fabric sub channel for inboundaggregator
2017-07-01T06:57:56 (527)base.state INFO: inboundfeedlc - transitioning to state 1
2017-07-01T06:57:56 (527)loader.load INFO: Loading minemeld_nodes:minemeld.ft.redis.RedisSet
2017-07-01T06:57:56 (527)base.connect INFO: inboundfeedhc - requesting fabric sub channel for inboundaggregator
2017-07-01T06:57:56 (527)base.state INFO: inboundfeedhc - transitioning to state 1
2017-07-01T06:57:56 (527)loader.load INFO: Loading minemeld_nodes:minemeld.ft.http.HttpFT
2017-07-01T06:57:56 (527)base.read_checkpoint ERROR: spamhaus_DROP - Error reading last checkpoint
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/minemeld_core-0.9.40-py2.7-linux-x86_64.egg/minemeld/ft/base.py", line 255, in read_checkpoint
    with open(self.name+'.chkp', 'r') as f:
IOError: [Errno 2] No such file or directory: 'spamhaus_DROP.chkp'
2017-07-01T06:57:56 (527)base.state INFO: spamhaus_DROP - transitioning to state 1
2017-07-01T06:57:56 (527)loader.load INFO: Loading minemeld_nodes:minemeld.ft.local.YamlIPv4FT
2017-07-01T06:57:56 (527)base.read_checkpoint ERROR: wlWhiteListIPv4 - Error reading last checkpoint
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/minemeld_core-0.9.40-py2.7-linux-x86_64.egg/minemeld/ft/base.py", line 255, in read_checkpoint
    with open(self.name+'.chkp', 'r') as f:
IOError: [Errno 2] No such file or directory: 'wlWhiteListIPv4.chkp'
2017-07-01T06:57:56 (527)base.state INFO: wlWhiteListIPv4 - transitioning to state 1
2017-07-01T06:57:56 (527)loader.load INFO: Loading minemeld_nodes:minemeld.ft.ipop.AggregateIPv4FT
2017-07-01T06:57:56 (527)base.read_checkpoint ERROR: inboundaggregator - Error reading last checkpoint
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/minemeld_core-0.9.40-py2.7-linux-x86_64.egg/minemeld/ft/base.py", line 255, in read_checkpoint
    with open(self.name+'.chkp', 'r') as f:
IOError: [Errno 2] No such file or directory: 'inboundaggregator.chkp'
2017-07-01T06:57:56 (527)base.connect INFO: inboundaggregator - requesting fabric sub channel for spamhaus_DROP
2017-07-01T06:57:56 (527)base.connect INFO: inboundaggregator - requesting fabric sub channel for spamhaus_EDROP
2017-07-01T06:57:56 (527)base.connect INFO: inboundaggregator - requesting fabric sub channel for dshield_blocklist
2017-07-01T06:57:56 (527)base.connect INFO: inboundaggregator - requesting fabric sub channel for wlWhiteListIPv4
2017-07-01T06:57:56 (527)base.state INFO: inboundaggregator - transitioning to state 1
2017-07-01T06:57:56 (527)loader.load INFO: Loading minemeld_nodes:minemeld.ft.redis.RedisSet
2017-07-01T06:57:56 (527)base.connect INFO: inboundfeedmc - requesting fabric sub channel for inboundaggregator
2017-07-01T06:57:56 (527)base.state INFO: inboundfeedmc - transitioning to state 1
2017-07-01T06:57:58 (522)mgmtbus.init_graph INFO: state: {u'mbus:slave:wlWhiteListIPv4': {u'checkpoint': None, u'is_source': True, u'state': 1}, u'mbus:slave:spamhaus_DROP': {u'checkpoint': None, u'is_source': True, u'state': 1}, u'mbus:slave:inboundfeedhc': {u'checkpoint': None, u'is_source': False, u'state': 1}, u'mbus:slave:inboundaggregator': {u'checkpoint': None, u'is_source': False, u'state': 1}, u'mbus:slave:dshield_blocklist': {u'checkpoint': None, u'is_source': True, u'state': 1}, u'mbus:slave:inboundfeedlc': {u'checkpoint': None, u'is_source': False, u'state': 1}, u'mbus:slave:inboundfeedmc': {u'checkpoint': None, u'is_source': False, u'state': 1}, u'mbus:slave:spamhaus_EDROP': {u'checkpoint': None, u'is_source': True, u'state': 1}}
2017-07-01T06:57:58 (522)mgmtbus.init_graph INFO: changes: []
2017-07-01T06:57:58 (522)startupplanner._plan_subgraph INFO: state_info: {u'spamhaus_EDROP': {u'checkpoint': None, u'is_source': True, u'state': 1}, u'dshield_blocklist': {u'checkpoint': None, u'is_source': True, u'state': 1}, u'inboundfeedlc': {u'checkpoint': None, u'is_source': False, u'state': 1}, u'inboundfeedhc': {u'checkpoint': None, u'is_source': False, u'state': 1}, u'spamhaus_DROP': {u'checkpoint': None, u'is_source': True, u'state': 1}, u'wlWhiteListIPv4': {u'checkpoint': None, u'is_source': True, u'state': 1}, u'inboundaggregator': {u'checkpoint': None, u'is_source': False, u'state': 1}, u'inboundfeedmc': {u'checkpoint': None, u'is_source': False, u'state': 1}}
2017-07-01T06:57:58 (522)startupplanner._plan_subgraph INFO: planning for subgraph ['spamhaus_EDROP', 'dshield_blocklist', 'inboundfeedlc', 'inboundfeedhc', 'spamhaus_DROP', 'wlWhiteListIPv4', 'inboundaggregator', 'inboundfeedmc']
2017-07-01T06:57:58 (522)startupplanner._plan_subgraph INFO: No checkpoints, new graph: reset
2017-07-01T06:57:58 (522)mgmtbus.init_graph INFO: spamhaus_EDROP <= reset
2017-07-01T06:57:58 (527)base.state INFO: spamhaus_EDROP - transitioning to state 3
2017-07-01T06:57:58 (522)mgmtbus._status_loop ERROR: Exception in _status_loop
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/minemeld_core-0.9.40-py2.7-linux-x86_64.egg/minemeld/mgmtbus.py", line 413, in _status_loop
    loop_interval
  File "/usr/lib/python2.7/site-packages/minemeld_core-0.9.40-py2.7-linux-x86_64.egg/minemeld/mgmtbus.py", line 350, in _send_collectd_metrics
    interval=interval
  File "/usr/lib/python2.7/site-packages/minemeld_core-0.9.40-py2.7-linux-x86_64.egg/minemeld/collectd.py", line 96, in putval
    self._send_cmd(command)
  File "/usr/lib/python2.7/site-packages/minemeld_core-0.9.40-py2.7-linux-x86_64.egg/minemeld/collectd.py", line 57, in _send_cmd
    self._open_socket()
  File "/usr/lib/python2.7/site-packages/minemeld_core-0.9.40-py2.7-linux-x86_64.egg/minemeld/collectd.py", line 42, in _open_socket
    _socket.connect(self.path)
  File "/usr/lib/python2.7/site-packages/gevent-1.0.2-py2.7-linux-x86_64.egg/gevent/socket.py", line 351, in connect
    raise error(result, strerror(result))
error: [Errno 2] No such file or directory
2017-07-01T06:57:58 (527)base.state INFO: spamhaus_EDROP - transitioning to state 4
2017-07-01T06:57:58 (522)mgmtbus.init_graph INFO: dshield_blocklist <= reset
2017-07-01T06:57:58 (527)base.state INFO: dshield_blocklist - transitioning to state 3
2017-07-01T06:57:58 (527)base.state INFO: dshield_blocklist - transitioning to state 4
2017-07-01T06:57:58 (522)mgmtbus.init_graph INFO: inboundfeedlc <= reset
2017-07-01T06:57:58 (527)base.state INFO: inboundfeedlc - transitioning to state 3
2017-07-01T06:57:58 (527)base.state INFO: inboundfeedlc - transitioning to state 4
2017-07-01T06:57:58 (522)mgmtbus.init_graph INFO: inboundfeedhc <= reset
2017-07-01T06:57:58 (527)base.state INFO: inboundfeedhc - transitioning to state 3
2017-07-01T06:57:58 (527)base.state INFO: inboundfeedhc - transitioning to state 4
2017-07-01T06:57:58 (522)mgmtbus.init_graph INFO: spamhaus_DROP <= reset
2017-07-01T06:57:58 (527)base.state INFO: spamhaus_DROP - transitioning to state 3
2017-07-01T06:57:58 (527)base.state INFO: spamhaus_DROP - transitioning to state 4
2017-07-01T06:57:58 (522)mgmtbus.init_graph INFO: wlWhiteListIPv4 <= reset
2017-07-01T06:57:58 (527)base.state INFO: wlWhiteListIPv4 - transitioning to state 3
2017-07-01T06:57:58 (527)base.state INFO: wlWhiteListIPv4 - transitioning to state 4
2017-07-01T06:57:58 (522)mgmtbus.init_graph INFO: inboundaggregator <= reset
2017-07-01T06:57:58 (527)base.state INFO: inboundaggregator - transitioning to state 3
2017-07-01T06:57:58 (527)base.state INFO: inboundaggregator - transitioning to state 4
2017-07-01T06:57:58 (522)mgmtbus.init_graph INFO: inboundfeedmc <= reset
2017-07-01T06:57:58 (527)base.state INFO: inboundfeedmc - transitioning to state 3
2017-07-01T06:57:58 (527)base.state INFO: inboundfeedmc - transitioning to state 4
2017-07-01T06:57:58 (527)chassis.mgmtbus_start INFO: chassis - start received from mgmtbus
2017-07-01T06:57:58 (527)chassis.start INFO: chassis start called
2017-07-01T06:57:58 (527)base.state INFO: spamhaus_EDROP - transitioning to state 5
2017-07-01T06:57:58 (527)base.state INFO: dshield_blocklist - transitioning to state 5
2017-07-01T06:57:58 (527)base.state INFO: inboundaggregator - transitioning to state 5
2017-07-01T06:57:58 (527)base.state INFO: inboundfeedhc - transitioning to state 5
2017-07-01T06:57:58 (527)base.state INFO: spamhaus_DROP - transitioning to state 5
2017-07-01T06:57:58 (527)base.state INFO: wlWhiteListIPv4 - transitioning to state 5
2017-07-01T06:57:58 (527)base.state INFO: inboundfeedlc - transitioning to state 5
2017-07-01T06:57:58 (527)base.state INFO: inboundfeedmc - transitioning to state 5
2017-07-01T06:57:58 (527)basepoller._actor_loop INFO: spamhaus_EDROP - command: 1498892278644 age_out
2017-07-01T06:57:58 (527)table._query_by_index INFO: Deleted in scan of _age_out: 0
2017-07-01T06:57:58 (527)basepoller._actor_loop INFO: dshield_blocklist - command: 1498892278644 age_out
2017-07-01T06:57:58 (527)table._query_by_index INFO: Deleted in scan of _age_out: 0
2017-07-01T06:57:58 (527)basepoller._actor_loop INFO: spamhaus_DROP - command: 1498892278644 age_out
2017-07-01T06:57:58 (527)table._query_by_index INFO: Deleted in scan of _age_out: 0
2017-07-01T06:57:58 (527)basepoller._actor_loop INFO: wlWhiteListIPv4 - command: 1498892278645 age_out
2017-07-01T06:57:58 (527)table._query_by_index INFO: Deleted in scan of _age_out: 0
2017-07-01T06:57:58 (527)basepoller._huppable_wait INFO: hup is clear: False
2017-07-01T06:57:58 (527)basepoller._huppable_wait INFO: hup is clear: False
2017-07-01T06:57:58 (527)basepoller._huppable_wait INFO: hup is clear: False
2017-07-01T06:57:58 (527)basepoller._actor_loop INFO: dshield_blocklist - command: 1498892278672 poll
2017-07-01T06:57:58 (527)basepoller._polling_loop INFO: Polling dshield_blocklist
2017-07-01T06:57:58 (527)connectionpool._new_conn INFO: Starting new HTTPS connection (1): www.dshield.org
2017-07-01T06:57:58 (527)basepoller._actor_loop INFO: spamhaus_DROP - command: 1498892278674 poll
2017-07-01T06:57:58 (527)basepoller._polling_loop INFO: Polling spamhaus_DROP
2017-07-01T06:57:58 (527)connectionpool._new_conn INFO: Starting new HTTPS connection (1): www.spamhaus.org
2017-07-01T06:57:58 (527)basepoller._actor_loop INFO: wlWhiteListIPv4 - command: 1498892278675 poll
2017-07-01T06:57:58 (527)basepoller._polling_loop INFO: Polling wlWhiteListIPv4
2017-07-01T06:57:58 (527)basepoller._actor_loop INFO: wlWhiteListIPv4 - command: 1498892278675 sudden_death
2017-07-01T06:57:58 (527)basepoller._actor_loop INFO: wlWhiteListIPv4 - command: 1498892278675 age_out
2017-07-01T06:57:58 (527)table._query_by_index INFO: Deleted in scan of _age_out: 0
2017-07-01T06:57:58 (527)basepoller._actor_loop INFO: wlWhiteListIPv4 - command: 1498892278675 gc
2017-07-01T06:57:58 (527)table._query_by_index INFO: Deleted in scan of _withdrawn: 0
2017-07-01T06:57:59 (522)launcher.main INFO: One of the chassis has stopped, exit

[mterron]

Just for completeness sake, this is a run with the Ubuntu based container (jtschichold/minemeld)

root@e17b9142d88b:/opt/minemeld# engine/current/bin/mm-run /opt/minemeld/local/config/
2017-07-02T23:53:47 (419)launcher.main INFO: Starting mm-run.py version 0.9.40
2017-07-02T23:53:47 (419)launcher.main INFO: mm-run.py arguments: Namespace(config='/opt/minemeld/local/config/', multiprocessing=0, nodes_per_chassis=15.0, verbose=False)
2017-07-02T23:53:54 (419)config._load_config_from_dir INFO: Switching to candidate config
2017-07-02T23:53:54 (419)config._load_config_from_dir INFO: Changes in config: []
2017-07-02T23:53:54 (419)config._destroy_old_nodes INFO: Destroyed nodes: []
2017-07-02T23:53:54 (419)launcher.main INFO: mm-run.py config: _Config(nodes={'spamhaus_EDROP': {'output': True, 'config': {'indicator': {'regex': '^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}/[0-9]{1,2}'}, 'source_name': 'spamhaus.EDROP', 'age_out': {'default': None, 'sudden_death': True, 'interval': 677}, 'url': 'https://www.spamhaus.org/drop/edrop.txt', 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '^;.*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'dshield_blocklist': {'output': True, 'config': {'indicator': {'regex': '^([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})\\t([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})', 'transform': '\\1-\\2'}, 'source_name': 'dshield.block', 'age_out': {'default': None, 'sudden_death': True, 'interval': 257}, 'url': 'https://www.dshield.org/block.txt', 'fields': {'dshield_name': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t([^\\t]+)', 'transform': '\\1'}, 'dshield_country': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t[^\\t]+\\t([A-Z]+)', 'transform': '\\1'}, 'dshield_nattacks': {'regex': '^.*\\t.*\\t[0-9]+\\t([0-9]+)', 'transform': '\\1'}, 'dshield_email': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t[^\\t]+\\t[A-Z]+\\t(\\S+)', 'transform': '\\1'}}, 'interval': 619, 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '[#S].*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'inboundaggregator': {'inputs': ['spamhaus_DROP', 'spamhaus_EDROP', 'dshield_blocklist', 'wlWhiteListIPv4'], 'config': {'whitelist_prefixes': ['wl'], 'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ["type == 'IPv4'", "direction == 'inbound'"], 'name': 'accept inbound IPv4', 'actions': ['accept']}, {'conditions': ["type == 'IPv4'", 'direction == null'], 'name': 'accept generic IPv4', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.ipop.AggregateIPv4FT', 'output': True}, 'inboundfeedhc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence > 75', "share_level == 'green'"], 'name': 'accept confidence > 75 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}, 'spamhaus_DROP': {'output': True, 'config': {'indicator': {'regex': '^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}/[0-9]{1,2}'}, 'source_name': 'spamhaus.DROP', 'age_out': {'default': None, 'sudden_death': True, 'interval': 677}, 'url': 'https://www.spamhaus.org/drop/drop.txt', 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '^;.*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'wlWhiteListIPv4': {'inputs': [], 'config': {'attributes': {'confidence': 100, 'share_level': 'red'}, 'interval': 3600, 'age_out': {'default': None, 'sudden_death': True, 'interval': 67}}, 'class': 'minemeld.ft.local.YamlIPv4FT', 'output': True}, 'inboundfeedlc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence < 50', "share_level == 'green'"], 'name': 'accept confidence < 50 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}, 'inboundfeedmc': {'inputs': ['inboundaggregator'], 'config': {'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ['confidence >= 50', 'confidence < 75', "share_level == 'green'"], 'name': 'accept confidence 50-75 and share level green', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.redis.RedisSet', 'output': False}}, fabric={'config': {'priority': -2, 'num_connections': 50}, 'class': 'AMQP'}, mgmtbus={'slave': {}, 'master': {}, 'transport': {'config': {'priority': 2, 'num_connections': 10}, 'class': 'AMQP'}}, changes=[])
2017-07-02T23:53:54 (419)launcher.main INFO: multiprocessing: #cores: 2
2017-07-02T23:53:54 (419)launcher.main INFO: multiprocessing: max #chassis: 2
2017-07-02T23:53:54 (419)launcher.main INFO: Number of chassis: 1
2017-07-02T23:53:54 (426)loader.load INFO: Loading minemeld_nodes:minemeld.ft.http.HttpFT
2017-07-02T23:53:55 (426)base.read_checkpoint ERROR: spamhaus_EDROP - Error reading last checkpoint
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.40/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 255, in read_checkpoint
    with open(self.name+'.chkp', 'r') as f:
IOError: [Errno 2] No such file or directory: 'spamhaus_EDROP.chkp'
2017-07-02T23:53:55 (426)base.state INFO: spamhaus_EDROP - transitioning to state 1
2017-07-02T23:53:55 (426)loader.load INFO: Loading minemeld_nodes:minemeld.ft.http.HttpFT
2017-07-02T23:53:55 (426)base.read_checkpoint ERROR: dshield_blocklist - Error reading last checkpoint
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.40/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 255, in read_checkpoint
    with open(self.name+'.chkp', 'r') as f:
IOError: [Errno 2] No such file or directory: 'dshield_blocklist.chkp'
2017-07-02T23:53:55 (426)base.state INFO: dshield_blocklist - transitioning to state 1
2017-07-02T23:53:55 (426)loader.load INFO: Loading minemeld_nodes:minemeld.ft.redis.RedisSet
2017-07-02T23:53:55 (426)base.connect INFO: inboundfeedlc - requesting fabric sub channel for inboundaggregator
2017-07-02T23:53:55 (426)base.state INFO: inboundfeedlc - transitioning to state 1
2017-07-02T23:53:55 (426)loader.load INFO: Loading minemeld_nodes:minemeld.ft.redis.RedisSet
2017-07-02T23:53:55 (426)base.connect INFO: inboundfeedhc - requesting fabric sub channel for inboundaggregator
2017-07-02T23:53:55 (426)base.state INFO: inboundfeedhc - transitioning to state 1
2017-07-02T23:53:55 (426)loader.load INFO: Loading minemeld_nodes:minemeld.ft.http.HttpFT
2017-07-02T23:53:55 (426)base.read_checkpoint ERROR: spamhaus_DROP - Error reading last checkpoint
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.40/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 255, in read_checkpoint
    with open(self.name+'.chkp', 'r') as f:
IOError: [Errno 2] No such file or directory: 'spamhaus_DROP.chkp'
2017-07-02T23:53:55 (426)base.state INFO: spamhaus_DROP - transitioning to state 1
2017-07-02T23:53:55 (426)loader.load INFO: Loading minemeld_nodes:minemeld.ft.local.YamlIPv4FT
2017-07-02T23:53:55 (426)base.read_checkpoint ERROR: wlWhiteListIPv4 - Error reading last checkpoint
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.40/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 255, in read_checkpoint
    with open(self.name+'.chkp', 'r') as f:
IOError: [Errno 2] No such file or directory: 'wlWhiteListIPv4.chkp'
2017-07-02T23:53:55 (426)base.state INFO: wlWhiteListIPv4 - transitioning to state 1
2017-07-02T23:53:55 (426)loader.load INFO: Loading minemeld_nodes:minemeld.ft.ipop.AggregateIPv4FT
2017-07-02T23:53:56 (426)base.read_checkpoint ERROR: inboundaggregator - Error reading last checkpoint
Traceback (most recent call last):
  File "/opt/minemeld/engine/0.9.40/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 255, in read_checkpoint
    with open(self.name+'.chkp', 'r') as f:
IOError: [Errno 2] No such file or directory: 'inboundaggregator.chkp'
2017-07-02T23:53:56 (426)base.connect INFO: inboundaggregator - requesting fabric sub channel for spamhaus_DROP
2017-07-02T23:53:56 (426)base.connect INFO: inboundaggregator - requesting fabric sub channel for spamhaus_EDROP
2017-07-02T23:53:56 (426)base.connect INFO: inboundaggregator - requesting fabric sub channel for dshield_blocklist
2017-07-02T23:53:56 (426)base.connect INFO: inboundaggregator - requesting fabric sub channel for wlWhiteListIPv4
2017-07-02T23:53:56 (426)base.state INFO: inboundaggregator - transitioning to state 1
2017-07-02T23:53:56 (426)loader.load INFO: Loading minemeld_nodes:minemeld.ft.redis.RedisSet
2017-07-02T23:53:56 (426)base.connect INFO: inboundfeedmc - requesting fabric sub channel for inboundaggregator
2017-07-02T23:53:56 (426)base.state INFO: inboundfeedmc - transitioning to state 1
2017-07-02T23:53:56 (419)mgmtbus.init_graph INFO: state: {u'mbus:slave:wlWhiteListIPv4': {u'checkpoint': None, u'is_source': True, u'state': 1}, u'mbus:slave:spamhaus_DROP': {u'checkpoint': None, u'is_source': True, u'state': 1}, u'mbus:slave:inboundaggregator': {u'checkpoint': None, u'is_source': False, u'state': 1}, u'mbus:slave:dshield_blocklist': {u'checkpoint': None, u'is_source': True, u'state': 1}, u'mbus:slave:inboundfeedlc': {u'checkpoint': None, u'is_source': False, u'state': 1}, u'mbus:slave:inboundfeedmc': {u'checkpoint': None, u'is_source': False, u'state': 1}, u'mbus:slave:inboundfeedhc': {u'checkpoint': None, u'is_source': False, u'state': 1}, u'mbus:slave:spamhaus_EDROP': {u'checkpoint': None, u'is_source': True, u'state': 1}}
2017-07-02T23:53:56 (419)mgmtbus.init_graph INFO: changes: []
2017-07-02T23:53:56 (419)startupplanner._plan_subgraph INFO: state_info: {u'spamhaus_EDROP': {u'checkpoint': None, u'is_source': True, u'state': 1}, u'dshield_blocklist': {u'checkpoint': None, u'is_source': True, u'state': 1}, u'inboundfeedlc': {u'checkpoint': None, u'is_source': False, u'state': 1}, u'inboundfeedhc': {u'checkpoint': None, u'is_source': False, u'state': 1}, u'spamhaus_DROP': {u'checkpoint': None, u'is_source': True, u'state': 1}, u'wlWhiteListIPv4': {u'checkpoint': None, u'is_source': True, u'state': 1}, u'inboundaggregator': {u'checkpoint': None, u'is_source': False, u'state': 1}, u'inboundfeedmc': {u'checkpoint': None, u'is_source': False, u'state': 1}}
2017-07-02T23:53:56 (419)startupplanner._plan_subgraph INFO: planning for subgraph ['spamhaus_EDROP', 'dshield_blocklist', 'inboundfeedlc', 'inboundfeedhc', 'spamhaus_DROP', 'wlWhiteListIPv4', 'inboundaggregator', 'inboundfeedmc']
2017-07-02T23:53:56 (419)startupplanner._plan_subgraph INFO: No checkpoints, new graph: reset
2017-07-02T23:53:56 (419)mgmtbus.init_graph INFO: spamhaus_EDROP <= reset
2017-07-02T23:53:56 (426)base.state INFO: spamhaus_EDROP - transitioning to state 3
2017-07-02T23:53:57 (426)base.state INFO: spamhaus_EDROP - transitioning to state 4
2017-07-02T23:53:57 (419)mgmtbus.init_graph INFO: dshield_blocklist <= reset
2017-07-02T23:53:57 (426)base.state INFO: dshield_blocklist - transitioning to state 3
2017-07-02T23:53:57 (426)base.state INFO: dshield_blocklist - transitioning to state 4
2017-07-02T23:53:57 (419)mgmtbus.init_graph INFO: inboundfeedlc <= reset
2017-07-02T23:53:57 (426)base.state INFO: inboundfeedlc - transitioning to state 3
2017-07-02T23:53:57 (426)base.state INFO: inboundfeedlc - transitioning to state 4
2017-07-02T23:53:57 (419)mgmtbus.init_graph INFO: inboundfeedhc <= reset
2017-07-02T23:53:57 (426)base.state INFO: inboundfeedhc - transitioning to state 3
2017-07-02T23:53:57 (426)base.state INFO: inboundfeedhc - transitioning to state 4
2017-07-02T23:53:57 (419)mgmtbus.init_graph INFO: spamhaus_DROP <= reset
2017-07-02T23:53:57 (426)base.state INFO: spamhaus_DROP - transitioning to state 3
2017-07-02T23:53:57 (426)base.state INFO: spamhaus_DROP - transitioning to state 4
2017-07-02T23:53:57 (419)mgmtbus.init_graph INFO: wlWhiteListIPv4 <= reset
2017-07-02T23:53:57 (426)base.state INFO: wlWhiteListIPv4 - transitioning to state 3
2017-07-02T23:53:57 (426)base.state INFO: wlWhiteListIPv4 - transitioning to state 4
2017-07-02T23:53:57 (419)mgmtbus.init_graph INFO: inboundaggregator <= reset
2017-07-02T23:53:57 (426)base.state INFO: inboundaggregator - transitioning to state 3
2017-07-02T23:53:57 (426)base.state INFO: inboundaggregator - transitioning to state 4
2017-07-02T23:53:57 (419)mgmtbus.init_graph INFO: inboundfeedmc <= reset
2017-07-02T23:53:57 (426)base.state INFO: inboundfeedmc - transitioning to state 3
2017-07-02T23:53:57 (426)base.state INFO: inboundfeedmc - transitioning to state 4
2017-07-02T23:53:57 (426)chassis.mgmtbus_start INFO: chassis - start received from mgmtbus
2017-07-02T23:53:57 (426)chassis.start INFO: chassis start called
2017-07-02T23:53:57 (426)base.state INFO: spamhaus_EDROP - transitioning to state 5
2017-07-02T23:53:57 (426)base.state INFO: dshield_blocklist - transitioning to state 5
2017-07-02T23:53:57 (426)base.state INFO: inboundaggregator - transitioning to state 5
2017-07-02T23:53:57 (426)base.state INFO: inboundfeedhc - transitioning to state 5
2017-07-02T23:53:57 (426)basepoller._actor_loop INFO: spamhaus_EDROP - command: 1499039637380 age_out
2017-07-02T23:53:57 (426)table._query_by_index INFO: Deleted in scan of _age_out: 0
2017-07-02T23:53:57 (426)basepoller._actor_loop INFO: dshield_blocklist - command: 1499039637380 age_out
2017-07-02T23:53:57 (426)table._query_by_index INFO: Deleted in scan of _age_out: 0
2017-07-02T23:53:57 (426)basepoller._huppable_wait INFO: hup is clear: False
2017-07-02T23:53:57 (426)base.state INFO: spamhaus_DROP - transitioning to state 5
2017-07-02T23:53:57 (426)base.state INFO: wlWhiteListIPv4 - transitioning to state 5
2017-07-02T23:53:57 (426)base.state INFO: inboundfeedlc - transitioning to state 5
2017-07-02T23:53:57 (426)base.state INFO: inboundfeedmc - transitioning to state 5
2017-07-02T23:53:57 (426)basepoller._actor_loop INFO: dshield_blocklist - command: 1499039637394 poll
2017-07-02T23:53:57 (426)basepoller._polling_loop INFO: Polling dshield_blocklist
2017-07-02T23:53:57 (426)connectionpool._new_conn INFO: Starting new HTTPS connection (1): www.dshield.org
2017-07-02T23:53:57 (426)basepoller._actor_loop INFO: spamhaus_DROP - command: 1499039637450 age_out
2017-07-02T23:53:57 (426)table._query_by_index INFO: Deleted in scan of _age_out: 0
2017-07-02T23:53:57 (426)basepoller._actor_loop INFO: wlWhiteListIPv4 - command: 1499039637450 age_out
2017-07-02T23:53:57 (426)table._query_by_index INFO: Deleted in scan of _age_out: 0
2017-07-02T23:53:58 (419)launcher.main INFO: One of the chassis has stopped, exit

[jtschichold]

Hi @mterron, good point about naming engine and API processes, we will fix this in the next major.

The ERRORs about the chkp files are benign, they show up the first time you start the engine with a new node. The problem I see in the Alpine log is that collectd socket is not available. Did collectd install successfully on Alpine ?

[mterron]

Yes, collectd is running. The main issue I see is that the error doesn't tell you what is wrong.

(14063)mgmtbus._status_loop ERROR: Exception in _status_loop
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/minemeld_core-0.9.40-py2.7-linux-x86_64.egg/minemeld/mgmtbus.py", line 413, in _status_loop
    loop_interval
  File "/usr/lib/python2.7/site-packages/minemeld_core-0.9.40-py2.7-linux-x86_64.egg/minemeld/mgmtbus.py", line 350, in _send_collectd_metrics
    interval=interval
  File "/usr/lib/python2.7/site-packages/minemeld_core-0.9.40-py2.7-linux-x86_64.egg/minemeld/collectd.py", line 96, in putval
    self._send_cmd(command)
  File "/usr/lib/python2.7/site-packages/minemeld_core-0.9.40-py2.7-linux-x86_64.egg/minemeld/collectd.py", line 57, in _send_cmd
    self._open_socket()
  File "/usr/lib/python2.7/site-packages/minemeld_core-0.9.40-py2.7-linux-x86_64.egg/minemeld/collectd.py", line 42, in _open_socket
    _socket.connect(self.path)
  File "/usr/lib/python2.7/site-packages/gevent-1.0.2-py2.7-linux-x86_64.egg/gevent/socket.py", line 351, in connect
    raise error(result, strerror(result))
error: [Errno 2] No such file or directory

What file is it trying to open? No idea. The way I see it, I get lots of meaningless errors (if the checkpoint is not mandatory, it shouldn't be an error), maybe output that as debug information. And when you get meaningful errors, like the collectd one, there's no information for troubleshooting.

I managed to find the issue by reading the source and finding that it is looking for the collectd socket at /var/run/collect.sock instead of the default for collectd /usr/var/run/collectd-unixsock. It'd would have been way faster if the error was: error: [Errno 2] /var/run/collectd.sock No such file or directory

After that, the next hurdle is this:

RuntimeError: Error communicating with collectd Type `minemeld_counter' isn't defined.

Any ideas? If the counter is not defined, why don't you define it and carry on?

[davidjmeier]

The Python exception does show the problem. Line 42 of collect.py has failed to make the socket connection and since there is no exception handling in place to retry or terminate gracefully the exception terminates with standard Python exception trace output.

I would suggest making sure collectd is working because if it is then you have a Docker networking problem wherein the service is either exposed incorrectly or not available.

It would also be rather helpful if you provided context into your Dockerfile or docker-compose because the error very much looks to be a problem with your environment.

On Jul 4, 2017 5:57 PM, "Terror" notifications@github.com wrote:

Yes, collectd is running. The main issue I see is that the error doesn't tell you what is wrong.

(14063)mgmtbus._status_loop ERROR: Exception in _status_loop Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/minemeld_core-0.9.40-py2.7-linux-x86_64.egg/minemeld/mgmtbus.py", line 413, in _status_loop loop_interval File "/usr/lib/python2.7/site-packages/minemeld_core-0.9.40-py2.7-linux-x86_64.egg/minemeld/mgmtbus.py", line 350, in _send_collectd_metrics interval=interval File "/usr/lib/python2.7/site-packages/minemeld_core-0.9.40-py2.7-linux-x86_64.egg/minemeld/collectd.py", line 96, in putval self._send_cmd(command) File "/usr/lib/python2.7/site-packages/minemeld_core-0.9.40-py2.7-linux-x86_64.egg/minemeld/collectd.py", line 57, in _send_cmd self._open_socket() File "/usr/lib/python2.7/site-packages/minemeld_core-0.9.40-py2.7-linux-x86_64.egg/minemeld/collectd.py", line 42, in _open_socket _socket.connect(self.path) File "/usr/lib/python2.7/site-packages/gevent-1.0.2-py2.7-linux-x86_64.egg/gevent/socket.py", line 351, in connect raise error(result, strerror(result)) error: [Errno 2] No such file or directory

What file is it trying to open? No idea. The way I see it, I get lots of meaningless errors (if the checkpoint is not mandatory, it shouldn't be an error. Maybe output that as debug information. And when you get meaningful errors, like the collectd one, there's no information for troubleshooting.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/PaloAltoNetworks/minemeld-core/issues/227#issuecomment-312964361, or mute the thread https://github.com/notifications/unsubscribe-auth/AAykEzMzGXM7pcirfS9yYP---3-o5R3Lks5sKsNYgaJpZM4OLt5D .

[mterron]

A socket connection to where @windexh8er? I was trying to make a point, as you see I found what socket by reading the source. Nowhere in the error it tells what it is trying to connect to. From an operators perspective, it is a nightmare.

BTW, this is all running locally, no network involved. I have not run into any Docker issues so far, just minemeld configuration.

Moving on, EUREKA, not in this repo, but in the ansible script repo, there is a file where 2 custom collectd types are defined! After adding those 2 types to collectd's types.db, error is gone. I feel like Indiana Jones right now. I just removed the golden idol and replaced it with the sand bag! Aaaand now there's a giant rock trying to crush me.

Now I see no errors except for the checkpoint ones, but still crashes :( This is the new log:

2017-07-05T01:48:39 (36217)launcher.main INFO: Starting mm-run.py version 0.9.40
2017-07-05T01:48:39 (36217)launcher.main INFO: mm-run.py arguments: Namespace(config='/etc/minemeld/', multiprocessing=0, nodes_per_chassis=15.0, verbose=False)
2017-07-05T01:48:44 (36217)config._load_config_from_dir INFO: Switching to candidate config
2017-07-05T01:48:44 (36217)config._load_config_from_dir INFO: Changes in config: []
2017-07-05T01:48:44 (36217)config._destroy_old_nodes INFO: Destroyed nodes: []
2017-07-05T01:48:44 (36217)launcher.main INFO: mm-run.py config: _Config(nodes={'dshield_blocklist': {'output': True, 'config': {'indicator': {'regex': '^([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})\\t([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})', 'transform': '\\1-\\2'}, 'source_name': 'dshield.block', 'age_out': {'default': None, 'sudden_death': True, 'interval': 257}, 'url': 'https://www.dshield.org/block.txt', 'fields': {'dshield_name': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t([^\\t]+)', 'transform': '\\1'}, 'dshield_country': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t[^\\t]+\\t([A-Z]+)', 'transform': '\\1'}, 'dshield_nattacks': {'regex': '^.*\\t.*\\t[0-9]+\\t([0-9]+)', 'transform': '\\1'}, 'dshield_email': {'regex': '^.*\\t.*\\t[0-9]+\\t[0-9]+\\t[^\\t]+\\t[A-Z]+\\t(\\S+)', 'transform': '\\1'}}, 'interval': 619, 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '[#S].*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'wlWhiteListIPv4': {'inputs': [], 'config': {'attributes': {'confidence': 100, 'share_level': 'red'}, 'interval': 3600, 'age_out': {'default': None, 'sudden_death': True, 'interval': 67}}, 'class': 'minemeld.ft.local.YamlIPv4FT', 'output': True}, 'spamhaus_EDROP': {'output': True, 'config': {'indicator': {'regex': '^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}/[0-9]{1,2}'}, 'source_name': 'spamhaus.EDROP', 'age_out': {'default': None, 'sudden_death': True, 'interval': 677}, 'url': 'https://www.spamhaus.org/drop/edrop.txt', 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '^;.*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'spamhaus_DROP': {'output': True, 'config': {'indicator': {'regex': '^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}/[0-9]{1,2}'}, 'source_name': 'spamhaus.DROP', 'age_out': {'default': None, 'sudden_death': True, 'interval': 677}, 'url': 'https://www.spamhaus.org/drop/drop.txt', 'attributes': {'direction': 'inbound', 'type': 'IPv4', 'confidence': 100, 'share_level': 'green'}, 'ignore_regex': '^;.*'}, 'class': 'minemeld.ft.http.HttpFT'}, 'inboundaggregator': {'inputs': ['dshield_blocklist', 'spamhaus_DROP', 'spamhaus_EDROP', 'wlWhiteListIPv4'], 'config': {'whitelist_prefixes': ['wl'], 'infilters': [{'conditions': ["__method == 'withdraw'"], 'name': 'accept withdraws', 'actions': ['accept']}, {'conditions': ["type == 'IPv4'", "direction == 'inbound'"], 'name': 'accept inbound IPv4', 'actions': ['accept']}, {'conditions': ["type == 'IPv4'", 'direction == null'], 'name': 'accept generic IPv4', 'actions': ['accept']}, {'name': 'drop all', 'actions': ['drop']}]}, 'class': 'minemeld.ft.ipop.AggregateIPv4FT', 'output': True}}, fabric={'config': {'priority': -2, 'num_connections': 50}, 'class': 'AMQP'}, mgmtbus={'slave': {}, 'master': {}, 'transport': {'config': {'priority': 2, 'num_connections': 10}, 'class': 'AMQP'}}, changes=[])
2017-07-05T01:48:44 (36217)launcher.main INFO: multiprocessing: #cores: 2
2017-07-05T01:48:44 (36217)launcher.main INFO: multiprocessing: max #chassis: 2
2017-07-05T01:48:44 (36217)launcher.main INFO: Number of chassis: 1
2017-07-05T01:48:44 (36222)loader.load INFO: Loading minemeld_nodes:minemeld.ft.http.HttpFT
2017-07-05T01:48:45 (36222)base.read_checkpoint ERROR: dshield_blocklist - Error reading last checkpoint
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/minemeld_core-0.9.40-py2.7-linux-x86_64.egg/minemeld/ft/base.py", line 255, in read_checkpoint
    with open(self.name+'.chkp', 'r') as f:
IOError: [Errno 2] No such file or directory: 'dshield_blocklist.chkp'
2017-07-05T01:48:45 (36222)base.state INFO: dshield_blocklist - transitioning to state 1
2017-07-05T01:48:45 (36222)loader.load INFO: Loading minemeld_nodes:minemeld.ft.local.YamlIPv4FT
2017-07-05T01:48:45 (36222)base.read_checkpoint ERROR: wlWhiteListIPv4 - Error reading last checkpoint
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/minemeld_core-0.9.40-py2.7-linux-x86_64.egg/minemeld/ft/base.py", line 255, in read_checkpoint
    with open(self.name+'.chkp', 'r') as f:
IOError: [Errno 2] No such file or directory: 'wlWhiteListIPv4.chkp'
2017-07-05T01:48:45 (36222)base.state INFO: wlWhiteListIPv4 - transitioning to state 1
2017-07-05T01:48:45 (36222)loader.load INFO: Loading minemeld_nodes:minemeld.ft.http.HttpFT
2017-07-05T01:48:45 (36222)base.read_checkpoint ERROR: spamhaus_EDROP - Error reading last checkpoint
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/minemeld_core-0.9.40-py2.7-linux-x86_64.egg/minemeld/ft/base.py", line 255, in read_checkpoint
    with open(self.name+'.chkp', 'r') as f:
IOError: [Errno 2] No such file or directory: 'spamhaus_EDROP.chkp'
2017-07-05T01:48:45 (36222)base.state INFO: spamhaus_EDROP - transitioning to state 1
2017-07-05T01:48:45 (36222)loader.load INFO: Loading minemeld_nodes:minemeld.ft.http.HttpFT
2017-07-05T01:48:45 (36222)base.read_checkpoint ERROR: spamhaus_DROP - Error reading last checkpoint
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/minemeld_core-0.9.40-py2.7-linux-x86_64.egg/minemeld/ft/base.py", line 255, in read_checkpoint
    with open(self.name+'.chkp', 'r') as f:
IOError: [Errno 2] No such file or directory: 'spamhaus_DROP.chkp'
2017-07-05T01:48:45 (36222)base.state INFO: spamhaus_DROP - transitioning to state 1
2017-07-05T01:48:45 (36222)loader.load INFO: Loading minemeld_nodes:minemeld.ft.ipop.AggregateIPv4FT
2017-07-05T01:48:45 (36222)base.read_checkpoint ERROR: inboundaggregator - Error reading last checkpoint
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/minemeld_core-0.9.40-py2.7-linux-x86_64.egg/minemeld/ft/base.py", line 255, in read_checkpoint
    with open(self.name+'.chkp', 'r') as f:
IOError: [Errno 2] No such file or directory: 'inboundaggregator.chkp'
2017-07-05T01:48:45 (36222)base.connect INFO: inboundaggregator - requesting fabric sub channel for dshield_blocklist
2017-07-05T01:48:45 (36222)base.connect INFO: inboundaggregator - requesting fabric sub channel for spamhaus_DROP
2017-07-05T01:48:45 (36222)base.connect INFO: inboundaggregator - requesting fabric sub channel for spamhaus_EDROP
2017-07-05T01:48:45 (36222)base.connect INFO: inboundaggregator - requesting fabric sub channel for wlWhiteListIPv4
2017-07-05T01:48:45 (36222)base.state INFO: inboundaggregator - transitioning to state 1
2017-07-05T01:48:46 (36217)mgmtbus.init_graph INFO: state: {u'mbus:slave:spamhaus_EDROP': {u'checkpoint': None, u'is_source': True, u'state': 1}, u'mbus:slave:dshield_blocklist': {u'checkpoint': None, u'is_source': True, u'state': 1}, u'mbus:slave:spamhaus_DROP': {u'checkpoint': None, u'is_source': True, u'state': 1}, u'mbus:slave:wlWhiteListIPv4': {u'checkpoint': None, u'is_source': True, u'state': 1}, u'mbus:slave:inboundaggregator': {u'checkpoint': None, u'is_source': False, u'state': 1}}
2017-07-05T01:48:46 (36217)mgmtbus.init_graph INFO: changes: []
2017-07-05T01:48:46 (36217)startupplanner._plan_subgraph INFO: state_info: {u'dshield_blocklist': {u'checkpoint': None, u'is_source': True, u'state': 1}, u'wlWhiteListIPv4': {u'checkpoint': None, u'is_source': True, u'state': 1}, u'spamhaus_EDROP': {u'checkpoint': None, u'is_source': True, u'state': 1}, u'spamhaus_DROP': {u'checkpoint': None, u'is_source': True, u'state': 1}, u'inboundaggregator': {u'checkpoint': None, u'is_source': False, u'state': 1}}
2017-07-05T01:48:46 (36217)startupplanner._plan_subgraph INFO: planning for subgraph ['dshield_blocklist', 'wlWhiteListIPv4', 'spamhaus_DROP', 'spamhaus_EDROP', 'inboundaggregator']
2017-07-05T01:48:46 (36217)startupplanner._plan_subgraph INFO: No checkpoints, new graph: reset
2017-07-05T01:48:46 (36217)mgmtbus.init_graph INFO: dshield_blocklist <= reset
2017-07-05T01:48:46 (36222)base.state INFO: dshield_blocklist - transitioning to state 3
2017-07-05T01:48:46 (36222)base.state INFO: dshield_blocklist - transitioning to state 4
2017-07-05T01:48:46 (36217)mgmtbus.init_graph INFO: wlWhiteListIPv4 <= reset
2017-07-05T01:48:46 (36222)base.state INFO: wlWhiteListIPv4 - transitioning to state 3
2017-07-05T01:48:46 (36222)base.state INFO: wlWhiteListIPv4 - transitioning to state 4
2017-07-05T01:48:46 (36217)mgmtbus.init_graph INFO: spamhaus_DROP <= reset
2017-07-05T01:48:46 (36222)base.state INFO: spamhaus_DROP - transitioning to state 3
2017-07-05T01:48:46 (36222)base.state INFO: spamhaus_DROP - transitioning to state 4
2017-07-05T01:48:46 (36217)mgmtbus.init_graph INFO: spamhaus_EDROP <= reset
2017-07-05T01:48:46 (36222)base.state INFO: spamhaus_EDROP - transitioning to state 3
2017-07-05T01:48:46 (36222)base.state INFO: spamhaus_EDROP - transitioning to state 4
2017-07-05T01:48:46 (36217)mgmtbus.init_graph INFO: inboundaggregator <= reset
2017-07-05T01:48:47 (36222)base.state INFO: inboundaggregator - transitioning to state 3
2017-07-05T01:48:47 (36222)base.state INFO: inboundaggregator - transitioning to state 4
2017-07-05T01:48:47 (36222)chassis.mgmtbus_start INFO: chassis - start received from mgmtbus
2017-07-05T01:48:47 (36222)chassis.start INFO: chassis start called
2017-07-05T01:48:47 (36222)base.state INFO: dshield_blocklist - transitioning to state 5
2017-07-05T01:48:47 (36222)base.state INFO: wlWhiteListIPv4 - transitioning to state 5
2017-07-05T01:48:47 (36222)base.state INFO: spamhaus_EDROP - transitioning to state 5
2017-07-05T01:48:47 (36222)base.state INFO: spamhaus_DROP - transitioning to state 5
2017-07-05T01:48:47 (36222)base.state INFO: inboundaggregator - transitioning to state 5
2017-07-05T01:48:47 (36222)basepoller._actor_loop INFO: dshield_blocklist - command: 1499219327133 age_out
2017-07-05T01:48:47 (36222)table._query_by_index INFO: Deleted in scan of _age_out: 0
2017-07-05T01:48:47 (36222)basepoller._actor_loop INFO: wlWhiteListIPv4 - command: 1499219327133 age_out
2017-07-05T01:48:47 (36222)table._query_by_index INFO: Deleted in scan of _age_out: 0
2017-07-05T01:48:47 (36222)basepoller._actor_loop INFO: spamhaus_EDROP - command: 1499219327133 age_out
2017-07-05T01:48:47 (36222)table._query_by_index INFO: Deleted in scan of _age_out: 0
2017-07-05T01:48:47 (36222)basepoller._actor_loop INFO: spamhaus_DROP - command: 1499219327134 age_out
2017-07-05T01:48:47 (36222)table._query_by_index INFO: Deleted in scan of _age_out: 0
2017-07-05T01:48:47 (36222)basepoller._huppable_wait INFO: hup is clear: False
2017-07-05T01:48:47 (36222)basepoller._huppable_wait INFO: hup is clear: False
2017-07-05T01:48:47 (36222)basepoller._actor_loop INFO: spamhaus_EDROP - command: 1499219327160 poll
2017-07-05T01:48:47 (36222)basepoller._polling_loop INFO: Polling spamhaus_EDROP
2017-07-05T01:48:47 (36222)connectionpool._new_conn INFO: Starting new HTTPS connection (1): www.spamhaus.org
2017-07-05T01:48:47 (36222)basepoller._actor_loop INFO: spamhaus_DROP - command: 1499219327164 poll
2017-07-05T01:48:47 (36222)basepoller._polling_loop INFO: Polling spamhaus_DROP
2017-07-05T01:48:47 (36222)connectionpool._new_conn INFO: Starting new HTTPS connection (1): www.spamhaus.org
2017-07-05T01:48:48 (36217)launcher.main INFO: One of the chassis has stopped, exit

Ignoring the checkpoint "errors", the only thing that doesn't look right is the init_graph reset lines. They say INFO and not error, yet reset sounds like something is not right. Either that or the huppable_wait. I'll try to keep debugging it later.

[mterron]

Any ideas?

[jtschichold]

Hi @mterron, the reset lines are normal. It seems you are trying to install minemeld as a traditional python package, but MineMeld is an application and requires a surrounding infrastructure to run. The easiest way to have MineMeld running on Alpine is checking is adding Alpine support on the ansible playbooks, have you already tried that ? https://github.com/PaloAltoNetworks/minemeld-ansible

[mterron]

@jtschichold, I'm running all the dependencies I could find (collectd, redis and rabbitmq). Is there anything I'm missing?

Also I'm not fluent in Ansible so it'll be much easier for me if there was a dependency list somewhere I could use instead of having to learn ansible to deploy MM.

Can anyone take a look at this and point me in some direction? I have no idea why this thing is crashing when all the pre requisites (that I'm aware of) are up and running.

[mterron]

@jtschichold any update on this? Keep trying every now and then, but results are always the same though I'm sure I'm running all the dependencies. No obvious error in the log but the chassis dies.

I'm running RabbitMQ, CollectD, Redis, Supervisord and through it mm-supervisord-listener, mm-trace & mm-run

Latest run in debug mode:

...
2018-01-19T02:45:39 (12415)basepoller._actor_loop INFO: wlWhiteListIPv4 - command: 1516329939891 poll
2018-01-19T02:45:39 (12415)basepoller._polling_loop INFO: Polling wlWhiteListIPv4
2018-01-19T02:45:39 (12415)basepoller._poll DEBUG: wlWhiteListIPv4 - End of polling - #indicators: 0
2018-01-19T02:45:39 (12415)basepoller._actor_loop INFO: wlWhiteListIPv4 - command: 1516329939891 sudden_death
2018-01-19T02:45:39 (12415)basepoller._actor_loop INFO: wlWhiteListIPv4 - command: 1516329939891 age_out
2018-01-19T02:45:39 (12415)table._query_by_index INFO: Deleted in scan of _age_out: 0
2018-01-19T02:45:39 (12415)basepoller._actor_loop INFO: wlWhiteListIPv4 - command: 1516329939891 gc

[mterron]

Found out it was an issue with libffi segfaulting due to missing grsec flags. All sorted, thanks for the help!

[jtschichold]

Hi @mterron, thanks for the update. Would you mind sharing the procedure to install MineMeld on Alpine?

Thanks! luigi

[mterron]

I don't have a stable build yet, you can follow my experiments on github.com/mterron/minemeld

The particular issue I was facing was that Docker drops extended attributes (except for an undocumented list of supported ones) and that was breaking my python installation. To fix it I had to recreate the GRSEC attributes before running the engine: root@host:~ $ setfattr -n user.pax.flags -v E $(which python) /usr/lib/libffi.so.*

[mterron]

@jtschichold You can pull a working version from the Docker hub. The basics seem to be working. Miners are pulling correctly and the processor seem to be aggregating alright. adding a new miner also works. I had to do some rather nasty things to the config files that I'm pulling from the ansible repo and also completely got rid of the virtualenv and installed everything as system packages but since it's a docker container I'm happy with that. There are some pip options to change the installation directory so that could be an improvement to remain close to the official engine path and therefore avoid modifying the supervisor config definitions. Also I'm using native python packages from Alpine's repo whenever I can as it cuts the build time in half in my not so powerful build server.

[mterron]

The docker image is now pretty stable. The only thing that it's broken and I can't figure out why is adding extensions. The git part seems to work as it pulls the different branches. However, installation fails:

I can't find any useful information in the logs unfortunately. There is a file not found but that's as far as it goes:

[2018-02-01 13:11:46 ] [633] [INFO] AUDIT - {"msg": null, "action": "POST /extensions/git-install", "params": [["jsonbody", "{\"ref\": \"0.2\", \"ep\":
 \"https://github.com/PaloAltoNetworks/minemeld-threatconnect.git\"}"]], "user": "admin/admin"}
[2018-02-01 13:11:46 ] [633] [INFO] Executing job mm-jobs-extensions-git-5ba26800-a60d-4a1d-b564-70f2ba3dadd9 - ['/tmp/mm-extension-uploadqelExA'] cwd:
 /tmp/mm-jobs-extensions-git-5ba26800-a60d-4a1d-b564-70f2ba3dadd9ktZUyX logfile: /opt/minemeld/log/mm-jobs-extensions-git-5ba26800-a60d-4a1d-b564-70f2b
a3dadd9.log
[2018-02-01 13:11:46 ] [633] [INFO] redis connection pool: in use: 0 available: 1
127.0.0.1 - - [01/Feb/2018:13:11:46 +1300] "POST /extensions/git-install?_=1517443906 HTTP/1.0" 200 55 "https://127.0.0.1/" "Mozilla/5.0 (Windows NT 10
.0; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0"
[2018-02-01 13:11:47 ] [633] [ERROR] Error starting job mm-jobs-extensions-git-5ba26800-a60d-4a1d-b564-70f2ba3dadd9
Traceback (most recent call last):
  File "/opt/minemeld/engine/core/minemeld/flask/jobs.py", line 113, in _job_monitor_glet
    stderr=subprocess.STDOUT
  File "/usr/lib/python2.7/site-packages/gevent/subprocess.py", line 238, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/site-packages/gevent/subprocess.py", line 756, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

[jtschichold]

Please, could you check the temporary directory to see if there are script starting with mm-extension-*? This is the API call that is failing: https://github.com/PaloAltoNetworks/minemeld-core/blob/master/minemeld/flask/extensionsapi.py#L451

[mterron]

/tmp is empty. Maybe it is created and deleted before my watch picks it? Did you try running the container @jtschichold?

[mterron]

@jtschichold did you have a chance to check this? I'm not sure where to go next in this troubleshooting exercise. Thanks!

[mterron]

I just updated to v0.9.46 but the issue is still there, can't install extensions.

Can you please take a look? I've uploaded a docker image to Docker Hub and also the Dockerfile is in /etc and also on https://github.com/mterron/minemeld

[mterron]

0.9.50 - No changes