search

StamusNetworks / SELKS

A Suricata based IDS/IPS/NSM distro
https://www.stamus-networks.com/open-source/#selks
GNU General Public License v3.0
1.48k stars 285 forks source link

molochpcapread-selks.service - Moloch Pcap Read failed to start #204

Open michal25 opened 4 years ago

michal25 commented 4 years ago

After moloch update (script selks-upgrade_stamus) molochpcapread-selks.service is not able to start

Nov 20 16:05:22 SELKS2 systemd[1]: Started Moloch Viewer. ● molochpcapread-selks.service - Moloch Pcap Read Loaded: loaded (/etc/systemd/system/molochpcapread-selks.service; enabled; vendor preset: enabled) Active: activating (auto-restart) (Result: exit-code) since Wed 2019-11-20 19:28:37 CET; 1min 28s ago Process: 11581 ExecStart=/bin/sh -c /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m --copy --delete -R /data/nsm/ >> /data/moloch/logs/capture.log 2>&1 (code=exited, status=1/FAILURE) Main PID: 11581 (code=exited, status=1/FAILURE)

Nov 20 19:28:37 SELKS2 systemd[1]: molochpcapread-selks.service: Unit entered failed state. Nov 20 19:28:37 SELKS2 systemd[1]: molochpcapread-selks.service: Failed with result 'exit-code'. scirius RUNNING pid 4102, uptime 3:25:51 ii elasticsearch 6.8.4 all Elasticsearch is a distributed RESTful search engine built for the cloud. Reference documentation can be found at https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html and the 'Elasticsearch: The Definitive Guide' book can be found at https://www.elastic.co/guide/en/elasticsearch/guide/current/index.html ii elasticsearch-curator 5.8.1 amd64 Have indices in Elasticsearch? This is the tool for you!\n\nLike a museum curator manages the exhibits and collections on display, \nElasticsearch Curator helps you curate, or manage your indices. ii evebox 1:0.10.2 amd64 no description given ii kibana 6.8.4 amd64 Explore and visualize your Elasticsearch data ii kibana-dashboards-stamus 2019030501 amd64 Kibana 6 dashboard templates. ii logstash 1:6.8.4-1 all An extensible logging pipeline ii moloch 2.1.0-1 amd64 Moloch Full Packet System ii scirius 3.2.0-1 amd64 Django application to manage Suricata ruleset ii suricata 2019082101-0stamus0 amd64 Suricata open source multi-thread IDS/IPS/NSM system. Filesystem Type Size Used Avail Use% Mounted on udev devtmpfs 7.8G 0 7.8G 0% /dev tmpfs tmpfs 1.6G 8.9M 1.6G 1% /run /dev/md0 ext3 887G 37G 806G 5% / tmpfs tmpfs 7.8G 0 7.8G 0% /dev/shm tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup tmpfs tmpfs 1.6G 0 1.6G 0% /run/user/1001

pevma commented 4 years ago

Hi, If you restart the service does it help?

-- Regards, Peter Manev

On 20 Nov 2019, at 19:32, michal25 notifications@github.com wrote:

 After moloch update (script selks-upgrade_stamus) molochpcapread-selks.service is not able to start

Nov 20 16:05:22 SELKS2 systemd[1]: Started Moloch Viewer. ● molochpcapread-selks.service - Moloch Pcap Read Loaded: loaded (/etc/systemd/system/molochpcapread-selks.service; enabled; vendor preset: enabled) Active: activating (auto-restart) (Result: exit-code) since Wed 2019-11-20 19:28:37 CET; 1min 28s ago Process: 11581 ExecStart=/bin/sh -c /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m --copy --delete -R /data/nsm/ >> /data/moloch/logs/capture.log 2>&1 (code=exited, status=1/FAILURE) Main PID: 11581 (code=exited, status=1/FAILURE)

Nov 20 19:28:37 SELKS2 systemd[1]: molochpcapread-selks.service: Unit entered failed state. Nov 20 19:28:37 SELKS2 systemd[1]: molochpcapread-selks.service: Failed with result 'exit-code'. scirius RUNNING pid 4102, uptime 3:25:51 ii elasticsearch 6.8.4 all Elasticsearch is a distributed RESTful search engine built for the cloud. Reference documentation can be found at https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html and the 'Elasticsearch: The Definitive Guide' book can be found at https://www.elastic.co/guide/en/elasticsearch/guide/current/index.html ii elasticsearch-curator 5.8.1 amd64 Have indices in Elasticsearch? This is the tool for you!\n\nLike a museum curator manages the exhibits and collections on display, \nElasticsearch Curator helps you curate, or manage your indices. ii evebox 1:0.10.2 amd64 no description given ii kibana 6.8.4 amd64 Explore and visualize your Elasticsearch data ii kibana-dashboards-stamus 2019030501 amd64 Kibana 6 dashboard templates. ii logstash 1:6.8.4-1 all An extensible logging pipeline ii moloch 2.1.0-1 amd64 Moloch Full Packet System ii scirius 3.2.0-1 amd64 Django application to manage Suricata ruleset ii suricata 2019082101-0stamus0 amd64 Suricata open source multi-thread IDS/IPS/NSM system. Filesystem Type Size Used Avail Use% Mounted on udev devtmpfs 7.8G 0 7.8G 0% /dev tmpfs tmpfs 1.6G 8.9M 1.6G 1% /run /dev/md0 ext3 887G 37G 806G 5% / tmpfs tmpfs 7.8G 0 7.8G 0% /dev/shm tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup tmpfs tmpfs 1.6G 0 1.6G 0% /run/user/1001

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

michal25 commented 4 years ago

The service restart have no effect. OS restart have also no effect..

● molochpcapread-selks.service - Moloch Pcap Read Loaded: loaded (/etc/systemd/system/molochpcapread-selks.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Wed 2019-11-20 19:34:38 CET; 1h 31min ago Process: 11781 ExecStart=/bin/sh -c /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m --copy --delete -R /data/nsm/ >> /data/moloch/logs/capture.log 2>&1 (code=exited, status=1/FAILURE) Main PID: 11781 (code=exited, status=1/FAILURE)

Nov 20 19:33:08 SELKS2 systemd[1]: molochpcapread-selks.service: Unit entered failed state. Nov 20 19:33:08 SELKS2 systemd[1]: molochpcapread-selks.service: Failed with result 'exit-code'. Nov 20 19:34:38 SELKS2 systemd[1]: molochpcapread-selks.service: Service hold-off time over, scheduling restart. Nov 20 19:34:38 SELKS2 systemd[1]: Stopped Moloch Pcap Read. Nov 20 19:34:38 SELKS2 systemd[1]: molochpcapread-selks.service: Start request repeated too quickly. Nov 20 19:34:38 SELKS2 systemd[1]: Failed to start Moloch Pcap Read. Nov 20 19:34:38 SELKS2 systemd[1]: molochpcapread-selks.service: Unit entered failed state. Nov 20 19:34:38 SELKS2 systemd[1]: molochpcapread-selks.service: Failed with result 'exit-code'.

root@SELKS2:~# systemctl restart molochpcapread-selks root@SELKS2:~# selks-health-check_stamus ● molochpcapread-selks.service - Moloch Pcap Read Loaded: loaded (/etc/systemd/system/molochpcapread-selks.service; enabled; vendor preset: enabled) Active: activating (auto-restart) (Result: exit-code) since Wed 2019-11-20 21:07:14 CET; 3s ago Process: 15122 ExecStart=/bin/sh -c /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m --copy --delete -R /data/nsm/ >> /data/moloch/logs/capture.log 2>&1 (code=exited, status=1/FAILURE) Main PID: 15122 (code=exited, status=1/FAILURE)

Nov 20 21:07:14 SELKS2 systemd[1]: molochpcapread-selks.service: Unit entered failed state. Nov 20 21:07:14 SELKS2 systemd[1]: molochpcapread-selks.service: Failed with result 'exit-code'.

pevma commented 4 years ago

Could you please share the full output of the selks-health-check_stamus ?

Also there should be some pointers in - /data/moloch/logs/capture.log

Thank you

-- Regards, Peter Manev

On 20 Nov 2019, at 21:09, michal25 notifications@github.com wrote:

 The service restart have no effect. OS restart have also no effect..

● molochpcapread-selks.service - Moloch Pcap Read Loaded: loaded (/etc/systemd/system/molochpcapread-selks.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Wed 2019-11-20 19:34:38 CET; 1h 31min ago Process: 11781 ExecStart=/bin/sh -c /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m --copy --delete -R /data/nsm/ >> /data/moloch/logs/capture.log 2>&1 (code=exited, status=1/FAILURE) Main PID: 11781 (code=exited, status=1/FAILURE)

Nov 20 19:33:08 SELKS2 systemd[1]: molochpcapread-selks.service: Unit entered failed state. Nov 20 19:33:08 SELKS2 systemd[1]: molochpcapread-selks.service: Failed with result 'exit-code'. Nov 20 19:34:38 SELKS2 systemd[1]: molochpcapread-selks.service: Service hold-off time over, scheduling restart. Nov 20 19:34:38 SELKS2 systemd[1]: Stopped Moloch Pcap Read. Nov 20 19:34:38 SELKS2 systemd[1]: molochpcapread-selks.service: Start request repeated too quickly. Nov 20 19:34:38 SELKS2 systemd[1]: Failed to start Moloch Pcap Read. Nov 20 19:34:38 SELKS2 systemd[1]: molochpcapread-selks.service: Unit entered failed state. Nov 20 19:34:38 SELKS2 systemd[1]: molochpcapread-selks.service: Failed with result 'exit-code'.

root@SELKS2:# systemctl restart molochpcapread-selks root@SELKS2:# selks-health-check_stamus ● molochpcapread-selks.service - Moloch Pcap Read Loaded: loaded (/etc/systemd/system/molochpcapread-selks.service; enabled; vendor preset: enabled) Active: activating (auto-restart) (Result: exit-code) since Wed 2019-11-20 21:07:14 CET; 3s ago Process: 15122 ExecStart=/bin/sh -c /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m --copy --delete -R /data/nsm/ >> /data/moloch/logs/capture.log 2>&1 (code=exited, status=1/FAILURE) Main PID: 15122 (code=exited, status=1/FAILURE)

Nov 20 21:07:14 SELKS2 systemd[1]: molochpcapread-selks.service: Unit entered failed state. Nov 20 21:07:14 SELKS2 systemd[1]: molochpcapread-selks.service: Failed with result 'exit-code'.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

michal25 commented 4 years ago 
root@SELKS2:~# selks-health-check_stamus 
● suricata.service - LSB: Next Generation IDS/IPS
   Loaded: loaded (/etc/init.d/suricata; generated; vendor preset: enabled)
   Active: active (running) since Thu 2019-11-21 02:02:47 CET; 8h ago
     Docs: man:systemd-sysv-generator(8)
  Process: 23192 ExecStop=/etc/init.d/suricata stop (code=exited, status=0/SUCCESS)
  Process: 23209 ExecStart=/etc/init.d/suricata start (code=exited, status=0/SUCCESS)
    Tasks: 10 (limit: 4915)
   CGroup: /system.slice/suricata.service
           └─23217 /usr/bin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid --af-packet -D -v --user=logstash

Nov 21 02:02:47 SELKS2 systemd[1]: Starting LSB: Next Generation IDS/IPS...
Nov 21 02:02:47 SELKS2 suricata[23209]: Starting suricata in IDS (af-packet) mode... done.
Nov 21 02:02:47 SELKS2 systemd[1]: Started LSB: Next Generation IDS/IPS.
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-11-20 16:04:15 CET; 18h ago
     Docs: http://www.elastic.co
 Main PID: 4026 (java)
    Tasks: 95 (limit: 4915)
   CGroup: /system.slice/elasticsearch.service
           ├─4026 /usr/bin/java -Xms4g -Xmx4g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly …et
           └─4181 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Nov 20 16:04:15 SELKS2 systemd[1]: Started Elasticsearch.
Nov 20 16:04:15 SELKS2 elasticsearch[4026]: warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME
● logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-11-20 15:53:36 CET; 19h ago
 Main PID: 2348 (java)
    Tasks: 36 (limit: 4915)
   CGroup: /system.slice/logstash.service
           └─2348 /usr/bin/java -Xms4g -Xmx4g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly …sh

Nov 20 16:04:19 SELKS2 logstash[2348]: [2019-11-20T16:04:19,072][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to ela…8}
Nov 20 16:04:19 SELKS2 logstash[2348]: [2019-11-20T16:04:19,055][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to ela…8}
Nov 20 16:04:19 SELKS2 logstash[2348]: [2019-11-20T16:04:19,212][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to ela…8}
Nov 20 16:04:23 SELKS2 logstash[2348]: [2019-11-20T16:04:23,079][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to de…"}
Nov 20 16:04:27 SELKS2 logstash[2348]: [2019-11-20T16:04:27,209][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to ela…6}
Nov 20 16:04:27 SELKS2 logstash[2348]: [2019-11-20T16:04:27,210][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to ela…6}
Nov 20 16:04:27 SELKS2 logstash[2348]: [2019-11-20T16:04:27,211][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to ela…6}
Nov 20 16:04:27 SELKS2 logstash[2348]: [2019-11-20T16:04:27,218][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to ela…6}
Nov 20 16:04:28 SELKS2 logstash[2348]: [2019-11-20T16:04:28,091][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to de…"}
Nov 20 16:04:33 SELKS2 logstash[2348]: [2019-11-20T16:04:33,100][WARN ][logstash.outputs.elasticsearch] Restored connection to ES insta….1:9200/"}
Hint: Some lines were ellipsized, use -l to show in full.
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-11-20 16:04:15 CET; 18h ago
 Main PID: 4034 (node)
    Tasks: 11 (limit: 4915)
   CGroup: /system.slice/kibana.service
           └─4034 /usr/share/kibana/bin/../node/bin/node --no-warnings --max-http-header-size=65536 /usr/share/kibana/bin/../src/cli -c /etc/ki…ml

Nov 20 16:04:32 SELKS2 kibana[4034]: {"type":"log","@timestamp":"2019-11-20T15:04:32Z","tags":["status","plugin:rollup@6.8.4","info"],"…rmation."}
Nov 20 16:04:32 SELKS2 kibana[4034]: {"type":"log","@timestamp":"2019-11-20T15:04:32Z","tags":["status","plugin:remote_clusters@6.8.4",…rmation."}
Nov 20 16:04:32 SELKS2 kibana[4034]: {"type":"log","@timestamp":"2019-11-20T15:04:32Z","tags":["status","plugin:cross_cluster_replicati…rmation."}
Nov 20 16:04:32 SELKS2 kibana[4034]: {"type":"log","@timestamp":"2019-11-20T15:04:32Z","tags":["status","plugin:reporting@6.8.4","info"…rmation."}
Nov 20 16:04:32 SELKS2 kibana[4034]: {"type":"log","@timestamp":"2019-11-20T15:04:32Z","tags":["info","monitoring-ui","kibana-monitorin…llection"}
Nov 20 16:04:32 SELKS2 kibana[4034]: {"type":"log","@timestamp":"2019-11-20T15:04:32Z","tags":["status","plugin:security@6.8.4","info"]…rmation."}
Nov 20 16:04:32 SELKS2 kibana[4034]: {"type":"log","@timestamp":"2019-11-20T15:04:32Z","tags":["status","plugin:maps@6.8.4","info"],"pi…rmation."}
Nov 20 16:04:32 SELKS2 kibana[4034]: {"type":"log","@timestamp":"2019-11-20T15:04:32Z","tags":["license","info","xpack"],"pid":4034,"me…: active"}
Nov 20 16:04:33 SELKS2 kibana[4034]: {"type":"log","@timestamp":"2019-11-20T15:04:33Z","tags":["listening","info"],"pid":4034,"message"…ost:5601"}
Nov 20 16:04:33 SELKS2 kibana[4034]: {"type":"log","@timestamp":"2019-11-20T15:04:33Z","tags":["status","plugin:spaces@6.8.4","info"],"…rmation."}
Hint: Some lines were ellipsized, use -l to show in full.
● evebox.service - EveBox Server
   Loaded: loaded (/lib/systemd/system/evebox.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-11-20 15:36:19 CET; 19h ago
 Main PID: 625 (evebox)
    Tasks: 10 (limit: 4915)
   CGroup: /system.slice/evebox.service
           └─625 /usr/bin/evebox server

Nov 20 15:36:25 SELKS2 evebox[625]: 2019-11-20 15:36:25 (server.go:350) <Error> -- Failed to ping Elastic Search, delaying startup: : G…on refused
Nov 20 15:36:28 SELKS2 evebox[625]: 2019-11-20 15:36:28 (server.go:350) <Error> -- Failed to ping Elastic Search, delaying startup: : G…on refused
Nov 20 15:36:31 SELKS2 evebox[625]: 2019-11-20 15:36:31 (server.go:350) <Error> -- Failed to ping Elastic Search, delaying startup: : G…on refused
Nov 20 15:36:34 SELKS2 evebox[625]: 2019-11-20 15:36:34 (server.go:350) <Error> -- Failed to ping Elastic Search, delaying startup: : G…on refused
Nov 20 15:36:37 SELKS2 evebox[625]: 2019-11-20 15:36:37 (server.go:353) <Info> -- Connected to Elastic Search (version: 6.8.4)
Nov 20 15:36:37 SELKS2 evebox[625]: 2019-11-20 15:36:37 (elasticsearch.go:199) <Info> -- Found templates [logstash]
Nov 20 15:36:37 SELKS2 evebox[625]: 2019-11-20 15:36:37 (elasticsearch.go:238) <Info> -- Found Elastic Search keyword suffix to be: keyword
Nov 20 15:36:37 SELKS2 evebox[625]: 2019-11-20 15:36:37 (server.go:131) <Info> -- Session reaper started
Nov 20 15:36:37 SELKS2 evebox[625]: 2019-11-20 15:36:37 (server.go:165) <Info> -- Authentication disabled.
Nov 20 15:36:37 SELKS2 evebox[625]: 2019-11-20 15:36:37 (server.go:276) <Info> -- Listening on 0.0.0.0:5636
Hint: Some lines were ellipsized, use -l to show in full.
● molochviewer-selks.service - Moloch Viewer
   Loaded: loaded (/etc/systemd/system/molochviewer-selks.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-11-20 16:05:22 CET; 18h ago
 Main PID: 4370 (sh)
    Tasks: 12 (limit: 4915)
   CGroup: /system.slice/molochviewer-selks.service
           ├─4370 /bin/sh -c /data/moloch/bin/node viewer.js -c /data/moloch/etc/config.ini >> /data/moloch/logs/viewer.log 2>&1
           └─4372 /data/moloch/bin/node viewer.js -c /data/moloch/etc/config.ini

Nov 20 16:05:22 SELKS2 systemd[1]: Started Moloch Viewer.
● molochpcapread-selks.service - Moloch Pcap Read
   Loaded: loaded (/etc/systemd/system/molochpcapread-selks.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2019-11-20 21:13:15 CET; 13h ago
  Process: 15314 ExecStart=/bin/sh -c /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m --copy --delete -R /data/nsm/  >> /data/moloch/logs/capture.log 2>&1 (code=exited, status=1/FAILURE)
 Main PID: 15314 (code=exited, status=1/FAILURE)

Nov 20 21:11:44 SELKS2 systemd[1]: molochpcapread-selks.service: Unit entered failed state.
Nov 20 21:11:44 SELKS2 systemd[1]: molochpcapread-selks.service: Failed with result 'exit-code'.
Nov 20 21:13:15 SELKS2 systemd[1]: molochpcapread-selks.service: Service hold-off time over, scheduling restart.
Nov 20 21:13:15 SELKS2 systemd[1]: Stopped Moloch Pcap Read.
Nov 20 21:13:15 SELKS2 systemd[1]: molochpcapread-selks.service: Start request repeated too quickly.
Nov 20 21:13:15 SELKS2 systemd[1]: Failed to start Moloch Pcap Read.
Nov 20 21:13:15 SELKS2 systemd[1]: molochpcapread-selks.service: Unit entered failed state.
Nov 20 21:13:15 SELKS2 systemd[1]: molochpcapread-selks.service: Failed with result 'exit-code'.
scirius                          RUNNING   pid 4102, uptime 18:49:38
ii  elasticsearch                   6.8.4                          all          Elasticsearch is a distributed RESTful search engine built for the cloud. Reference documentation can be found at https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html and the 'Elasticsearch: The Definitive Guide' book can be found at https://www.elastic.co/guide/en/elasticsearch/guide/current/index.html
ii  elasticsearch-curator           5.8.1                          amd64        Have indices in Elasticsearch? This is the tool for you!\n\nLike a museum curator manages the exhibits and collections on display, \nElasticsearch Curator helps you curate, or manage your indices.
ii  evebox                          1:0.10.2                       amd64        no description given
ii  kibana                          6.8.4                          amd64        Explore and visualize your Elasticsearch data
ii  kibana-dashboards-stamus        2019030501                     amd64        Kibana 6 dashboard templates.
ii  logstash                        1:6.8.4-1                      all          An extensible logging pipeline
ii  moloch                          2.1.0-1                        amd64        Moloch Full Packet System
ii  scirius                         3.2.0-1                        amd64        Django application to manage Suricata ruleset
ii  suricata                        2019082101-0stamus0            amd64        Suricata open source multi-thread IDS/IPS/NSM system.
Filesystem     Type      Size  Used Avail Use% Mounted on
udev           devtmpfs  7.8G     0  7.8G   0% /dev
tmpfs          tmpfs     1.6G  8.9M  1.6G   1% /run
/dev/md0       ext3      887G   33G  809G   4% /
tmpfs          tmpfs     7.8G     0  7.8G   0% /dev/shm
tmpfs          tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs          tmpfs     7.8G     0  7.8G   0% /sys/fs/cgroup
tmpfs          tmpfs     1.6G     0  1.6G   0% /run/user/1001

And /data/moloch/logs/capture.log BINGO! /usr/lib/x86_64-linux-gnu/libssl.so.1.1: version `OPENSSL_1_1_1' not found (required by /data/moloch/bin/moloch-capture)

How to workaround now?

pevma commented 4 years ago

What is the output of dpkg -l |grep ssl ?

michal25 commented 4 years ago 
root@SELKS2:~# dpkg -l |grep ssl
ii  libflac8:amd64                  1.3.2-1                        amd64        Free Lossless Audio Codec - runtime C library
ii  libio-socket-ssl-perl           2.044-1                        all          Perl module implementing object oriented interface to SSL sockets
ii  libnet-smtp-ssl-perl            1.04-1                         all          Perl module providing SSL support to Net::SMTP
ii  libnet-ssleay-perl              1.80-1                         amd64        Perl module for Secure Sockets Layer (SSL)
ii  libssl1.0.2:amd64               1.0.2t-1~deb9u1                amd64        Secure Sockets Layer toolkit - shared libraries
ii  libssl1.1:amd64                 1.1.0l-1~deb9u1                amd64        Secure Sockets Layer toolkit - shared libraries
ii  openssl                         1.1.0l-1~deb9u1                amd64        Secure Sockets Layer toolkit - cryptographic utility
ii  perl-openssl-defaults:amd64     3                              amd64        version compatibility baseline for Perl OpenSSL packages
pevma commented 4 years ago

It seems you need 1.1.1 which is interesting why is it not avail in the distro. Maybe you can try back-porting that package from Buster (https://packages.debian.org/buster/openssl) I would recommend to test it out in a Qa/Test environment first.

michal25 commented 4 years ago

Because /etc/apt/sources.list deb http://ftp.cz.debian.org/debian/ stretch main

and openssl 1.1.1 is deb http://ftp.de.debian.org/debian buster main

pevma commented 4 years ago

But you should just backport that package only - not the whole OS relevant otherwise it will most likely upgrade other stuff too (which may be unwanted in some cases i guess)

michal25 commented 4 years ago

I will try to upgrade the whole OS and the process the script selks-upgrade_stamus.

And will see what happens :-)

michal25 commented 4 years ago

Well, after the full upgrade crashes the package python2-minimal and suricata package not starts (of coure). I will try to workaround and report here.

michal25 commented 4 years ago

Well, in this state, now :-)

Setting up python2-minimal (2.7.16-1) ... dpkg: error processing package python2-minimal (--configure): installed python2-minimal package post-installation script subprocess returned error exit status 1 Errors were encountered while processing: python2-minimal E: Sub-process /usr/bin/dpkg returned an error code (1) scirius: stopped scirius: started

root@SELKS2:~# selks-health-check_stamus ● suricata.service - Suricata IDS/IDP daemon Loaded: loaded (/lib/systemd/system/suricata.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Thu 2019-11-21 13:42:01 CET; 38s ago Docs: man:suricata(8) man:suricatasc(8) https://suricata-ids.org/docs/ Process: 13708 ExecStart=/usr/bin/suricata -D --af-packet -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid (code=exited, status=0/SUCCESS) Main PID: 13709 (code=exited, status=1/FAILURE)

Nov 21 13:42:01 SELKS2 systemd[1]: suricata.service: Main process exited, code=exited, status=1/FAILURE Nov 21 13:42:01 SELKS2 systemd[1]: suricata.service: Failed with result 'exit-code'. Nov 21 13:42:01 SELKS2 systemd[1]: suricata.service: Service RestartSec=100ms expired, scheduling restart. Nov 21 13:42:01 SELKS2 systemd[1]: suricata.service: Scheduled restart job, restart counter is at 5. Nov 21 13:42:01 SELKS2 systemd[1]: Stopped Suricata IDS/IDP daemon. Nov 21 13:42:01 SELKS2 systemd[1]: suricata.service: Start request repeated too quickly. Nov 21 13:42:01 SELKS2 systemd[1]: suricata.service: Failed with result 'exit-code'. Nov 21 13:42:01 SELKS2 systemd[1]: Failed to start Suricata IDS/IDP daemon. ● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2019-11-21 13:41:56 CET; 43s ago Docs: http://www.elastic.co Main PID: 13488 (java) Tasks: 94 (limit: 4915) Memory: 4.4G CGroup: /system.slice/elasticsearch.service ├─13488 /usr/bin/java -Xms4g -Xmx4g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -… └─13648 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Nov 21 13:41:56 SELKS2 systemd[1]: Started Elasticsearch. Nov 21 13:41:56 SELKS2 elasticsearch[13488]: warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME ● logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2019-11-20 15:53:36 CET; 21h ago Main PID: 2348 (java) Tasks: 37 (limit: 4915) Memory: 1.0M CGroup: /system.slice/logstash.service └─2348 /usr/bin/java -Xms4g -Xmx4g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -D…

Nov 21 12:46:59 SELKS2 logstash[2348]: [2019-11-21T12:46:59,885][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elast… Nov 21 12:47:00 SELKS2 logstash[2348]: [2019-11-21T12:47:00,675][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead… Nov 21 12:47:01 SELKS2 logstash[2348]: [2019-11-21T12:47:01,142][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead… Nov 21 12:47:05 SELKS2 logstash[2348]: [2019-11-21T12:47:05,787][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead… Nov 21 12:47:06 SELKS2 logstash[2348]: [2019-11-21T12:47:06,151][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead… Nov 21 12:47:07 SELKS2 logstash[2348]: [2019-11-21T12:47:07,818][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elast… Nov 21 12:47:07 SELKS2 logstash[2348]: [2019-11-21T12:47:07,886][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elast… Nov 21 12:47:07 SELKS2 logstash[2348]: [2019-11-21T12:47:07,954][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elast… Nov 21 12:47:10 SELKS2 logstash[2348]: [2019-11-21T12:47:10,809][WARN ][logstash.outputs.elasticsearch] Restored connection to ES inst…0.1:9200/"} Nov 21 12:47:11 SELKS2 logstash[2348]: [2019-11-21T12:47:11,163][WARN ][logstash.outputs.elasticsearch] Restored connection to ES inst…0.1:9200/"} Hint: Some lines were ellipsized, use -l to show in full. ● kibana.service - Kibana Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2019-11-21 13:41:56 CET; 43s ago Main PID: 13495 (node) Tasks: 11 (limit: 4915) Memory: 244.1M CGroup: /system.slice/kibana.service └─13495 /usr/share/kibana/bin/../node/bin/node --no-warnings --max-http-header-size=65536 /usr/share/kibana/bin/../src/cli -c /etc/kib…

Nov 21 13:42:14 SELKS2 kibana[13495]: {"type":"log","@timestamp":"2019-11-21T12:42:14Z","tags":["status","plugin:remote_clusters@6.8.5…ormation."} Nov 21 13:42:14 SELKS2 kibana[13495]: {"type":"log","@timestamp":"2019-11-21T12:42:14Z","tags":["status","plugin:cross_cluster_replica…ormation."} Nov 21 13:42:14 SELKS2 kibana[13495]: {"type":"log","@timestamp":"2019-11-21T12:42:14Z","tags":["status","plugin:reporting@6.8.5","inf…ormation."} Nov 21 13:42:14 SELKS2 kibana[13495]: {"type":"log","@timestamp":"2019-11-21T12:42:14Z","tags":["info","monitoring-ui","kibana-monitor…ollection"} Nov 21 13:42:14 SELKS2 kibana[13495]: {"type":"log","@timestamp":"2019-11-21T12:42:14Z","tags":["status","plugin:security@6.8.5","info…ormation."} Nov 21 13:42:14 SELKS2 kibana[13495]: {"type":"log","@timestamp":"2019-11-21T12:42:14Z","tags":["status","plugin:maps@6.8.5","info"],"…ormation."} Nov 21 13:42:14 SELKS2 kibana[13495]: {"type":"log","@timestamp":"2019-11-21T12:42:14Z","tags":["license","info","xpack"],"pid":13495,…s: active"} Nov 21 13:42:15 SELKS2 kibana[13495]: {"type":"log","@timestamp":"2019-11-21T12:42:15Z","tags":["error","task_manager"],"pid":13495,"message":"Fa… Nov 21 13:42:18 SELKS2 kibana[13495]: {"type":"log","@timestamp":"2019-11-21T12:42:18Z","tags":["listening","info"],"pid":13495,"messa…host:5601"} Nov 21 13:42:18 SELKS2 kibana[13495]: {"type":"log","@timestamp":"2019-11-21T12:42:18Z","tags":["status","plugin:spaces@6.8.5","info"]…ormation."} Hint: Some lines were ellipsized, use -l to show in full. ● evebox.service - EveBox Server Loaded: loaded (/lib/systemd/system/evebox.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2019-11-20 15:36:19 CET; 22h ago Main PID: 625 (evebox) Tasks: 10 (limit: 4915) Memory: 0B CGroup: /system.slice/evebox.service └─625 /usr/bin/evebox server

Nov 20 15:36:25 SELKS2 evebox[625]: 2019-11-20 15:36:25 (server.go:350) -- Failed to ping Elastic Search, delaying startup: : …ion refused Nov 20 15:36:28 SELKS2 evebox[625]: 2019-11-20 15:36:28 (server.go:350) -- Failed to ping Elastic Search, delaying startup: : …ion refused Nov 20 15:36:31 SELKS2 evebox[625]: 2019-11-20 15:36:31 (server.go:350) -- Failed to ping Elastic Search, delaying startup: : …ion refused Nov 20 15:36:34 SELKS2 evebox[625]: 2019-11-20 15:36:34 (server.go:350) -- Failed to ping Elastic Search, delaying startup: : …ion refused Nov 20 15:36:37 SELKS2 evebox[625]: 2019-11-20 15:36:37 (server.go:353) -- Connected to Elastic Search (version: 6.8.4) Nov 20 15:36:37 SELKS2 evebox[625]: 2019-11-20 15:36:37 (elasticsearch.go:199) -- Found templates [logstash] Nov 20 15:36:37 SELKS2 evebox[625]: 2019-11-20 15:36:37 (elasticsearch.go:238) -- Found Elastic Search keyword suffix to be: keyword Nov 20 15:36:37 SELKS2 evebox[625]: 2019-11-20 15:36:37 (server.go:131) -- Session reaper started Nov 20 15:36:37 SELKS2 evebox[625]: 2019-11-20 15:36:37 (server.go:165) -- Authentication disabled. Nov 20 15:36:37 SELKS2 evebox[625]: 2019-11-20 15:36:37 (server.go:276) -- Listening on 0.0.0.0:5636 Hint: Some lines were ellipsized, use -l to show in full. ● molochviewer-selks.service - Moloch Viewer Loaded: loaded (/etc/systemd/system/molochviewer-selks.service; enabled; vendor preset: enabled) Active: activating (auto-restart) (Result: exit-code) since Thu 2019-11-21 13:41:59 CET; 41s ago Process: 13617 ExecStart=/bin/sh -c /data/moloch/bin/node viewer.js -c /data/moloch/etc/config.ini >> /data/moloch/logs/viewer.log 2>&1 (code=exited, status=1/FAILURE) Main PID: 13617 (code=exited, status=1/FAILURE)

Nov 21 13:41:59 SELKS2 systemd[1]: molochviewer-selks.service: Main process exited, code=exited, status=1/FAILURE Nov 21 13:41:59 SELKS2 systemd[1]: molochviewer-selks.service: Failed with result 'exit-code'. ● molochpcapread-selks.service - Moloch Pcap Read Loaded: loaded (/etc/systemd/system/molochpcapread-selks.service; enabled; vendor preset: enabled) Active: activating (auto-restart) (Result: exit-code) since Thu 2019-11-21 13:41:58 CET; 41s ago Process: 13614 ExecStart=/bin/sh -c /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m --copy --delete -R /data/nsm/ >> /data/moloch/logs/capture.log 2>&1 (code=exited, status=1/FAILURE) Main PID: 13614 (code=exited, status=1/FAILURE)

Nov 21 13:41:58 SELKS2 systemd[1]: molochpcapread-selks.service: Main process exited, code=exited, status=1/FAILURE Nov 21 13:41:58 SELKS2 systemd[1]: molochpcapread-selks.service: Failed with result 'exit-code'. scirius RUNNING pid 13552, uptime 0:00:43 ii elasticsearch 6.8.5 all Elasticsearch is a distributed RESTful search engine built for the cloud. Reference documentation can be found at https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html and the 'Elasticsearch: The Definitive Guide' book can be found at https://www.elastic.co/guide/en/elasticsearch/guide/current/index.html ii elasticsearch-curator 5.8.1 amd64 Have indices in Elasticsearch? This is the tool for you!\n\nLike a museum curator manages the exhibits and collections on display, \nElasticsearch Curator helps you curate, or manage your indices. ii evebox 1:0.10.2 amd64 no description given ii kibana 6.8.5 amd64 Explore and visualize your Elasticsearch data ii kibana-dashboards-stamus 2019030501 amd64 Kibana 6 dashboard templates. ii logstash 1:6.8.5-1 all An extensible logging pipeline ii moloch 2.1.0-1 amd64 Moloch Full Packet System ii scirius 3.2.0-1 amd64 Django application to manage Suricata ruleset ii suricata 1:4.1.2-2 amd64 Next Generation Intrusion Detection and Prevention Tool ii suricata-oinkmaster 1:4.1.2-2 all Integration package between suricata and oinkmaster Filesystem Type Size Used Avail Use% Mounted on udev devtmpfs 7.8G 0 7.8G 0% /dev tmpfs tmpfs 1.6G 8.8M 1.6G 1% /run /dev/md0 ext3 887G 30G 812G 4% / tmpfs tmpfs 7.8G 0 7.8G 0% /dev/shm tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup tmpfs tmpfs 1.6G 0 1.6G 0% /run/user/1001

michal25 commented 4 years ago

Look for serious incompatibility between python debian 10 repository and selks (debian 9) repository.

What will be better now?

  1. Reinstall SELKS device with https://www.stamus-networks.com/sn-dl/selks/e571611b374462f67ed7588a1b9f5e81c7fcac50f953df45a278ff238914ade8/SELKS-5.0-nodesktop.iso

  2. Wait until Stamus will update SELKS repository for debian 10

  3. Another way

pevma commented 4 years ago

You can reinstall python2-minimal and continue with the update something like 

rm /var/lib/dpkg/info/python-minimal* ; rm /var/lib/dpkg/info/python2-minimal* ;
apt --fix-broken install
michal25 commented 4 years ago

Well, now is broken this package python2-minimal Setting up python2-minimal (2.7.16-1) ... dpkg: error processing package python2-minimal (--configure): installed python2-minimal package post-installation script subprocess returned error exit status 1 Errors were encountered while processing: python2-minimal E: Sub-process /usr/bin/dpkg returned an error code (1)

and suricata is not running

the rest of SELKS binaries is running now

root@SELKS2:~# selks-health-check_stamus ● suricata.service - Suricata IDS/IDP daemon Loaded: loaded (/lib/systemd/system/suricata.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Thu 2019-11-21 14:43:01 CET; 2min 54s ago Docs: man:suricata(8) man:suricatasc(8) https://suricata-ids.org/docs/ Process: 1448 ExecStart=/usr/bin/suricata -D --af-packet -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid (code=exited, status=0/SUCCESS) Main PID: 1449 (code=exited, status=1/FAILURE)

Nov 21 14:43:01 SELKS2 systemd[1]: suricata.service: Failed with result 'exit-code'. Nov 21 14:43:01 SELKS2 systemd[1]: suricata.service: Service RestartSec=100ms expired, scheduling restart. Nov 21 14:43:01 SELKS2 systemd[1]: suricata.service: Scheduled restart job, restart counter is at 5. Nov 21 14:43:01 SELKS2 systemd[1]: Stopped Suricata IDS/IDP daemon. Nov 21 14:43:01 SELKS2 systemd[1]: suricata.service: Start request repeated too quickly. Nov 21 14:43:01 SELKS2 systemd[1]: suricata.service: Failed with result 'exit-code'. Nov 21 14:43:01 SELKS2 systemd[1]: Failed to start Suricata IDS/IDP daemon. Nov 21 14:43:02 SELKS2 systemd[1]: suricata.service: Start request repeated too quickly. Nov 21 14:43:02 SELKS2 systemd[1]: suricata.service: Failed with result 'exit-code'. Nov 21 14:43:02 SELKS2 systemd[1]: Failed to start Suricata IDS/IDP daemon. ● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2019-11-21 14:27:39 CET; 18min ago Docs: http://www.elastic.co Main PID: 661 (java) Tasks: 77 (limit: 4915) Memory: 4.9G CGroup: /system.slice/elasticsearch.service ├─661 /usr/bin/java -Xms4g -Xmx4g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -De… └─915 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Nov 21 14:27:39 SELKS2 systemd[1]: Started Elasticsearch. Nov 21 14:27:39 SELKS2 elasticsearch[661]: warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME ● logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2019-11-21 14:27:39 CET; 18min ago Main PID: 658 (java) Tasks: 36 (limit: 4915) Memory: 943.3M CGroup: /system.slice/logstash.service └─658 /usr/bin/java -Xms4g -Xmx4g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Dj…

Nov 21 14:28:09 SELKS2 logstash[658]: [2019-11-21T14:28:09,491][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_te… Nov 21 14:28:09 SELKS2 logstash[658]: [2019-11-21T14:28:09,492][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_te… Nov 21 14:28:09 SELKS2 logstash[658]: [2019-11-21T14:28:09,554][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch templa…te/logstash Nov 21 14:28:09 SELKS2 logstash[658]: [2019-11-21T14:28:09,556][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch templa…te/logstash Nov 21 14:28:09 SELKS2 logstash[658]: [2019-11-21T14:28:09,864][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/s…City.mmdb"} Nov 21 14:28:09 SELKS2 logstash[658]: [2019-11-21T14:28:09,881][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/s…City.mmdb"} Nov 21 14:28:10 SELKS2 logstash[658]: [2019-11-21T14:28:10,058][INFO ][logstash.pipeline ] Pipeline started successfully {:pipe…a93b run>"} Nov 21 14:28:10 SELKS2 logstash[658]: [2019-11-21T14:28:10,101][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch wi…collections Nov 21 14:28:10 SELKS2 logstash[658]: [2019-11-21T14:28:10,124][INFO ][logstash.agent ] Pipelines running {:count=>1, :runni…elines=>[]} Nov 21 14:28:10 SELKS2 logstash[658]: [2019-11-21T14:28:10,360][INFO ][logstash.agent ] Successfully started Logstash API en…port=>9600} Hint: Some lines were ellipsized, use -l to show in full. ● kibana.service - Kibana Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2019-11-21 14:27:39 CET; 18min ago Main PID: 640 (node) Tasks: 11 (limit: 4915) Memory: 521.9M CGroup: /system.slice/kibana.service └─640 /usr/share/kibana/bin/../node/bin/node --no-warnings --max-http-header-size=65536 /usr/share/kibana/bin/../src/cli -c /etc/kiban…

Nov 21 14:27:58 SELKS2 kibana[640]: {"type":"log","@timestamp":"2019-11-21T13:27:58Z","tags":["status","plugin:remote_clusters@6.8.5",…ormation."} Nov 21 14:27:58 SELKS2 kibana[640]: {"type":"log","@timestamp":"2019-11-21T13:27:58Z","tags":["status","plugin:cross_cluster_replicati…ormation."} Nov 21 14:27:58 SELKS2 kibana[640]: {"type":"log","@timestamp":"2019-11-21T13:27:58Z","tags":["status","plugin:reporting@6.8.5","info"…ormation."} Nov 21 14:27:58 SELKS2 kibana[640]: {"type":"log","@timestamp":"2019-11-21T13:27:58Z","tags":["info","monitoring-ui","kibana-monitorin…ollection"} Nov 21 14:27:58 SELKS2 kibana[640]: {"type":"log","@timestamp":"2019-11-21T13:27:58Z","tags":["status","plugin:security@6.8.5","info"]…ormation."} Nov 21 14:27:58 SELKS2 kibana[640]: {"type":"log","@timestamp":"2019-11-21T13:27:58Z","tags":["status","plugin:maps@6.8.5","info"],"pi…ormation."} Nov 21 14:27:58 SELKS2 kibana[640]: {"type":"log","@timestamp":"2019-11-21T13:27:58Z","tags":["license","info","xpack"],"pid":640,"mes…s: active"} Nov 21 14:28:00 SELKS2 kibana[640]: {"type":"log","@timestamp":"2019-11-21T13:28:00Z","tags":["error","task_manager"],"pid":640,"message":"Failed… Nov 21 14:28:01 SELKS2 kibana[640]: {"type":"log","@timestamp":"2019-11-21T13:28:01Z","tags":["listening","info"],"pid":640,"message":…host:5601"} Nov 21 14:28:01 SELKS2 kibana[640]: {"type":"log","@timestamp":"2019-11-21T13:28:01Z","tags":["status","plugin:spaces@6.8.5","info"],"…ormation."} Hint: Some lines were ellipsized, use -l to show in full. ● evebox.service - EveBox Server Loaded: loaded (/lib/systemd/system/evebox.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2019-11-21 14:27:39 CET; 18min ago Main PID: 645 (evebox) Tasks: 12 (limit: 4915) Memory: 37.0M CGroup: /system.slice/evebox.service └─645 /usr/bin/evebox server

Nov 21 14:27:55 SELKS2 evebox[645]: "minimum_index_compatibility_version" : "5.0.0" Nov 21 14:27:55 SELKS2 evebox[645]: }, Nov 21 14:27:55 SELKS2 evebox[645]: "tagline" : "You Know, for Search" Nov 21 14:27:55 SELKS2 evebox[645]: } Nov 21 14:27:58 SELKS2 evebox[645]: 2019-11-21 14:27:58 (server.go:353) -- Connected to Elastic Search (version: 6.8.5) Nov 21 14:27:58 SELKS2 evebox[645]: 2019-11-21 14:27:58 (elasticsearch.go:199) -- Found templates [logstash] Nov 21 14:27:58 SELKS2 evebox[645]: 2019-11-21 14:27:58 (elasticsearch.go:238) -- Found Elastic Search keyword suffix to be: keyword Nov 21 14:27:58 SELKS2 evebox[645]: 2019-11-21 14:27:58 (server.go:131) -- Session reaper started Nov 21 14:27:58 SELKS2 evebox[645]: 2019-11-21 14:27:58 (server.go:165) -- Authentication disabled. Nov 21 14:27:58 SELKS2 evebox[645]: 2019-11-21 14:27:58 (server.go:276) -- Listening on 0.0.0.0:5636 ● molochviewer-selks.service - Moloch Viewer Loaded: loaded (/etc/systemd/system/molochviewer-selks.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2019-11-21 14:29:11 CET; 16min ago Main PID: 1120 (sh) Tasks: 12 (limit: 4915) Memory: 42.7M CGroup: /system.slice/molochviewer-selks.service ├─1120 /bin/sh -c /data/moloch/bin/node viewer.js -c /data/moloch/etc/config.ini >> /data/moloch/logs/viewer.log 2>&1 └─1121 /data/moloch/bin/node viewer.js -c /data/moloch/etc/config.ini

Nov 21 14:29:11 SELKS2 systemd[1]: Started Moloch Viewer. ● molochpcapread-selks.service - Moloch Pcap Read Loaded: loaded (/etc/systemd/system/molochpcapread-selks.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2019-11-21 14:29:10 CET; 16min ago Main PID: 1105 (sh) Tasks: 6 (limit: 4915) Memory: 427.8M CGroup: /system.slice/molochpcapread-selks.service ├─1105 /bin/sh -c /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m --copy --delete -R /data/nsm/ >> /data/moloch/log… └─1106 /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m --copy --delete -R /data/nsm/

Nov 21 14:29:10 SELKS2 systemd[1]: Started Moloch Pcap Read. scirius RUNNING pid 853, uptime 0:18:16 ii elasticsearch 6.8.5 all Elasticsearch is a distributed RESTful search engine built for the cloud. Reference documentation can be found at https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html and the 'Elasticsearch: The Definitive Guide' book can be found at https://www.elastic.co/guide/en/elasticsearch/guide/current/index.html ii elasticsearch-curator 5.8.1 amd64 Have indices in Elasticsearch? This is the tool for you!\n\nLike a museum curator manages the exhibits and collections on display, \nElasticsearch Curator helps you curate, or manage your indices. ii evebox 1:0.10.2 amd64 no description given ii kibana 6.8.5 amd64 Explore and visualize your Elasticsearch data ii kibana-dashboards-stamus 2019030501 amd64 Kibana 6 dashboard templates. ii logstash 1:6.8.5-1 all An extensible logging pipeline ii moloch 2.1.0-1 amd64 Moloch Full Packet System ii scirius 3.2.0-1 amd64 Django application to manage Suricata ruleset ii suricata 1:4.1.2-2 amd64 Next Generation Intrusion Detection and Prevention Tool ii suricata-oinkmaster 1:4.1.2-2 all Integration package between suricata and oinkmaster Filesystem Type Size Used Avail Use% Mounted on udev devtmpfs 7.8G 0 7.8G 0% /dev tmpfs tmpfs 1.6G 8.9M 1.6G 1% /run /dev/md0 ext3 887G 30G 813G 4% / tmpfs tmpfs 7.8G 0 7.8G 0% /dev/shm tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup tmpfs tmpfs 1.6G 0 1.6G 0% /run/user/1001

And the problem is in suricata package, because I obtained the package from debian repository. I will try to install the stamus package now.

michal25 commented 4 years ago

Well. I downloaded from stamus/selks repository this binaries suricata_2019101501-0stamus0_amd64.deb libhtp2_0.5.31-0stamus3_amd64.deb

installed with dpkg -i and now scirius work, but moloch have known problem with "unknown field protocols".

root@SELKS2:~# selks-health-check_stamus ● suricata.service - LSB: Next Generation IDS/IPS Loaded: loaded (/etc/init.d/suricata; generated) Active: active (running) since Thu 2019-11-21 15:01:17 CET; 1min 40s ago Docs: man:systemd-sysv-generator(8) Process: 657 ExecStart=/etc/init.d/suricata start (code=exited, status=0/SUCCESS) Tasks: 10 (limit: 4915) Memory: 300.4M CGroup: /system.slice/suricata.service └─743 /usr/bin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid --af-packet -D -v --user=logstash

Nov 21 15:01:17 SELKS2 systemd[1]: Starting LSB: Next Generation IDS/IPS... Nov 21 15:01:17 SELKS2 suricata[657]: Starting suricata in IDS (af-packet) mode... done. Nov 21 15:01:17 SELKS2 systemd[1]: Started LSB: Next Generation IDS/IPS. ● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2019-11-21 15:01:17 CET; 1min 40s ago Docs: http://www.elastic.co Main PID: 656 (java) Tasks: 84 (limit: 4915) Memory: 4.9G CGroup: /system.slice/elasticsearch.service ├─656 /usr/bin/java -Xms4g -Xmx4g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -De… └─847 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Nov 21 15:01:17 SELKS2 systemd[1]: Started Elasticsearch. Nov 21 15:01:17 SELKS2 elasticsearch[656]: warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME ● logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2019-11-21 15:01:17 CET; 1min 40s ago Main PID: 642 (java) Tasks: 39 (limit: 4915) Memory: 1.0G CGroup: /system.slice/logstash.service └─642 /usr/bin/java -Xms4g -Xmx4g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Dj…

Nov 21 15:01:53 SELKS2 logstash[642]: [2019-11-21T15:01:53,543][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:p…late.json"} Nov 21 15:01:53 SELKS2 logstash[642]: [2019-11-21T15:01:53,547][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_te… Nov 21 15:01:53 SELKS2 logstash[642]: [2019-11-21T15:01:53,645][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch templa…te/logstash Nov 21 15:01:53 SELKS2 logstash[642]: [2019-11-21T15:01:53,652][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch templa…te/logstash Nov 21 15:01:53 SELKS2 logstash[642]: [2019-11-21T15:01:53,952][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/s…City.mmdb"} Nov 21 15:01:53 SELKS2 logstash[642]: [2019-11-21T15:01:53,968][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/s…City.mmdb"} Nov 21 15:01:54 SELKS2 logstash[642]: [2019-11-21T15:01:54,141][INFO ][logstash.pipeline ] Pipeline started successfully {:pipe…5111 run>"} Nov 21 15:01:54 SELKS2 logstash[642]: [2019-11-21T15:01:54,180][INFO ][logstash.agent ] Pipelines running {:count=>1, :runni…elines=>[]} Nov 21 15:01:54 SELKS2 logstash[642]: [2019-11-21T15:01:54,197][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch wi…collections Nov 21 15:01:54 SELKS2 logstash[642]: [2019-11-21T15:01:54,412][INFO ][logstash.agent ] Successfully started Logstash API en…port=>9600} Hint: Some lines were ellipsized, use -l to show in full. ● kibana.service - Kibana Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2019-11-21 15:01:17 CET; 1min 40s ago Main PID: 666 (node) Tasks: 11 (limit: 4915) Memory: 510.2M CGroup: /system.slice/kibana.service └─666 /usr/share/kibana/bin/../node/bin/node --no-warnings --max-http-header-size=65536 /usr/share/kibana/bin/../src/cli -c /etc/kiban…

Nov 21 15:01:36 SELKS2 kibana[666]: {"type":"log","@timestamp":"2019-11-21T14:01:36Z","tags":["status","plugin:remote_clusters@6.8.5",…nnections"} Nov 21 15:01:36 SELKS2 kibana[666]: {"type":"log","@timestamp":"2019-11-21T14:01:36Z","tags":["status","plugin:cross_cluster_replicati…nnections"} Nov 21 15:01:36 SELKS2 kibana[666]: {"type":"log","@timestamp":"2019-11-21T14:01:36Z","tags":["status","plugin:reporting@6.8.5","info"…nnections"} Nov 21 15:01:36 SELKS2 kibana[666]: {"type":"log","@timestamp":"2019-11-21T14:01:36Z","tags":["info","monitoring-ui","kibana-monitorin…ollection"} Nov 21 15:01:36 SELKS2 kibana[666]: {"type":"log","@timestamp":"2019-11-21T14:01:36Z","tags":["status","plugin:security@6.8.5","info"]…nnections"} Nov 21 15:01:36 SELKS2 kibana[666]: {"type":"log","@timestamp":"2019-11-21T14:01:36Z","tags":["status","plugin:maps@6.8.5","info"],"pi…nnections"} Nov 21 15:01:36 SELKS2 kibana[666]: {"type":"log","@timestamp":"2019-11-21T14:01:36Z","tags":["license","info","xpack"],"pid":666,"mes…s: active"} Nov 21 15:01:39 SELKS2 kibana[666]: {"type":"log","@timestamp":"2019-11-21T14:01:39Z","tags":["error","task_manager"],"pid":666,"message":"Failed… Nov 21 15:01:42 SELKS2 kibana[666]: {"type":"log","@timestamp":"2019-11-21T14:01:42Z","tags":["listening","info"],"pid":666,"message":…host:5601"} Nov 21 15:01:42 SELKS2 kibana[666]: {"type":"log","@timestamp":"2019-11-21T14:01:42Z","tags":["status","plugin:spaces@6.8.5","info"],"…nnections"} Hint: Some lines were ellipsized, use -l to show in full. ● evebox.service - EveBox Server Loaded: loaded (/lib/systemd/system/evebox.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2019-11-21 15:01:17 CET; 1min 40s ago Main PID: 659 (evebox) Tasks: 9 (limit: 4915) Memory: 36.8M CGroup: /system.slice/evebox.service └─659 /usr/bin/evebox server

Nov 21 15:01:23 SELKS2 evebox[659]: 2019-11-21 15:01:23 (server.go:350) -- Failed to ping Elastic Search, delaying startup: : …ion refused Nov 21 15:01:26 SELKS2 evebox[659]: 2019-11-21 15:01:26 (server.go:350) -- Failed to ping Elastic Search, delaying startup: : …ion refused Nov 21 15:01:29 SELKS2 evebox[659]: 2019-11-21 15:01:29 (server.go:350) -- Failed to ping Elastic Search, delaying startup: : …ion refused Nov 21 15:01:32 SELKS2 evebox[659]: 2019-11-21 15:01:32 (server.go:350) -- Failed to ping Elastic Search, delaying startup: : …ion refused Nov 21 15:01:36 SELKS2 evebox[659]: 2019-11-21 15:01:36 (server.go:353) -- Connected to Elastic Search (version: 6.8.5) Nov 21 15:01:36 SELKS2 evebox[659]: 2019-11-21 15:01:36 (elasticsearch.go:199) -- Found templates [logstash] Nov 21 15:01:36 SELKS2 evebox[659]: 2019-11-21 15:01:36 (elasticsearch.go:238) -- Found Elastic Search keyword suffix to be: keyword Nov 21 15:01:36 SELKS2 evebox[659]: 2019-11-21 15:01:36 (server.go:131) -- Session reaper started Nov 21 15:01:36 SELKS2 evebox[659]: 2019-11-21 15:01:36 (server.go:165) -- Authentication disabled. Nov 21 15:01:36 SELKS2 evebox[659]: 2019-11-21 15:01:36 (server.go:276) -- Listening on 0.0.0.0:5636 Hint: Some lines were ellipsized, use -l to show in full. ● molochviewer-selks.service - Moloch Viewer Loaded: loaded (/etc/systemd/system/molochviewer-selks.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2019-11-21 15:02:49 CET; 8s ago Main PID: 1071 (sh) Tasks: 12 (limit: 4915) Memory: 43.0M CGroup: /system.slice/molochviewer-selks.service ├─1071 /bin/sh -c /data/moloch/bin/node viewer.js -c /data/moloch/etc/config.ini >> /data/moloch/logs/viewer.log 2>&1 └─1072 /data/moloch/bin/node viewer.js -c /data/moloch/etc/config.ini

Nov 21 15:02:49 SELKS2 systemd[1]: molochviewer-selks.service: Service RestartSec=1min 30s expired, scheduling restart. Nov 21 15:02:49 SELKS2 systemd[1]: molochviewer-selks.service: Scheduled restart job, restart counter is at 1. Nov 21 15:02:49 SELKS2 systemd[1]: Stopped Moloch Viewer. Nov 21 15:02:49 SELKS2 systemd[1]: Started Moloch Viewer. ● molochpcapread-selks.service - Moloch Pcap Read Loaded: loaded (/etc/systemd/system/molochpcapread-selks.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2019-11-21 15:02:48 CET; 10s ago Main PID: 1060 (sh) Tasks: 6 (limit: 4915) Memory: 453.9M CGroup: /system.slice/molochpcapread-selks.service ├─1060 /bin/sh -c /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m --copy --delete -R /data/nsm/ >> /data/moloch/log… └─1061 /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m --copy --delete -R /data/nsm/

Nov 21 15:02:48 SELKS2 systemd[1]: Started Moloch Pcap Read. scirius RUNNING pid 815, uptime 0:01:40 ii elasticsearch 6.8.5 all Elasticsearch is a distributed RESTful search engine built for the cloud. Reference documentation can be found at https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html and the 'Elasticsearch: The Definitive Guide' book can be found at https://www.elastic.co/guide/en/elasticsearch/guide/current/index.html ii elasticsearch-curator 5.8.1 amd64 Have indices in Elasticsearch? This is the tool for you!\n\nLike a museum curator manages the exhibits and collections on display, \nElasticsearch Curator helps you curate, or manage your indices. ii evebox 1:0.10.2 amd64 no description given ii kibana 6.8.5 amd64 Explore and visualize your Elasticsearch data ii kibana-dashboards-stamus 2019030501 amd64 Kibana 6 dashboard templates. ii logstash 1:6.8.5-1 all An extensible logging pipeline ii moloch 2.1.0-1 amd64 Moloch Full Packet System ii scirius 3.2.0-1 amd64 Django application to manage Suricata ruleset ii suricata 1:2019101501-0stamus0 amd64 Suricata open source multi-thread IDS/IPS/NSM system. rc suricata-oinkmaster 1:4.1.2-2 all Integration package between suricata and oinkmaster Filesystem Type Size Used Avail Use% Mounted on udev devtmpfs 7.8G 0 7.8G 0% /dev tmpfs tmpfs 1.6G 8.9M 1.6G 1% /run /dev/md0 ext3 887G 29G 813G 4% / tmpfs tmpfs 7.8G 0 7.8G 0% /dev/shm tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup tmpfs tmpfs 1.6G 0 1.6G 0% /run/user/1001 Screenshot_20191121_150431 Screenshot_20191121_150549

pevma commented 4 years ago

Executed as root (the dashboard reset)?

michal25 commented 4 years ago

Yes. As root.

  1. listopadu 2019 15:16:40 SEČ, Peter Manev notifications@github.com napsal:

    Executed as root (the dashboard reset)?

    -- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/StamusNetworks/SELKS/issues/204#issuecomment-557103904

-- Odesláno z mého telefonu s Androidem pomocí pošty K-9 Mail. Omluvte prosím moji stručnost.

pevma commented 4 years ago

I assume all moloch services have been restarted?
Maybe you can try running the moloch first time set up script again ?

michal25 commented 4 years ago

I tried the selks-first-time-setup_stamus script ,which gives the Moloch ImportError, and the selks-molochdb-init-setup_stamus, which passes Ok, but the Moloch "Unknown field protocol" problem remains.

I think, the problem is in python 2.7 libraries, because the python2-minimal package still remains unconfigured.

michal25 commented 4 years ago

And here is the problem.

root@SELKS2:~# dpkg -i python2_2.7.16-1_amd64.deb 
dpkg: regarding python2_2.7.16-1_amd64.deb containing python2, pre-dependency problem:
 python2 pre-depends on python2-minimal (= 2.7.16-1)
  python2-minimal is unpacked, but has never been configured.

dpkg: error processing archive python2_2.7.16-1_amd64.deb (--install):
 pre-dependency problem - not installing python2
Errors were encountered while processing:
 python2_2.7.16-1_amd64.deb
root@SELKS2:~# dpkg -i python2-minimal_2.7.16-1_amd64.deb 
(Reading database ... 207325 files and directories currently installed.)
Preparing to unpack python2-minimal_2.7.16-1_amd64.deb ...
Unpacking python2-minimal (2.7.16-1) over (2.7.16-1) ...
Setting up python2-minimal (2.7.16-1) ...
dpkg: error processing package python2-minimal (--install):
 installed python2-minimal package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 python2-minimal
pevma commented 4 years ago

Can you try that - https://github.com/StamusNetworks/SELKS/issues/204#issuecomment-557082849 ?

michal25 commented 4 years ago

BINGO! root@SELKS2:~# rm /var/lib/dpkg/info/python-minimal ; rm /var/lib/dpkg/info/python2-minimal ; root@SELKS2:~# apt --fix-broken install Reading package lists... Done Building dependency tree
Reading state information... Done 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. 1 not fully installed or removed. After this operation, 0 B of additional disk space will be used. Setting up python2-minimal (2.7.16-1) ... root@SELKS2:~#

michal25 commented 4 years ago

But the problem still remains 

root@SELKS2:~# selks-first-time-setup_stamus 
START of first time setup script - Fri Nov 22 14:04:06 CET 2019 

### Setting up sniffing interface  ###

Please supply a network interface(s) to set up SELKS Suricata IDPS thread detection on
0: enp0s31f6
1: enp1s0
2: lo
Please type in interface or space delimited interfaces below and hit "Enter".
Example: eth1
OR
Example: eth1 eth2 eth3

Configure threat detection for INTERFACE(S): 
enp0s31f6

The supplied network interface(s):  enp0s31f6 

DONE!
FPC - Full Packet Capture. Suricata will rotate and delete the pcap captured files.
FPC_Retain - Full Packet Capture with having Moloch's pcap retention/rotation. Keeps the pcaps as long as there is space available.
None - disable packet capture

1) FPC
2) FPC_Retain
3) NONE
Please choose an option. Type in a number and hit "Enter" 2
Enable Full Pcacket Capture with pcap retaining 

### Starting Moloch DB set up ###

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   406  100   406    0     0   396k      0 --:--:-- --:--:-- --:--:--  396k
{"cluster_name":"elasticsearch","status":"yellow","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":184,"active_shards":184,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":97.35449735449735}

### Setting up Moloch ###

WARNING elasticsearch health is 'yellow' instead of 'green', things may be broken

It is STRONGLY recommended that you stop ALL moloch captures and viewers before proceeding.  Use 'db.pl http://localhost:9200 backup' to backup db first.

There is 1 elastic search data node, if you expect more please fix first before proceeding.

It appears this elastic search cluster already has moloch installed (version 64), this will delete ALL data in elastic search! (It does not delete the pcap files on disk.)

Type "INIT" to continue - do you want to erase everything??
Erasing
Creating

Finished
Found interfaces: enp0s31f6;enp1s0;lo
Semicolon ';' seperated list of interfaces to monitor [eth1] Install Elasticsearch server locally for demo, must have at least 3G of memory, NOT recommended for production use (yes or no) [no] Elasticsearch server URL [http://localhost:9200] Password to encrypt S2S and other things [no-default] Moloch - Creating configuration files
Not overwriting /data/moloch/etc/config.ini, delete and run again if update required (usually not), or edit by hand
Installing systemd start files, use systemctl
Download GEO files? (yes or no) [yes] Moloch - Downloading GEO files
WARNING: timestamping does nothing in combination with -O. See the manual
for details.

2019-11-22 14:04:48 URL:https://updates.maxmind.com/app/update_secure?edition_id=GeoLite2-Country [2032773/2032773] -> "GeoLite2-Country.mmdb.gz" [1]
WARNING: timestamping does nothing in combination with -O. See the manual
for details.

2019-11-22 14:04:49 URL:https://updates.maxmind.com/app/update_secure?edition_id=GeoLite2-ASN [3656764/3656764] -> "GeoLite2-ASN.mmdb.gz" [1]
2019-11-22 14:04:49 URL:https://raw.githubusercontent.com/wireshark/wireshark/master/manuf [1647100/1647100] -> "oui.txt" [1]

Moloch - Configured - Now continue with step 4 in /data/moloch/README.txt

      /sbin/start elasticsearch # for upstart/Centos 6/Ubuntu 14.04
      systemctl start elasticsearch.service # for systemd/Centos 7/Ubuntu 16.04
 5) Initialize/Upgrade Elasticsearch Moloch configuration
  a) If this is the first install, or want to delete all data
      /data/moloch/db/db.pl http://ESHOST:9200 init
  b) If this is an update to moloch package
      /data/moloch/db/db.pl http://ESHOST:9200 upgrade
 6) Add an admin user if a new install or after an init
      /data/moloch/bin/moloch_add_user.sh admin "Admin User" THEPASSWORD --admin
 7) Start everything
   a) If using upstart (Centos 6 or sometimes Ubuntu 14.04):
      /sbin/start molochcapture
      /sbin/start molochviewer
   b) If using systemd (Centos 7 or Ubuntu 16.04 or sometimes Ubuntu 14.04)
      systemctl start molochcapture.service
      systemctl start molochviewer.service
 8) Look at log files for errors
      /data/moloch/logs/viewer.log
      /data/moloch/logs/capture.log
 9) Visit http://MOLOCHHOST:8005 with your favorite browser.
      user: admin
      password: THEPASSWORD from step #6

Any configuration changes can be made to /data/moloch/etc/config.ini
See https://molo.ch/faq#moloch-is-not-working for issues

Additional information can be found at:
  * https://molo.ch/faq
  * https://molo.ch/settings
Added

### Setting up Moloch configs and services ###

Would you like to setup a retention policy now? (y/n)
y

Please specify the maximum file size in Gigabytes. The disk should have room for at least 10 times the specified value. (default is 12)
25

 Setting maxFileSizeG to 25 Gigabyte.

Please specify the maximum rotation time in minutes. (default is none)
600

 Setting maxFileTimeM to 600 minutes.

### Setting up and restarting services ###

### Setting up Scirius/Moloch proxy user ###

Added
Traceback (most recent call last):
  File "bin/manage.py", line 8, in <module>
    from django.core.management import execute_from_command_line
  File "/usr/share/python/scirius/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 10, in <module>
    from django.apps import apps
  File "/usr/share/python/scirius/local/lib/python2.7/site-packages/django/apps/__init__.py", line 1, in <module>
    from .config import AppConfig
  File "/usr/share/python/scirius/local/lib/python2.7/site-packages/django/apps/config.py", line 4, in <module>
    from django.core.exceptions import ImproperlyConfigured
  File "/usr/share/python/scirius/local/lib/python2.7/site-packages/django/core/exceptions.py", line 5, in <module>
    from django.utils.encoding import force_text
  File "/usr/share/python/scirius/local/lib/python2.7/site-packages/django/utils/encoding.py", line 10, in <module>
    from django.utils.functional import Promise
  File "/usr/share/python/scirius/local/lib/python2.7/site-packages/django/utils/functional.py", line 1, in <module>
    import copy
  File "/usr/lib/python2.7/copy.py", line 52, in <module>
    import weakref
  File "/usr/lib/python2.7/weakref.py", line 14, in <module>
    from _weakref import (
ImportError: cannot import name _remove_dead_weakref
Dashboards loading set up job failed...Exiting...
### Exited with ERROR  ###

FINISH of first time setup script - Fri Nov 22 14:05:11 CET 2019 

Exited with FAILED
Full log located at - /opt/selks/log/selks-first-time-setup_stamus.log
Press enter to continue
root@SELKS2:~#
pevma commented 4 years ago

Can you try that command below as root

cd /usr/share/python/scirius/ && . bin/activate && python bin/manage.py kibana_reset && deactivate
michal25 commented 4 years ago

root@SELKS2:~# cd /usr/share/python/scirius/ && . bin/activate && python bin/manage.py kibana_reset && deactivate Traceback (most recent call last): File "bin/manage.py", line 8, in from django.core.management import execute_from_command_line File "/usr/share/python/scirius/local/lib/python2.7/site-packages/django/core/management/init.py", line 10, in from django.apps import apps File "/usr/share/python/scirius/local/lib/python2.7/site-packages/django/apps/init.py", line 1, in from .config import AppConfig File "/usr/share/python/scirius/local/lib/python2.7/site-packages/django/apps/config.py", line 4, in from django.core.exceptions import ImproperlyConfigured File "/usr/share/python/scirius/local/lib/python2.7/site-packages/django/core/exceptions.py", line 5, in from django.utils.encoding import force_text File "/usr/share/python/scirius/local/lib/python2.7/site-packages/django/utils/encoding.py", line 10, in from django.utils.functional import Promise File "/usr/share/python/scirius/local/lib/python2.7/site-packages/django/utils/functional.py", line 1, in import copy File "/usr/lib/python2.7/copy.py", line 52, in import weakref File "/usr/lib/python2.7/weakref.py", line 14, in from _weakref import ( ImportError: cannot import name _remove_dead_weakref (scirius)root@SELKS2:/usr/share/python/scirius#

michal25 commented 4 years ago

But, I restarted the SELKS device and Moloch WORKS fine now. With Debian 10 root@SELKS2:/usr/share/python/scirius# cat /etc/issue Debian GNU/Linux 10 \n \l

michal25 commented 4 years ago

Now (another SELKS device) I'm trying this method

It seems you need 1.1.1 which is interesting why is it not avail in the distro. Maybe you can try back-porting that package from Buster >(https://packages.debian.org/buster/openssl)

I had to download and dpkg -i (install) this binaries openssl_1.1.1d-0+deb10u2_amd64.deb libssl1.1_1.1.1d-0+deb10u2_amd64.deb libc-bin_2.28-10_amd64.deb libc-l10n_2.28-10_all.deb libc6_2.28-10_amd64.deb locales_2.28-10_all.deb

michal25 commented 4 years ago

And works! root@SELKS:~# selks-health-check_stamus ● suricata.service - LSB: Next Generation IDS/IPS Loaded: loaded (/etc/init.d/suricata; generated; vendor preset: enabled) Active: active (running) since Sat 2019-11-23 22:24:00 CET; 29min ago Docs: man:systemd-sysv-generator(8) Process: 642 ExecStart=/etc/init.d/suricata start (code=exited, status=0/SUCCESS) Tasks: 8 (limit: 4915) CGroup: /system.slice/suricata.service └─693 /usr/bin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid --af-packet -D -v --user=logstash

Nov 23 22:24:00 SELKS systemd[1]: Starting LSB: Next Generation IDS/IPS... Nov 23 22:24:00 SELKS suricata[642]: Starting suricata in IDS (af-packet) mode... done. Nov 23 22:24:00 SELKS systemd[1]: Started LSB: Next Generation IDS/IPS. ● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2019-11-23 22:24:00 CET; 29min ago Docs: http://www.elastic.co Main PID: 639 (java) Tasks: 64 (limit: 4915) CGroup: /system.slice/elasticsearch.service ├─639 /usr/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.…et └─889 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Nov 23 22:24:00 SELKS systemd[1]: Started Elasticsearch. Nov 23 22:24:00 SELKS elasticsearch[639]: warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME ● logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2019-11-23 22:23:58 CET; 29min ago Main PID: 408 (java) Tasks: 31 (limit: 4915) CGroup: /system.slice/logstash.service └─408 /usr/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=…sh

Nov 23 22:25:43 SELKS logstash[408]: [2019-11-23T22:25:43,260][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template…}} Nov 23 22:25:43 SELKS logstash[408]: [2019-11-23T22:25:43,263][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template…}} Nov 23 22:25:43 SELKS logstash[408]: [2019-11-23T22:25:43,355][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/logstash Nov 23 22:25:43 SELKS logstash[408]: [2019-11-23T22:25:43,357][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/logstash Nov 23 22:25:44 SELKS logstash[408]: [2019-11-23T22:25:44,994][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/share/logstash/vendo…-City.mmdb"} Nov 23 22:25:45 SELKS logstash[408]: [2019-11-23T22:25:45,076][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/share/logstash/vendo…-City.mmdb"} Nov 23 22:25:45 SELKS logstash[408]: [2019-11-23T22:25:45,615][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :t…0bb07 run>"} Nov 23 22:25:45 SELKS logstash[408]: [2019-11-23T22:25:45,770][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:mai…pelines=>[]} Nov 23 22:25:45 SELKS logstash[408]: [2019-11-23T22:25:45,798][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections Nov 23 22:25:48 SELKS logstash[408]: [2019-11-23T22:25:48,104][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} Hint: Some lines were ellipsized, use -l to show in full. ● kibana.service - Kibana Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2019-11-23 22:26:12 CET; 27min ago Main PID: 1514 (node) Tasks: 11 (limit: 4915) CGroup: /system.slice/kibana.service └─1514 /usr/share/kibana/bin/../node/bin/node --no-warnings --max-http-header-size=65536 /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Nov 23 22:26:20 SELKS kibana[1514]: {"type":"log","@timestamp":"2019-11-23T21:26:20Z","tags":["status","plugin:cross_cluster_replication@6.8.5","info"],…sticsearch"} Nov 23 22:26:20 SELKS kibana[1514]: {"type":"log","@timestamp":"2019-11-23T21:26:20Z","tags":["info","monitoring-ui","kibana-monitoring"],"pid":1514,"me…collection"} Nov 23 22:26:20 SELKS kibana[1514]: {"type":"log","@timestamp":"2019-11-23T21:26:20Z","tags":["status","plugin:security@6.8.5","info"],"pid":1514,"state…sticsearch"} Nov 23 22:26:20 SELKS kibana[1514]: {"type":"log","@timestamp":"2019-11-23T21:26:20Z","tags":["status","plugin:maps@6.8.5","info"],"pid":1514,"state":"g…sticsearch"} Nov 23 22:26:20 SELKS kibana[1514]: {"type":"log","@timestamp":"2019-11-23T21:26:20Z","tags":["license","info","xpack"],"pid":1514,"message":"Imported l…us: active"} Nov 23 22:26:20 SELKS kibana[1514]: {"type":"log","@timestamp":"2019-11-23T21:26:20Z","tags":["reporting","browser-driver","warning"],"pid":1514,"messag…rotection."} Nov 23 22:26:20 SELKS kibana[1514]: {"type":"log","@timestamp":"2019-11-23T21:26:20Z","tags":["reporting","warning"],"pid":1514,"message":"Generating a …kibana.yml"} Nov 23 22:26:20 SELKS kibana[1514]: {"type":"log","@timestamp":"2019-11-23T21:26:20Z","tags":["status","plugin:reporting@6.8.5","info"],"pid":1514,"stat…nitialized"} Nov 23 22:26:21 SELKS kibana[1514]: {"type":"log","@timestamp":"2019-11-23T21:26:21Z","tags":["listening","info"],"pid":1514,"message":"Server running a…lhost:5601"} Nov 23 22:26:21 SELKS kibana[1514]: {"type":"log","@timestamp":"2019-11-23T21:26:21Z","tags":["status","plugin:spaces@6.8.5","info"],"pid":1514,"state":…sticsearch"} Hint: Some lines were ellipsized, use -l to show in full. ● evebox.service - EveBox Server Loaded: loaded (/lib/systemd/system/evebox.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2019-11-23 22:23:58 CET; 29min ago Main PID: 410 (evebox) Tasks: 8 (limit: 4915) CGroup: /system.slice/evebox.service └─410 /usr/bin/evebox server

Nov 23 22:24:39 SELKS evebox[410]: "minimum_index_compatibility_version" : "5.0.0" Nov 23 22:24:39 SELKS evebox[410]: }, Nov 23 22:24:39 SELKS evebox[410]: "tagline" : "You Know, for Search" Nov 23 22:24:39 SELKS evebox[410]: } Nov 23 22:24:42 SELKS evebox[410]: 2019-11-23 22:24:42 (server.go:353) -- Connected to Elastic Search (version: 6.8.5) Nov 23 22:24:42 SELKS evebox[410]: 2019-11-23 22:24:42 (elasticsearch.go:199) -- Found templates [logstash] Nov 23 22:24:42 SELKS evebox[410]: 2019-11-23 22:24:42 (elasticsearch.go:238) -- Found Elastic Search keyword suffix to be: keyword Nov 23 22:24:42 SELKS evebox[410]: 2019-11-23 22:24:42 (server.go:131) -- Session reaper started Nov 23 22:24:42 SELKS evebox[410]: 2019-11-23 22:24:42 (server.go:165) -- Authentication disabled. Nov 23 22:24:42 SELKS evebox[410]: 2019-11-23 22:24:42 (server.go:276) -- Listening on 0.0.0.0:5636 ● molochviewer-selks.service - Moloch Viewer Loaded: loaded (/etc/systemd/system/molochviewer-selks.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2019-11-23 22:25:35 CET; 27min ago Main PID: 1442 (sh) Tasks: 12 (limit: 4915) CGroup: /system.slice/molochviewer-selks.service ├─1442 /bin/sh -c /data/moloch/bin/node viewer.js -c /data/moloch/etc/config.ini >> /data/moloch/logs/viewer.log 2>&1 └─1443 /data/moloch/bin/node viewer.js -c /data/moloch/etc/config.ini

Nov 23 22:25:35 SELKS systemd[1]: molochviewer-selks.service: Service hold-off time over, scheduling restart. Nov 23 22:25:35 SELKS systemd[1]: Stopped Moloch Viewer. Nov 23 22:25:35 SELKS systemd[1]: Started Moloch Viewer. ● molochpcapread-selks.service - Moloch Pcap Read Loaded: loaded (/etc/systemd/system/molochpcapread-selks.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2019-11-23 22:25:30 CET; 27min ago Main PID: 1426 (sh) Tasks: 6 (limit: 4915) CGroup: /system.slice/molochpcapread-selks.service ├─1426 /bin/sh -c /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m --copy --delete -R /data/nsm/ >> /data/moloch/logs/capture.log 2>&1 └─1427 /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m --copy --delete -R /data/nsm/

Nov 23 22:25:30 SELKS systemd[1]: molochpcapread-selks.service: Service hold-off time over, scheduling restart. Nov 23 22:25:30 SELKS systemd[1]: Stopped Moloch Pcap Read. Nov 23 22:25:30 SELKS systemd[1]: Started Moloch Pcap Read. scirius RUNNING pid 743, uptime 0:29:11 ii elasticsearch 6.8.5 all Elasticsearch is a distributed RESTful search engine built for the cloud. Reference documentation can be found at https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html and the 'Elasticsearch: The Definitive Guide' book can be found at https://www.elastic.co/guide/en/elasticsearch/guide/current/index.html ii elasticsearch-curator 5.8.1 amd64 Have indices in Elasticsearch? This is the tool for you!\n\nLike a museum curator manages the exhibits and collections on display, \nElasticsearch Curator helps you curate, or manage your indices. ii evebox 1:0.10.2 amd64 no description given ii kibana 6.8.5 amd64 Explore and visualize your Elasticsearch data ii kibana-dashboards-stamus 2019030501 amd64 Kibana 6 dashboard templates. ii logstash 1:6.8.5-1 all An extensible logging pipeline ii moloch 2.1.0-1 amd64 Moloch Full Packet System ii scirius 3.2.0-1 amd64 Django application to manage Suricata ruleset ii suricata 2019082101-0stamus0 amd64 Suricata open source multi-thread IDS/IPS/NSM system. Filesystem Type Size Used Avail Use% Mounted on udev devtmpfs 3.0G 0 3.0G 0% /dev tmpfs tmpfs 598M 8.0M 590M 2% /run /dev/sda1 ext4 229G 87G 130G 41% / tmpfs tmpfs 3.0G 0 3.0G 0% /dev/shm tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs tmpfs 3.0G 0 3.0G 0% /sys/fs/cgroup tmpfs tmpfs 598M 0 598M 0% /run/user/1001 root@SELKS:~#

root@SELKS:~# dpkg -l |grep ssl ii libflac8:amd64 1.3.2-1 amd64 Free Lossless Audio Codec - runtime C library ii libio-socket-ssl-perl 2.044-1 all Perl module implementing object oriented interface to SSL sockets ii libnet-smtp-ssl-perl 1.04-1 all Perl module providing SSL support to Net::SMTP ii libnet-ssleay-perl 1.80-1 amd64 Perl module for Secure Sockets Layer (SSL) ii libssl1.0.2:amd64 1.0.2t-1~deb9u1 amd64 Secure Sockets Layer toolkit - shared libraries ii libssl1.1:amd64 1.1.1d-0+deb10u2 amd64 Secure Sockets Layer toolkit - shared libraries ii openssl 1.1.1d-0+deb10u2 amd64 Secure Sockets Layer toolkit - cryptographic utility ii perl-openssl-defaults:amd64 3 amd64 version compatibility baseline for Perl OpenSSL packages root@SELKS:~#

michal25 commented 4 years ago

wget http://ftp.de.debian.org/debian/pool/main/o/openssl/openssl_1.1.1d-0+deb10u2_amd64.deb wget http://ftp.de.debian.org/debian/pool/main/o/openssl/libssl1.1_1.1.1d-0+deb10u2_amd64.deb wget http://ftp.de.debian.org/debian/pool/main/g/glibc/locales_2.28-10_all.deb wget http://ftp.de.debian.org/debian/pool/main/g/glibc/libc6_2.28-10_amd64.deb wget http://ftp.de.debian.org/debian/pool/main/g/glibc/libc-l10n_2.28-10_all.deb wget http://ftp.de.debian.org/debian/pool/main/g/glibc/libc-bin_2.28-10_amd64.deb

kenzoawa commented 4 years ago

Hey @michal25 did you installed those in any particular order or ? because i am getting almost on every one of them :

`dpkg: dependency problems prevent configuration of locales: locales depends on libc-bin (>> 2.28); however: Version of libc-bin on system is 2.24-11+deb9u4. locales depends on libc-l10n (>> 2.28); however: Version of libc-l10n on system is 2.24-11+deb9u4.

dpkg: error processing package locales (--install): dependency problems - leaving unconfigured Processing triggers for man-db (2.7.6.1-2) ... Errors were encountered while processing: locales `

michal25 commented 4 years ago

Yes, I received exact the same error message. Try to apt upgrade In my case this command passed ok and solved the dependencies.

After this, you have to restart all the SELKS device and moloch will start OK.

kenzoawa commented 4 years ago

Still getting the same error, rebooted the server and now some elasticsearch cannot start.

I've tried dpkg -i on all the files and apt upgrade, reboot and no luck. If you can share some thoughts that would be much appreciated.

michal25 commented 4 years ago

What exactly reports apt upgrade?

kenzoawa commented 4 years ago

root@SELKS:/home/admin# apt upgrade Reading package lists... Done Building dependency tree Reading state information... Done You might want to run 'apt --fix-broken install' to correct these. The following packages have unmet dependencies: libc-bin : Depends: libc6 (> 2.28) but 2.24-11+deb9u4 is installed libssl1.1 : Depends: libc6 (>= 2.25) but 2.24-11+deb9u4 is installed E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).

and after you run apt --fix-broken install

Reading package lists... Done Building dependency tree
Reading state information... Done Correcting dependencies... failed. The following packages have unmet dependencies: libc-bin : Depends: libc6 (> 2.28) but 2.24-11+deb9u4 is installed libssl1.1 : Depends: libc6 (>= 2.25) but 2.24-11+deb9u4 is installed E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages. E: Unable to correct dependencies

michal25 commented 4 years ago

what shows dpkg -l|grep ssl?

you need this output root@SELKS:~# dpkg -l|grep ssl ii libflac8:amd64 1.3.2-1 amd64 Free Lossless Audio Codec - runtime C library ii libio-socket-ssl-perl 2.044-1 all Perl module implementing object oriented interface to SSL sockets ii libnet-smtp-ssl-perl 1.04-1 all Perl module providing SSL support to Net::SMTP ii libnet-ssleay-perl 1.80-1 amd64 Perl module for Secure Sockets Layer (SSL) ii libssl1.0.2:amd64 1.0.2t-1~deb9u1 amd64 Secure Sockets Layer toolkit - shared libraries ii libssl1.1:amd64 1.1.1d-0+deb10u2 amd64 Secure Sockets Layer toolkit - shared libraries ii openssl 1.1.1d-0+deb10u2 amd64 Secure Sockets Layer toolkit - cryptographic utility ii perl-openssl-defaults:amd64 3 amd64 version compatibility baseline for Perl OpenSSL packages root@SELKS:~#

kenzoawa commented 4 years ago

root@SELKS:/home/admin# dpkg -l|grep ssl ii libflac8:amd64 1.3.2-1 amd64 Free Lossless Audio Codec - runtime C library ii libio-socket-ssl-perl 2.044-1 all Perl module implementing object oriented interface to SSL sockets ii libnet-smtp-ssl-perl 1.04-1 all Perl module providing SSL support to Net::SMTP ii libnet-ssleay-perl 1.80-1 amd64 Perl module for Secure Sockets Layer (SSL) ii libssl1.0.2:amd64 1.0.2t-1~deb9u1 amd64 Secure Sockets Layer toolkit - shared libraries ii libssl1.1:amd64 1.1.1d-0+deb10u2 amd64 Secure Sockets Layer toolkit - shared libraries ii openssl 1.1.1d-0+deb10u2 amd64 Secure Sockets Layer toolkit - cryptographic utility ii perl-openssl-defaults:amd64 3 amd64 version compatibility baseline for Perl OpenSSL packages ii python-openssl 16.2.0-1 all Python 2 wrapper around the OpenSSL library

kenzoawa commented 4 years ago

Still no luck, removing - adding this for OpenSSL seems like broke more libraries. Thanks for helping btw. If you can share your bash history in order to upgrade the packages in some order, that would be much helpful.

michal25 commented 4 years ago

So, only locales problem?

try to install locales again.

kenzoawa commented 4 years ago

will try and report back - also considering using beaver instead of stretch, maybe that would solve this openssl 1.1.1 problem..

pevma commented 4 years ago

Ok , please let us know if it solves the issue for you. I am guessing @michal25 did add in also the Buster repos to /apt/sources.d or did the whole upgrade?

michal25 commented 4 years ago

On this device I only wget this binaries wget http://ftp.de.debian.org/debian/pool/main/o/openssl/openssl_1.1.1d-0+deb10u2_amd64.deb wget http://ftp.de.debian.org/debian/pool/main/o/openssl/libssl1.1_1.1.1d-0+deb10u2_amd64.deb wget http://ftp.de.debian.org/debian/pool/main/g/glibc/locales_2.28-10_all.deb wget http://ftp.de.debian.org/debian/pool/main/g/glibc/libc6_2.28-10_amd64.deb wget http://ftp.de.debian.org/debian/pool/main/g/glibc/libc-l10n_2.28-10_all.deb wget http://ftp.de.debian.org/debian/pool/main/g/glibc/libc-bin_2.28-10_amd64.deb

As root, upgraded SELKS selks-upgrade_stamus

after this molochpcapread-selks.service failed to start so dpkg -i openssl_1.1.1d-0+deb10u2_amd64.deb libc6_2.28-10_amd64.deb libc-bin_2.28-10_amd64.deb libc-l10n_2.28-10_all.deb libssl1.1_1.1.1d-0+deb10u2_amd64.deb locales_2.28-10_all.deb

After this, I received the error message about dependencies on libc and locales.

I tried

apt upgrade

and this command solved the dependencies and finishes the dpkg -i install.

after this

reboot

and SELKS device started all scirius services OK.

No /apt/sources.d changes, the SELKS device is still running on Debian 9.

The debian 10 solution is described in this task from this timestamp https://github.com/StamusNetworks/SELKS/issues/204#issuecomment-557050830 to this timestamp https://github.com/StamusNetworks/SELKS/issues/204#issuecomment-557623400

kenzoawa commented 4 years ago

Unfortunately, this doesn't work on Debian 9.10 Stretch

Followed all the steps and :

Building dependency tree Reading state information... Done You might want to run 'apt --fix-broken install' to correct these. The following packages have unmet dependencies: libc-bin : Depends: libc6 (> 2.28) but 2.24-11+deb9u4 is installed openssl : Depends: libssl1.1 (>= 1.1.1) but 1.1.0l-1~deb9u1 is installed E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).

kenzoawa commented 4 years ago

Followed also this steps : https://github.com/Nimdy/SELKS-Install-from-source

FYI i am trying to install it on AWS.

pevma commented 4 years ago

Did you run apt --fix-broken install ?

kenzoawa commented 4 years ago

Yes - and i got this message.

root@ip-192-168-215-7:/home/admin# apt --fix-broken install Reading package lists... Done Building dependency tree
Reading state information... Done Correcting dependencies... Done The following packages were automatically installed and are no longer required: docutils-common libjbig0 libjpeg62-turbo liblcms2-2 libpaper-utils libpaper1 libpython-stdlib libpython2.7-minimal libpython2.7-stdlib libtiff5 libwebp6 libwebpdemux2 libwebpmux2 libyaml-0-2 net-tools python python-cffi-backend python-chardet python-cryptography python-enum34 python-idna python-ipaddress python-minimal python-openssl python-pkg-resources python-pyasn1 python-setuptools python-six python-urllib3 python2.7 python2.7-minimal python3-blinker python3-cffi-backend python3-chardet python3-colorama python3-configobj python3-cryptography python3-dateutil python3-debian python3-docutils python3-idna python3-jinja2 python3-jmespath python3-json-pointer python3-jsonpatch python3-jwt python3-markupsafe python3-oauthlib python3-pil python3-pkg-resources python3-prettytable python3-pyasn1 python3-pycurl python3-pygments python3-roman python3-rsa python3-setuptools python3-six python3-urllib3 python3-yaml Use 'sudo apt autoremove' to remove them. The following packages will be REMOVED: awscli ca-certificates cloud-init libc-bin locales openssl python-boto python-requests python3-boto python3-botocore python3-debianbts python3-httplib2 python3-pysimplesoap python3-reportbug

python3-requests python3-s3transfer reportbug WARNING: The following essential packages will be removed. This should NOT be done unless you know exactly what you are doing! libc-bin 0 upgraded, 0 newly installed, 17 to remove and 0 not upgraded. 3 not fully installed or removed. After this operation, 56,4 MB disk space will be freed. You are about to do something potentially harmful. To continue type in the phrase 'Yes, do as I say!' ?]

And after this some PATH is broken, total mess.

dpkg: warning: 'ldconfig' not found in PATH or not executable dpkg: error: 1 expected program not found in PATH or not executable Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin

Any chance SELKS would run on Ubuntu Server 18.04 ?

pevma commented 4 years ago

For AWS installs you could maybe follow the user contributed guide from here - https://github.com/Nimdy/SELKS-Install-from-source ?

kenzoawa commented 4 years ago

I was already following that guide but got the same error with moloch pcap. Upgraded to Debian Buster, got dozens of open jdk error, no /data/nsm/ path ..

Almost hopeless now, on a verge of quitting installing SELKS on aws.

kenzoawa commented 4 years ago

Ok - fortune favors the bold :)

  1. Install SELKS from https://github.com/Nimdy/SELKS-Install-from-source
  2. Upgrade OS to Buster
  3. apt --fix if python error minimal problem occurs
  4. run ./selks-upgrade
  5. Reboot
  6. run /.selks-first-time-setup

No reason to download additional packages 

root@ip-192-168-192-236:/usr/bin# dpkg -l | grep ssl
ii  libflac8:amd64                  1.3.2-3                      amd64        Free Lossless Audio Codec - runtime C library
ii  libio-socket-ssl-perl           2.060-3                      all          Perl module implementing object oriented interface to SSL sockets
ii  libnet-smtp-ssl-perl            1.04-1                       all          Perl module providing SSL support to Net::SMTP
ii  libnet-ssleay-perl              1.85-2+b1                    amd64        Perl module for Secure Sockets Layer (SSL)
ii  libssl1.0.2:amd64               1.0.2t-1~deb9u1              amd64        Secure Sockets Layer toolkit - shared libraries
ii  libssl1.1:amd64                 1.1.1d-0+deb10u2             amd64        Secure Sockets Layer toolkit - shared libraries
ii  libzstd1:amd64                  1.3.8+dfsg-3                 amd64        fast lossless compression algorithm
ii  openssl                         1.1.1d-0+deb10u2             amd64        Secure Sockets Layer toolkit - cryptographic utility
ii  perl-openssl-defaults:amd64     3                            amd64        version compatibility baseline for Perl OpenSSL packages
ii  python-openssl                  19.0.0-1                     all          Python 2 wrapper around the OpenSSL library

all services are up and running, thanks guys ! :)

` root@ip-192-168-192-236:/usr/bin# selks-health-check_stamus ● suricata.service - LSB: Next Generation IDS/IPS Loaded: loaded (]8;;file://ip-192-168-192-236/etc/init.d/suricata/etc/init.d/suricata]8;;; generated) Active: active (running) since Tue 2019-11-26 12:34:24 UTC; 1min 4s ago Docs: ]8;;man:systemd-sysv-generator(8)man:systemd-sysv-generator(8)]8;; Tasks: 8 (limit: 4915) Memory: 253.1M CGroup: /system.slice/suricata.service └─1219 /usr/bin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid --af-packet -D -v --user=logstash

ное 26 12:34:24 ip-192-168-192-236 systemd[1]: suricata.service: Succeeded. ное 26 12:34:24 ip-192-168-192-236 systemd[1]: Stopped LSB: Next Generation IDS/IPS. ное 26 12:34:24 ip-192-168-192-236 systemd[1]: Starting LSB: Next Generation IDS/IPS... ное 26 12:34:24 ip-192-168-192-236 suricata[1211]: Starting suricata in IDS (af-packet) mode... done. ное 26 12:34:24 ip-192-168-192-236 systemd[1]: Started LSB: Next Generation IDS/IPS. ● elasticsearch.service - Elasticsearch Loaded: loaded (]8;;file://ip-192-168-192-236/usr/lib/systemd/system/elasticsearch.service/usr/lib/systemd/system/elasticsearch.service]8;;; enabled; vendor preset: enabled) Active: active (running) since Tue 2019-11-26 12:30:53 UTC; 4min 34s ago Docs: ]8;;http://www.elastic.cohttp://www.elastic.co]8;; Main PID: 641 (java) Tasks: 62 (limit: 4915) Memory: 1.4G CGroup: /system.slice/elasticsearch.service ├─641 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress… └─841 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

ное 26 12:30:53 ip-192-168-192-236 systemd[1]: Started Elasticsearch. ное 26 12:30:54 ip-192-168-192-236 elasticsearch[641]: warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME ● logstash.service - logstash Loaded: loaded (]8;;file://ip-192-168-192-236/etc/systemd/system/logstash.service/etc/systemd/system/logstash.service]8;;; enabled; vendor preset: enabled) Active: active (running) since Tue 2019-11-26 12:35:24 UTC; 4s ago Main PID: 1480 (java) Tasks: 14 (limit: 4915) Memory: 248.6M CGroup: /system.slice/logstash.service └─1480 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djr…

ное 26 12:35:24 ip-192-168-192-236 systemd[1]: Started logstash. ● kibana.service - Kibana Loaded: loaded (]8;;file://ip-192-168-192-236/etc/systemd/system/kibana.service/etc/systemd/system/kibana.service]8;;; enabled; vendor preset: enabled) Active: active (running) since Tue 2019-11-26 12:30:53 UTC; 4min 34s ago Main PID: 634 (node) Tasks: 11 (limit: 4915) Memory: 500.5M CGroup: /system.slice/kibana.service └─634 /usr/share/kibana/bin/../node/bin/node --no-warnings --max-http-header-size=65536 /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

ное 26 12:31:40 ip-192-168-192-236 kibana[634]: {"type":"log","@timestamp":"2019-11-26T12:31:40Z","tags":["status","plugin:reporting@6.8.5","info"],"pid":634,"state":"green","me…information."} ное 26 12:31:40 ip-192-168-192-236 kibana[634]: {"type":"log","@timestamp":"2019-11-26T12:31:40Z","tags":["info","monitoring-ui","kibana-monitoring"],"pid":634,"message":"Starti…s collection"} ное 26 12:31:40 ip-192-168-192-236 kibana[634]: {"type":"log","@timestamp":"2019-11-26T12:31:40Z","tags":["status","plugin:security@6.8.5","info"],"pid":634,"state":"green","mes…information."} ное 26 12:31:40 ip-192-168-192-236 kibana[634]: {"type":"log","@timestamp":"2019-11-26T12:31:40Z","tags":["status","plugin:maps@6.8.5","info"],"pid":634,"state":"green","message…information."} ное 26 12:31:40 ip-192-168-192-236 kibana[634]: {"type":"log","@timestamp":"2019-11-26T12:31:40Z","tags":["license","info","xpack"],"pid":634,"message":"Imported license informa…atus: active"} ное 26 12:31:41 ip-192-168-192-236 kibana[634]: {"type":"log","@timestamp":"2019-11-26T12:31:41Z","tags":["listening","info"],"pid":634,"message":"Server running at http://localhost:5601"} ное 26 12:31:41 ip-192-168-192-236 kibana[634]: {"type":"log","@timestamp":"2019-11-26T12:31:41Z","tags":["status","plugin:spaces@6.8.5","info"],"pid":634,"state":"green","messa…information."} ное 26 12:34:42 ip-192-168-192-236 systemd[1]: kibana.service: Current command vanished from the unit file, execution of the command list won't be resumed. ное 26 12:35:13 ip-192-168-192-236 kibana[634]: {"type":"response","@timestamp":"2019-11-26T12:35:12Z","tags":[],"pid":634,"method":"post","statusCode":200,"req":{"url":"/api/spaces/space","m… ное 26 12:35:15 ip-192-168-192-236 kibana[634]: {"type":"response","@timestamp":"2019-11-26T12:35:13Z","tags":[],"pid":634,"method":"post","statusCode":200,"req":{"url":"/api/kibana/settings/… Hint: Some lines were ellipsized, use -l to show in full. ● evebox.service - EveBox Server Loaded: loaded (]8;;file://ip-192-168-192-236/lib/systemd/system/evebox.service/lib/systemd/system/evebox.service]8;;; enabled; vendor preset: enabled) Active: active (running) since Tue 2019-11-26 12:30:53 UTC; 4min 34s ago Main PID: 635 (evebox) Tasks: 8 (limit: 4915) Memory: 36.5M CGroup: /system.slice/evebox.service └─635 /usr/bin/evebox server

ное 26 12:31:37 ip-192-168-192-236 evebox[635]: "minimum_index_compatibility_version" : "5.0.0" ное 26 12:31:37 ip-192-168-192-236 evebox[635]: }, ное 26 12:31:37 ip-192-168-192-236 evebox[635]: "tagline" : "You Know, for Search" ное 26 12:31:37 ip-192-168-192-236 evebox[635]: } ное 26 12:31:40 ip-192-168-192-236 evebox[635]: 2019-11-26 12:31:40 (server.go:353) -- Connected to Elastic Search (version: 6.8.5) ное 26 12:31:40 ip-192-168-192-236 evebox[635]: 2019-11-26 12:31:40 (elasticsearch.go:199) -- Found templates [] ное 26 12:31:40 ip-192-168-192-236 evebox[635]: 2019-11-26 12:31:40 (elasticsearch.go:241) -- Failed to determine Elastic Search keyword suffix, things may not work right. ное 26 12:31:40 ip-192-168-192-236 evebox[635]: 2019-11-26 12:31:40 (server.go:131) -- Session reaper started ное 26 12:31:40 ip-192-168-192-236 evebox[635]: 2019-11-26 12:31:40 (server.go:165) -- Authentication disabled. ное 26 12:31:40 ip-192-168-192-236 evebox[635]: 2019-11-26 12:31:40 (server.go:276) -- Listening on 0.0.0.0:5636 ● molochviewer-selks.service - Moloch Viewer Loaded: loaded (]8;;file://ip-192-168-192-236/etc/systemd/system/molochviewer-selks.service/etc/systemd/system/molochviewer-selks.service]8;;; enabled; vendor preset: enabled) Active: active (running) since Tue 2019-11-26 12:32:30 UTC; 2min 58s ago Main PID: 999 (sh) Tasks: 12 (limit: 4915) Memory: 40.9M CGroup: /system.slice/molochviewer-selks.service ├─ 999 /bin/sh -c /data/moloch/bin/node viewer.js -c /data/moloch/etc/config.ini >> /data/moloch/logs/viewer.log 2>&1 └─1000 /data/moloch/bin/node viewer.js -c /data/moloch/etc/config.ini

ное 26 12:32:30 ip-192-168-192-236 systemd[1]: Started Moloch Viewer. ное 26 12:34:42 ip-192-168-192-236 systemd[1]: molochviewer-selks.service: Current command vanished from the unit file, execution of the command list won't be resumed. ● molochpcapread-selks.service - Moloch Pcap Read Loaded: loaded (]8;;file://ip-192-168-192-236/etc/systemd/system/molochpcapread-selks.service/etc/systemd/system/molochpcapread-selks.service]8;;; enabled; vendor preset: enabled) Active: active (running) since Tue 2019-11-26 12:32:24 UTC; 3min 4s ago Main PID: 983 (sh) Tasks: 6 (limit: 4915) Memory: 156.8M CGroup: /system.slice/molochpcapread-selks.service ├─983 /bin/sh -c /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m --copy --delete -R /data/nsm/ >> /data/moloch/logs/capture.log 2>&1 └─984 /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m --copy --delete -R /data/nsm/

ное 26 12:32:24 ip-192-168-192-236 systemd[1]: Started Moloch Pcap Read. ное 26 12:34:42 ip-192-168-192-236 systemd[1]: molochpcapread-selks.service: Current command vanished from the unit file, execution of the command list won't be resumed. scirius RUNNING pid 764, uptime 0:04:33 ii elasticsearch 6.8.5 all Elasticsearch is a distributed RESTful search engine built for the cloud. Reference documentation can be found at https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html and the 'Elasticsearch: The Definitive Guide' book can be found at https://www.elastic.co/guide/en/elasticsearch/guide/current/index.html ii elasticsearch-curator 5.8.1 amd64 Have indices in Elasticsearch? This is the tool for you!\n\nLike a museum curator manages the exhibits and collections on display, \nElasticsearch Curator helps you curate, or manage your indices. ii evebox 1:0.10.2 amd64 no description given ii kibana 6.8.5 amd64 Explore and visualize your Elasticsearch data ii kibana-dashboards-stamus 2019081202 amd64 Kibana 6 dashboard templates. ii logstash 1:6.8.5-1 all An extensible logging pipeline ii moloch 2.1.0-1 amd64 Moloch Full Packet System ii scirius 3.3.1-2 amd64 Django application to manage Suricata ruleset ii suricata 1:2019101501-0stamus0 amd64 Suricata open source multi-thread IDS/IPS/NSM system. Filesystem Type Size Used Avail Use% Mounted on udev devtmpfs 3,8G 0 3,8G 0% /dev tmpfs tmpfs 773M 8,4M 765M 2% /run /dev/nvme0n1p1 ext4 20G 6,0G 13G 32% / tmpfs tmpfs 3,8G 0 3,8G 0% /dev/shm tmpfs tmpfs 5,0M 0 5,0M 0% /run/lock tmpfs tmpfs 3,8G 0 3,8G 0% /sys/fs/cgroup tmpfs tmpfs 773M 0 773M 0% /run/user/1000 `

pevma commented 4 years ago

yes - unfortunately for Stretch you would need to stay with Moloch 2.0.1 or below due to the openssl 1.1.1 missing from Stretch.