search

StamusNetworks / SELKS

A Suricata based IDS/IPS/NSM distro
https://www.stamus-networks.com/open-source/#selks
GNU General Public License v3.0
1.44k stars 284 forks source link

Problems with SELKS after upgrading #337

Open christians86 opened 3 years ago

christians86 commented 3 years ago

Problem: I can not get a link to access moloch in the menu, and kibana shows a blank page. Health status says both are running, however I can only access mloch if I add while in a session "moloch/sessions"

Picture: image

Steps to reproduce for me:

  1. I started iso and installed, then ran first time setup without upgrading. Kibana worked fine from the link menu, molcoh was not present
  2. I ran update command and updated in terminal, then restarted the host
  3. I could no longer access kibana from the menu of selks and was shown a blank page

ps! I had to install twice so above is steps for 2. time, however the first install was

  1. Install iso
  2. run update
  3. run first time setup
  4. blank page in kibana when clicking

Suggestions?

pevma commented 3 years ago

To access Moloch you need to click on a FPC link in the Events lists in any Kibana dashboard. Or simply visit the https://yourselksiphere/moloch/address.

christians86 commented 3 years ago

https://yourselksiphere/moloch/

Getting

MaxRetryError at /moloch/

HTTPConnectionPool(host='localhost', port=8005): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f922d4b40f0>: Failed to establish a new connection: [Errno 111] Connection refused'))
Request Method: | GET -- | -- https://192.168.1.119/moloch/ 2.2.24 MaxRetryError HTTPConnectionPool(host='localhost', port=8005): Max retries exceeded with url: / (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) /usr/share/python/scirius/lib/python3.7/site-packages/urllib3/util/retry.py in increment, line 574 /usr/bin/python3 3.7.3 ['/usr/share/python/scirius/lib/python3.7/site-packages', '/usr/share/python/scirius/lib/python3.7/site-packages/git/ext/gitdb', '/usr/bin', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/usr/local/lib/python3.7/dist-packages', '/usr/lib/python3/dist-packages', '/usr/share/python/scirius/lib/python3.7/site-packages/gitdb/ext/smmap'] Fri, 17 Sep 2021 11:55:15 +0000

as a error now when trying, it worked fine yesterday, so not sure what the issue is now?

christians86 commented 3 years ago

But I'm not able to open kibana per the comment, suggestions there?

All i get is https://host/app/home and a blank black page

christians86 commented 3 years ago

So, I found the rootpassord and fixed nginx and now kibana works.

However moloch is still behaving like this through kibana and link

MaxRetryError at /moloch/sessions

HTTPConnectionPool(host='localhost', port=8005): Max retries exceeded with url: /sessions?expression=ip+%3D%3D+0.0.0.0+%26%26+port+%3D%3D+68+%26%26+ip+%3D%3D+255.255.255.255+%26%26+port+%3D%3D+67+%26%26+protocols+%3D%3D+udp&date=24 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd7499aec50>: Failed to establish a new connection: [Errno 111] Connection refused'))
Request Method: | GET -- | -- https://192.168.1.119/moloch/sessions?expression=ip%20%3D%3D%200.0.0.0%20%26%26%20port%20%3D%3D%2068%20%26%26%20ip%20%3D%3D%20255.255.255.255%20%26%26%20port%20%3D%3D%2067%20%26%26%20protocols%20%3D%3D%20udp&date=24 2.2.24 MaxRetryError HTTPConnectionPool(host='localhost', port=8005): Max retries exceeded with url: /sessions?expression=ip+%3D%3D+0.0.0.0+%26%26+port+%3D%3D+68+%26%26+ip+%3D%3D+255.255.255.255+%26%26+port+%3D%3D+67+%26%26+protocols+%3D%3D+udp&date=24 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) /usr/share/python/scirius/lib/python3.7/site-packages/urllib3/util/retry.py in increment, line 574 /usr/bin/python3 3.7.3 ['/usr/share/python/scirius/lib/python3.7/site-packages', '/usr/share/python/scirius/lib/python3.7/site-packages/git/ext/gitdb', '/usr/bin', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/usr/local/lib/python3.7/dist-packages', '/usr/lib/python3/dist-packages', '/usr/share/python/scirius/lib/python3.7/site-packages/gitdb/ext/smmap'] Fri, 17 Sep 2021 15:39:53 +0000
pevma commented 3 years ago

Did the first time setup finish ok without problems ? Did you use the selks upgrade command or regular apt upgrade? (ref: https://github.com/StamusNetworks/SELKS/wiki/SELKS-upgrades )

christians86 commented 3 years ago

1) Without problem exit with succsess or something 2) selks upgrade

However I can have done a db clean up also with selks command, can that have done something?

and might the solution be just to run

  1. new setup
  2. check for updates
  3. check that nginx config file is correct
  4. restart the entire vm

?

pevma commented 3 years ago

Yes - please check if the nginx file is as expected - https://github.com/StamusNetworks/SELKS/wiki/Kibana-did-not-load-properly

And could you please share the output of the health check ?

christians86 commented 3 years ago

The nginx file is a copy of what is in the link you gave, I have checked. Moloch failed prior and after that

Kibana worked fine when I edited the nginx file

Output of health check:

selks-user@SELKS:~$ selks-health-check_stamus ● suricata.service - LSB: Next Generation IDS/IPS Loaded: loaded (/etc/init.d/suricata; generated) Active: active (running) since Fri 2021-09-17 17:33:21 CEST; 37min ago Docs: man:systemd-sysv-generator(8) Process: 894 ExecStart=/etc/init.d/suricata start (code=exited, status=0/SUCCESS) Tasks: 14 (limit: 4915) Memory: 400.1M CGroup: /system.slice/suricata.service └─990 /usr/bin/suricata -c /etc/suricata/suricata.yaml --pidfile /va… ● elasticsearch.service - Elasticsearch Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2021-09-17 17:33:54 CEST; 36min ago Docs: https://www.elastic.co Main PID: 893 (java) Tasks: 118 (limit: 4915) Memory: 6.8G CGroup: /system.slice/elasticsearch.service ├─ 893 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.netwo… └─1305 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86… ● logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2021-09-17 17:33:20 CEST; 37min ago Main PID: 512 (java) Tasks: 59 (limit: 4915) Memory: 1.1G CGroup: /system.slice/logstash.service └─512 /usr/share/logstash/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMar… ● kibana.service - Kibana Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2021-09-17 17:33:21 CEST; 37min ago Docs: https://www.elastic.co Main PID: 898 (node) Tasks: 18 (limit: 4915) Memory: 517.9M CGroup: /system.slice/kibana.service ├─ 898 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/… └─1204 /usr/share/kibana/node/bin/node --preserve-symlinks-main --pr… ● evebox.service - EveBox Server Loaded: loaded (/lib/systemd/system/evebox.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2021-09-17 17:33:20 CEST; 37min ago Main PID: 493 (evebox) Tasks: 9 (limit: 4915) Memory: 20.3M CGroup: /system.slice/evebox.service └─493 /usr/bin/evebox server ● molochviewer-selks.service - Moloch Viewer Loaded: loaded (/etc/systemd/system/molochviewer-selks.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2021-09-17 17:33:21 CEST; 37min ago Main PID: 885 (sh) Tasks: 12 (limit: 4915) Memory: 85.7M CGroup: /system.slice/molochviewer-selks.service ├─885 /bin/sh -c /data/moloch/bin/node viewer.js -c /data/moloch/etc… └─892 /data/moloch/bin/node viewer.js -c /data/moloch/etc/config.ini ● molochpcapread-selks.service - Moloch Pcap Read Loaded: loaded (/etc/systemd/system/molochpcapread-selks.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2021-09-17 17:34:48 CEST; 35min ago Main PID: 1552 (sh) Tasks: 5 (limit: 4915) Memory: 306.6M CGroup: /system.slice/molochpcapread-selks.service ├─1552 /bin/sh -c /data/moloch/bin/moloch-capture -c /data/moloch/et… └─1553 /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.in… error: <class 'socket.error'>, [Errno 13] Permission denied: file: /usr/lib/python2.7/socket.py line: 228 ii elasticsearch 7.14.1 amd64 Distributed RESTful search engine built for the cloud ii elasticsearch-curator 5.8.4 amd64 Have indices in Elasticsearch? This is the tool for you!\n\nLike a museum curator manages the exhibits and collections on display, \nElasticsearch Curator helps you curate, or manage your indices. ii evebox 1:0.14.0 amd64 no description given ii kibana 7.14.1 amd64 Explore and visualize your Elasticsearch data ii kibana-dashboards-stamus 2020122001 amd64 Kibana 6 dashboard templates. ii logstash 1:7.14.1-1 amd64 An extensible logging pipeline ii moloch 3.0.0-1 amd64 Moloch Full Packet System ii scirius 3.7.0-6 amd64 Django application to manage Suricata ruleset ii suricata 1:2021090701-0stamus0 amd64 Suricata open source multi-thread IDS/IPS/NSM system. Filesystem Type Size Used Avail Use% Mounted on udev devtmpfs 5.8G 0 5.8G 0% /dev tmpfs tmpfs 1.2G 9.0M 1.2G 1% /run /dev/sda1 ext4 117G 11G 100G 10% / tmpfs tmpfs 5.8G 55M 5.7G 1% /dev/shm tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs tmpfs 5.8G 0 5.8G 0% /sys/fs/cgroup /dev/loop0 squashfs 100M 100M 0 100% /snap/core/11606 tmpfs
tmpfs 1.2G 12K 1.2G 1% /run/user/1000 selks-user@SELKS:~$

ps! Added the stacktrace in pdf from moloch to if that helps

stacktrace.pdf

christians86 commented 3 years ago

Trying first time setup gave this image

pevma commented 3 years ago

Ok That is at the end (probably ES was not yet ready) , so you might need to just reload those again

https://github.com/StamusNetworks/SELKS/wiki/How-to-load-or-update-dashboards#from-the-command-line

-- Regards, Peter Manev

On 18 Sep 2021, at 14:38, christians86 @.***> wrote:

 Trying first time setup gave this

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

christians86 commented 3 years ago

Ok, what about moloch then?

christians86 commented 3 years ago

I'm also getting these now:

image

Could latest update have broken a lot of things?

rezpc commented 3 years ago

For

Trying first time setup gave this image

Having the same issue with my install, I edited /etc/scirius/local_settings.py and added the following line: KIBANA7_DASHBOARDS_PATH = "/opt/selks/kibana6-dashboards/"

Rerun and it should complete, if not run the dashboards command: https://github.com/StamusNetworks/SELKS/wiki/First-time-setup#kibana-dashboards

I get the same issue with moloch, have to manually restart molochviewer.service and the dashboard comes back up (have to do this everytime after a server reboot.) Having issues getting moloch to capture traffic as well (crashes after a minute) so I'm thinking SELKS is probably out of date with the latest release of moloch (arkime.)

pevma commented 3 years ago

@christians86 - with regards to https://github.com/StamusNetworks/SELKS/issues/337#issuecomment-922324401 - if there is non of that type of data, it is a normal message

pevma commented 3 years ago

@rezpc with regards to - https://github.com/StamusNetworks/SELKS/issues/337#issuecomment-922408124 Moloch has a delay start iirc. Does the issue occur as well lets say after 5-10 min post restart ?

pevma commented 3 years ago

@christians86 - with regards to https://github.com/StamusNetworks/SELKS/issues/337#issuecomment-921916714 is there any errors in the moloch logs ? Did you enable FPC ( https://github.com/StamusNetworks/SELKS/wiki/First-time-setup#full-packet-capture-fpc ) just double checking which option ?

christians86 commented 3 years ago

@christians86 - with regards to #337 (comment) is there any errors in the moloch logs ? Did you enable FPC ( https://github.com/StamusNetworks/SELKS/wiki/First-time-setup#full-packet-capture-fpc ) just double checking which option ?

I sent you the stacktrace and what I found above, anywhere else I should look?

FPC was selected yes.

pevma commented 3 years ago

@christians86 - with regards to #337 (comment) is there any errors in the moloch logs ? Did you enable FPC ( https://github.com/StamusNetworks/SELKS/wiki/First-time-setup#full-packet-capture-fpc ) just double checking which option ?

I sent you the stacktrace and what I found above, anywhere else I should look?

FPC was selected yes.

Anything in /data/moloch/logs/, any err msgs there?

christians86 commented 3 years ago

image image image image image

pevma commented 2 years ago

It seems the connection to ES was refused? Anything else that might be interfering ?

christians86 commented 2 years ago

You would need to be more specific

christians86 commented 2 years ago

I made a new vm and the problem is the same there, so not sure what to look for as its a consistent feature of the image + update, and not something I have done

image

pevma commented 2 years ago

This might be ES not fully up yet. What I meant in my previous comment is that - no xpack other security tools locking down the system installed etc right ?

christians86 commented 2 years ago

nothing, this is your image and only your image running on the VM, nothing else

but ES is running, on the new one I have ES running fine and getting input, so then this could be a question of moloch being to early to the party and thinking ES is not working, and not retrying later?

pevma commented 2 years ago

Ok - so you have no issue with Kibana right ? Dashboards appear there everything is ok ? Let me try to reproduce

christians86 commented 2 years ago

The reinstall

where I set up a new Vm -> then tok update and wrote over all configs -> then took the nginx config you linked and then restarted the VM works perfectly image

christians86 commented 2 years ago

except from moloch links..

rezpc commented 2 years ago

@pevma sorry for the late reply, selks-health-status shows everything green but I get the same Moloch MaxRetryError message that was posted earlier in this thread (even after the server has been running 5-10 minutes.) I restart molochviewer.service and the dashboard comes up. (FPC w/Retain)

It might just be a config thing on my end, the guide doesn't mention if we need to configure anything within moloch or not, the one thing I change is the interface to listen on (interface=), seems to capture for a bit then the service dies. restarted the server today and it's all green but no pcap in moloch again, might just try rebuilding it again.

edit: user-error on changing the moloch interface, that was likely causing pcap service to fail. rebuilt vm, no issues with moloch pcap outside of having to manually restart viewer service to get past the maxretryerror issue.

rezpc commented 2 years ago

new build, right off the bat Moloch has the MaxTryError. 1. run initial setup, run update, edit nginx, same issues:

  1. Have to add KIBANA7_DASHBOARDS_PATH = "/opt/selks/kibana6-dashboards/" to local_settings.py to get the dashboards to reset (using either command on the wiki)
  2. MaxRetryError on Moloch regardless of how long the server has been up. Restarting molochviewer-selks.service fixes issue.
● suricata.service - LSB: Next Generation IDS/IPS
   Loaded: loaded (/etc/init.d/suricata; generated)
   Active: active (running) since Tue 2021-09-21 14:52:09 CDT; 22min ago
     Docs: man:systemd-sysv-generator(8)
    Tasks: 10 (limit: 4915)
   Memory: 316.6M
   CGroup: /system.slice/suricata.service
           └─1150 /usr/bin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid --af-packet -D -v --user=logstash

Sep 21 14:52:08 SELKS systemd[1]: Starting LSB: Next Generation IDS/IPS...
Sep 21 14:52:09 SELKS suricata[1071]: Starting suricata in IDS (af-packet) mode... done.
Sep 21 14:52:09 SELKS systemd[1]: Started LSB: Next Generation IDS/IPS.
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2021-09-21 14:52:35 CDT; 21min ago
     Docs: https://www.elastic.co
 Main PID: 1070 (java)
    Tasks: 90 (limit: 4915)
   Memory: 4.1G
   CGroup: /system.slice/elasticsearch.service
           ├─1070 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=SPI,COMPAT --add-opens=java.base/java.io=ALL-UNNAMED -XX:+UseG1GC -Djava.io.tmpdir=/tmp/elasticsearch-7577087113500054168 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Xms3989m -Xmx3989m -XX:MaxDirectMemorySize=2091909120 -XX:G1HeapRegionSize=4m -XX:InitiatingHeapOccupancyPercent=30 -XX:G1ReservePercent=15 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=deb -Des.bundled_jdk=true -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet
           └─1425 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Sep 21 14:52:08 SELKS systemd[1]: Starting Elasticsearch...
Sep 21 14:52:35 SELKS systemd[1]: Started Elasticsearch.
● logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2021-09-21 14:52:07 CDT; 22min ago
 Main PID: 705 (java)
    Tasks: 45 (limit: 4915)
   Memory: 893.8M
   CGroup: /system.slice/logstash.service
           └─705 /usr/share/logstash/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -Djruby.regexp.interruptible=true -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -Dlog4j2.isThreadContextMapInheritable=true -cp /usr/share/logstash/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/share/logstash/logstash-core/lib/jars/checker-compat-qual-2.0.0.jar:/usr/share/logstash/logstash-core/lib/jars/commons-codec-1.14.jar:/usr/share/logstash/logstash-core/lib/jars/commons-compiler-3.1.0.jar:/usr/share/logstash/logstash-core/lib/jars/commons-logging-1.2.jar:/usr/share/logstash/logstash-core/lib/jars/error_prone_annotations-2.1.3.jar:/usr/share/logstash/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/gradle-license-report-0.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/guava-24.1.1-jre.jar:/usr/share/logstash/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-annotations-2.9.10.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-core-2.9.10.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-databind-2.9.10.8.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.10.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-dataformat-yaml-2.9.10.jar:/usr/share/logstash/logstash-core/lib/jars/janino-3.1.0.jar:/usr/share/logstash/logstash-core/lib/jars/javassist-3.26.0-GA.jar:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.19.0.jar:/usr/share/logstash/logstash-core/lib/jars/jsr305-1.3.9.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-1.2-api-2.14.0.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-api-2.14.0.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.14.0.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-jcl-2.14.0.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.14.0.jar:/usr/share/logstash/logstash-core/lib/jars/logstash-core.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/reflections-0.9.11.jar:/usr/share/logstash/logstash-core/lib/jars/slf4j-api-1.7.30.jar:/usr/share/logstash/logstash-core/lib/jars/snakeyaml-1.23.jar org.logstash.Logstash --path.settings /etc/logstash

Sep 21 14:53:08 SELKS logstash[705]: [2021-09-21T14:53:08,324][INFO ][logstash.filters.geoip.downloadmanager] new database version detected? false
Sep 21 14:53:08 SELKS logstash[705]: [2021-09-21T14:53:08,476][INFO ][logstash.filters.geoip.databasemanager][main] By not manually configuring a database path with `database =>`, you accepted and agreed MaxMind EULA. For more details please visit https://www.maxmind.com/en/geolite2/eula
Sep 21 14:53:08 SELKS logstash[705]: [2021-09-21T14:53:08,478][INFO ][logstash.filters.geoip   ][main] Using geoip database {:path=>"/var/lib/logstash/plugins/filters/geoip/1632252402/GeoLite2-City.mmdb"}
Sep 21 14:53:08 SELKS logstash[705]: [2021-09-21T14:53:08,837][INFO ][logstash.filters.geoip.databasemanager][main] By not manually configuring a database path with `database =>`, you accepted and agreed MaxMind EULA. For more details please visit https://www.maxmind.com/en/geolite2/eula
Sep 21 14:53:08 SELKS logstash[705]: [2021-09-21T14:53:08,838][INFO ][logstash.filters.geoip   ][main] Using geoip database {:path=>"/var/lib/logstash/plugins/filters/geoip/1632252402/GeoLite2-City.mmdb"}
Sep 21 14:53:08 SELKS logstash[705]: [2021-09-21T14:53:08,925][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/logstash.conf", "/etc/logstash/conf.d/scirius-logstash.conf"], :thread=>"#<Thread:0x108d58e3 run>"}
Sep 21 14:53:10 SELKS logstash[705]: [2021-09-21T14:53:10,324][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.39}
Sep 21 14:53:10 SELKS logstash[705]: [2021-09-21T14:53:10,388][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
Sep 21 14:53:10 SELKS logstash[705]: [2021-09-21T14:53:10,432][INFO ][filewatch.observingtail  ][main][d4aef1d642dafd3cc0ec28e9e79530daa4bc5c58ba6b725806ceff6c4cfb1cf0] START, creating Discoverer, Watch with file and sincedb collections
Sep 21 14:53:10 SELKS logstash[705]: [2021-09-21T14:53:10,435][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2021-09-21 14:52:08 CDT; 22min ago
     Docs: https://www.elastic.co
 Main PID: 1068 (node)
    Tasks: 18 (limit: 4915)
   Memory: 437.2M
   CGroup: /system.slice/kibana.service
           ├─1068 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist --logging.dest=/var/log/kibana/kibana.log --pid.file=/run/kibana/kibana.pid
           └─1275 /usr/share/kibana/node/bin/node --preserve-symlinks-main --preserve-symlinks /usr/share/kibana/src/cli/dist --logging.dest=/var/log/kibana/kibana.log --pid.file=/run/kibana/kibana.pid

Sep 21 14:52:08 SELKS systemd[1]: Started Kibana.
● evebox.service - EveBox Server
   Loaded: loaded (/lib/systemd/system/evebox.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2021-09-21 14:52:07 CDT; 22min ago
 Main PID: 702 (evebox)
    Tasks: 5 (limit: 4915)
   Memory: 11.1M
   CGroup: /system.slice/evebox.service
           └─702 /usr/bin/evebox server

Sep 21 14:52:14 SELKS evebox[702]: 2021-09-21 14:52:14  WARN evebox::server::main: Failed to get Elasticsearch version, will try again: request: error sending request for url (http://localhost:9200/): error trying to connect: tcp connect error: Connection refused (os error 111)
Sep 21 14:52:17 SELKS evebox[702]: 2021-09-21 14:52:17  WARN evebox::server::main: Failed to get Elasticsearch version, will try again: request: error sending request for url (http://localhost:9200/): error trying to connect: tcp connect error: Connection refused (os error 111)
Sep 21 14:52:19 SELKS evebox[702]: 2021-09-21 14:52:19  WARN evebox::server::main: Failed to get Elasticsearch version, will try again: request: error sending request for url (http://localhost:9200/): error trying to connect: tcp connect error: Connection refused (os error 111)
Sep 21 14:52:22 SELKS evebox[702]: 2021-09-21 14:52:22  WARN evebox::server::main: Failed to get Elasticsearch version, will try again: request: error sending request for url (http://localhost:9200/): error trying to connect: tcp connect error: Connection refused (os error 111)
Sep 21 14:52:25 SELKS evebox[702]: 2021-09-21 14:52:25  WARN evebox::server::main: Failed to get Elasticsearch version, will try again: request: error sending request for url (http://localhost:9200/): error trying to connect: tcp connect error: Connection refused (os error 111)
Sep 21 14:52:28 SELKS evebox[702]: 2021-09-21 14:52:28  WARN evebox::server::main: Failed to get Elasticsearch version, will try again: request: error sending request for url (http://localhost:9200/): error trying to connect: tcp connect error: Connection refused (os error 111)
Sep 21 14:52:31 SELKS evebox[702]: 2021-09-21 14:52:31  WARN evebox::server::main: Failed to get Elasticsearch version, will try again: request: error sending request for url (http://localhost:9200/): error trying to connect: tcp connect error: Connection refused (os error 111)
Sep 21 14:52:34 SELKS evebox[702]: 2021-09-21 14:52:34  WARN evebox::server::main: Failed to get Elasticsearch version, will try again: request: error sending request for url (http://localhost:9200/): error trying to connect: tcp connect error: Connection refused (os error 111)
Sep 21 14:52:37 SELKS evebox[702]: 2021-09-21 14:52:37  INFO evebox::server::main: Found Elasticsearch version 7.14.2 at http://localhost:9200
Sep 21 14:52:37 SELKS evebox[702]: 2021-09-21 14:52:37  INFO evebox::server::main: Starting server on 127.0.0.1:5636, tls=false
● molochviewer-selks.service - Moloch Viewer
   Loaded: loaded (/etc/systemd/system/molochviewer-selks.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2021-09-21 14:52:08 CDT; 22min ago
 Main PID: 1067 (sh)
    Tasks: 12 (limit: 4915)
   Memory: 72.4M
   CGroup: /system.slice/molochviewer-selks.service
           ├─1067 /bin/sh -c /data/moloch/bin/node viewer.js -c /data/moloch/etc/config.ini >> /data/moloch/logs/viewer.log 2>&1
           └─1079 /data/moloch/bin/node viewer.js -c /data/moloch/etc/config.ini

Sep 21 14:52:08 SELKS systemd[1]: Started Moloch Viewer.
Sep 21 15:13:36 SELKS systemd[1]: molochviewer-selks.service: Current command vanished from the unit file, execution of the command list won't be resumed.
● molochpcapread-selks.service - Moloch Pcap Read
   Loaded: loaded (/etc/systemd/system/molochpcapread-selks.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2021-09-21 14:53:38 CDT; 20min ago
 Main PID: 1592 (sh)
    Tasks: 6 (limit: 4915)
   Memory: 144.4M
   CGroup: /system.slice/molochpcapread-selks.service
           ├─1592 /bin/sh -c /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m --copy --delete -R /data/nsm/  >> /data/moloch/logs/capture.log 2>&1
           └─1593 /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m --copy --delete -R /data/nsm/

Sep 21 14:53:38 SELKS systemd[1]: molochpcapread-selks.service: Service RestartSec=1min 30s expired, scheduling restart.
Sep 21 14:53:38 SELKS systemd[1]: molochpcapread-selks.service: Scheduled restart job, restart counter is at 1.
Sep 21 14:53:38 SELKS systemd[1]: Stopped Moloch Pcap Read.
Sep 21 14:53:38 SELKS systemd[1]: Started Moloch Pcap Read.
Sep 21 15:13:36 SELKS systemd[1]: molochpcapread-selks.service: Current command vanished from the unit file, execution of the command list won't be resumed.
scirius                          RUNNING   pid 1274, uptime 0:22:18
ii  elasticsearch                         7.14.2                                  amd64        Distributed RESTful search engine built for the cloud
ii  elasticsearch-curator                 5.8.4                                   amd64        Have indices in Elasticsearch? This is the tool for you!\n\nLike a museum curator manages the exhibits and collections on display, \nElasticsearch Curator helps you curate, or manage your indices.
ii  evebox                                1:0.14.0                                amd64        no description given
ii  kibana                                7.14.2                                  amd64        Explore and visualize your Elasticsearch data
ii  kibana-dashboards-stamus              2020122001                              amd64        Kibana 6 dashboard templates.
ii  logstash                              1:7.14.2-1                              amd64        An extensible logging pipeline
ii  moloch                                3.0.0-1                                 amd64        Moloch Full Packet System
ii  scirius                               3.7.0-6                                 amd64        Django application to manage Suricata ruleset
ii  suricata                              1:2021090701-0stamus0                   amd64        Suricata open source multi-thread IDS/IPS/NSM system.
Filesystem                 Type      Size  Used Avail Use% Mounted on
udev                       devtmpfs  3.9G     0  3.9G   0% /dev
tmpfs                      tmpfs     798M  9.1M  789M   2% /run
/dev/mapper/SELKS--vg-root ext4      188G  9.3G  169G   6% /
tmpfs                      tmpfs     3.9G   29M  3.9G   1% /dev/shm
tmpfs                      tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs                      tmpfs     3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda1                  ext2      236M  104M  120M  47% /boot
tmpfs                      tmpfs     798M   12K  798M   1% /run/user/1000
pevma commented 2 years ago

@christians86 and @rezpc - i think what you need to do is : 1 - make sure elasticsearch service is running systemctl status elasticsearch

2 - restart the viewer: systemctl restart molochviewer-selks
3 - try connecting/opening again the viewer

christians86 commented 2 years ago

Well, what I did, and lets see if it holds was to just run a second upgrade now, and then all of it worked suddenly. I got a kibana upgrade which wasn't there before as I did run upgrade last time to. But by the looks of it, things are working again...I'm just not sure if its stable

rezpc commented 2 years ago

@pevma the only consistent issue I have is having to manually restart molochview-selks anytime the VM is rebooted. Outside of that, it's been working pretty flawlessly. Thought about just setting a timer to restart it after a few minutes on bootup. Really been enjoying playing with SELKS (been feeding it malware PCAP to test various suricata triggers,) going to use it for an upcoming CTF event and see how it plays out.

pevma commented 2 years ago

Great to hear! Yes delayed start might be an option (or dependency on ES start rather). I think you can experience such an issue as when it starts it tries to connect to ES right away and if ES is not yet ready , it would do the err, hence the later restart needed.

christians86 commented 2 years ago

Still works stable, however I suggest that user manuals ask you to run upgrade twice, and before you change any configs just to overwrite maybe? Thats what I did, then I changed configs and restarted and since it has worked.