Closed firion4ik closed 4 years ago
pevma
commented
4 years ago Seems moloch was not upgraded due to wget/pull err - can you try to manually download -
wget https://files.molo.ch/builds/ubuntu-18.04/moloch_2.2.3-1_amd64.deb
dpkg -i moloch_2.2.3-1_amd64.deb?
firion4ik
commented
4 years ago Manual upgrade went well. Then I performed the initial setup script again, chose FPC_Retain, but health check still shows that it failed.
Checked the log and now it it complaining about openssl version:
/data/moloch/bin/moloch-capture: /usr/lib/x86_64-linux-gnu/libssl.so.1.1: versionOPENSSL_1_1_1' not found (required by /data/moloch/bin/moloch-capture)`
Found your comment here https://github.com/StamusNetworks/SELKS/issues/204 👍
pevma commented on Nov 27, 2019
yes - unfortunately for Stretch you would need to stay with Moloch 2.0.1 or below due to the openssl 1.1.1 missing from Stretch.I have Debian GNU/Linux 9 (stretch). What would you suggest to do?
pevma
commented
4 years ago You can try to upgrade - https://github.com/StamusNetworks/SELKS/wiki/SELKS-5.0-Buster-upgrade
firion4ik
commented
4 years ago Performed the upgrade, but the service is still failing to start, though the capture.log looks a bit different:
cat /data/moloch/logs/capture.log May 12 15:09:06 http.c:304 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/_template/sessions2_template?filter_path=**._meta&include_type_name=true 0/80 0ms 195ms May 12 15:09:06 http.c:304 moloch_http_send_sync(): 1/1 SYNC 404 http://localhost:9200/sequence/sequence/fn-inf8603 0/76 0ms 1ms May 12 15:09:07 http.c:304 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/files/file/_search?size=1&sort=num:desc&q=node:inf8603 0/137 0ms 1078ms May 12 15:09:07 http.c:304 moloch_http_send_sync(): 1/1 SYNC 404 http://localhost:9200/stats/stat/inf8603 0/66 0ms 1ms May 12 15:09:07 http.c:304 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/fields/field/_search?size=3000 0/13200 0ms 5ms May 12 15:09:07 config.c:819 moloch_config_monitor_file(): Couldn't stat country file file /data/moloch/etc/GeoLite2-Country.mmdb error No such file or directory May 12 15:10:38 http.c:304 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/_template/sessions2_template?filter_path=**._meta&include_type_name=true 0/80 0ms 2ms May 12 15:10:38 http.c:304 moloch_http_send_sync(): 1/1 SYNC 404 http://localhost:9200/sequence/sequence/fn-inf8603 0/76 0ms 0ms May 12 15:10:38 http.c:304 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/files/file/_search?size=1&sort=num:desc&q=node:inf8603 0/134 0ms 1ms May 12 15:10:38 http.c:304 moloch_http_send_sync(): 1/1 SYNC 404 http://localhost:9200/stats/stat/inf8603 0/66 0ms 0ms May 12 15:10:38 http.c:304 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/fields/field/_search?size=3000 0/13200 0ms 1ms May 12 15:10:38 config.c:819 moloch_config_monitor_file(): Couldn't stat country file file /data/moloch/etc/GeoLite2-Country.mmdb error No such file or directory May 12 15:12:09 http.c:304 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/_template/sessions2_template?filter_path=**._meta&include_type_name=true 0/80 0ms 689ms May 12 15:12:09 http.c:304 moloch_http_send_sync(): 1/1 SYNC 404 http://localhost:9200/sequence/sequence/fn-inf8603 0/76 0ms 0ms May 12 15:12:09 http.c:304 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/files/file/_search?size=1&sort=num:desc&q=node:inf8603 0/134 0ms 1ms May 12 15:12:09 http.c:304 moloch_http_send_sync(): 1/1 SYNC 404 http://localhost:9200/stats/stat/inf8603 0/66 0ms 0ms May 12 15:12:09 http.c:304 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/fields/field/_search?size=3000 0/13200 0ms 1ms May 12 15:12:09 config.c:819 moloch_config_monitor_file(): Couldn't stat country file file /data/moloch/etc/GeoLite2-Country.mmdb error No such file or directory May 12 15:13:39 http.c:304 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/_template/sessions2_template?filter_path=**._meta&include_type_name=true 0/80 0ms 802ms May 12 15:13:39 http.c:304 moloch_http_send_sync(): 1/1 SYNC 404 http://localhost:9200/sequence/sequence/fn-inf8603 0/76 0ms 1ms May 12 15:13:39 http.c:304 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/files/file/_search?size=1&sort=num:desc&q=node:inf8603 0/134 0ms 2ms May 12 15:13:39 http.c:304 moloch_http_send_sync(): 1/1 SYNC 404 http://localhost:9200/stats/stat/inf8603 0/66 0ms 7ms May 12 15:13:39 http.c:304 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/fields/field/_search?size=3000 0/13200 0ms 6ms May 12 15:13:39 config.c:819 moloch_config_monitor_file(): Couldn't stat country file file /data/moloch/etc/GeoLite2-Country.mmdb error No such file or directory
Health check:
● suricata.service - LSB: Next Generation IDS/IPS Loaded: loaded (/etc/init.d/suricata; generated) Active: active (running) since Tue 2020-05-12 15:07:50 BST; 26min ago Docs: man:systemd-sysv-generator(8) Tasks: 22 (limit: 4915) Memory: 4.2G CGroup: /system.slice/suricata.service └─24523 /usr/bin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid --af-packet -D -v --user=logstash
May 12 15:07:50 inf8603.swe6.unibet.com systemd[1]: Starting LSB: Next Generation IDS/IPS... May 12 15:07:50 inf8603.swe6.unibet.com suricata[24508]: Starting suricata in IDS (af-packet) mode... done. May 12 15:07:50 inf8603.swe6.unibet.com systemd[1]: Started LSB: Next Generation IDS/IPS. ● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2020-05-12 15:03:23 BST; 30min ago Docs: http://www.elastic.co Main PID: 22930 (java) Tasks: 150 (limit: 4915) Memory: 3.7G CGroup: /system.slice/elasticsearch.service ├─22930 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTou… └─23080 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
May 12 15:03:23 inf8603.swe6.unibet.com systemd[1]: Started Elasticsearch. May 12 15:03:23 inf8603.swe6.unibet.com elasticsearch[22930]: warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME ● logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2020-05-12 15:03:32 BST; 30min ago Main PID: 23119 (java) Tasks: 48 (limit: 4915) Memory: 1008.7M CGroup: /system.slice/logstash.service └─23119 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.ji…
May 12 15:04:01 inf8603.swe6.unibet.com logstash[23119]: [2020-05-12T15:04:01,019][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/etc/logstash/elasticsearch6-template.json"}
May 12 15:04:01 inf8603.swe6.unibet.com logstash[23119]: [2020-05-12T15:04:01,021][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"numbe…
May 12 15:04:01 inf8603.swe6.unibet.com logstash[23119]: [2020-05-12T15:04:01,040][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/logstash
May 12 15:04:01 inf8603.swe6.unibet.com logstash[23119]: [2020-05-12T15:04:01,042][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/logstash
May 12 15:04:01 inf8603.swe6.unibet.com logstash[23119]: [2020-05-12T15:04:01,397][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoi…oLite2-City.mmdb"}
May 12 15:04:01 inf8603.swe6.unibet.com logstash[23119]: [2020-05-12T15:04:01,418][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoi…oLite2-City.mmdb"}
May 12 15:04:01 inf8603.swe6.unibet.com logstash[23119]: [2020-05-12T15:04:01,658][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#
May 12 15:04:29 inf8603.swe6.unibet.com kibana[24240]: {"type":"log","@timestamp":"2020-05-12T14:04:29Z","tags":["reporting","warning"],"pid":24240,"message":"Generating a random key for xpack.reporting.encryptionKey. …ey in kibana.yml"} May 12 15:04:29 inf8603.swe6.unibet.com kibana[24240]: {"type":"log","@timestamp":"2020-05-12T14:04:29Z","tags":["status","plugin:reporting@6.8.8","info"],"pid":24240,"state":"green","message":"Status changed from unin…":"uninitialized"} May 12 15:04:29 inf8603.swe6.unibet.com kibana[24240]: {"type":"log","@timestamp":"2020-05-12T14:04:29Z","tags":["info","migrations"],"pid":24240,"message":"Creating index .kibana_1."} May 12 15:04:29 inf8603.swe6.unibet.com kibana[24240]: {"type":"log","@timestamp":"2020-05-12T14:04:29Z","tags":["info","migrations"],"pid":24240,"message":"Pointing alias .kibana to .kibana_1."} May 12 15:04:29 inf8603.swe6.unibet.com kibana[24240]: {"type":"log","@timestamp":"2020-05-12T14:04:29Z","tags":["info","migrations"],"pid":24240,"message":"Finished in 105ms."} May 12 15:04:29 inf8603.swe6.unibet.com kibana[24240]: {"type":"log","@timestamp":"2020-05-12T14:04:29Z","tags":["listening","info"],"pid":24240,"message":"Server running at http://localhost:5601"} May 12 15:04:29 inf8603.swe6.unibet.com kibana[24240]: {"type":"log","@timestamp":"2020-05-12T14:04:29Z","tags":["status","plugin:spaces@6.8.8","info"],"pid":24240,"state":"green","message":"Status changed from yellow …or Elasticsearch"} May 12 15:09:06 inf8603.swe6.unibet.com systemd[1]: kibana.service: Current command vanished from the unit file, execution of the command list won't be resumed. May 12 15:10:12 inf8603.swe6.unibet.com kibana[24240]: {"type":"response","@timestamp":"2020-05-12T14:10:12Z","tags":[],"pid":24240,"method":"post","statusCode":200,"req":{"url":"/api/spaces/space","method":"post","headers":{"accept-enc… May 12 15:10:14 inf8603.swe6.unibet.com kibana[24240]: {"type":"response","@timestamp":"2020-05-12T14:10:12Z","tags":[],"pid":24240,"method":"post","statusCode":200,"req":{"url":"/api/kibana/settings/defaultIndex","method":"post","heade… Hint: Some lines were ellipsized, use -l to show in full. ● evebox.service - EveBox Server Loaded: loaded (/lib/systemd/system/evebox.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2020-04-29 09:31:50 BST; 1 weeks 6 days ago Main PID: 10843 (evebox) Tasks: 12 (limit: 4915) Memory: 0B CGroup: /system.slice/evebox.service └─10843 /usr/bin/evebox server
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable. ● molochviewer-selks.service - Moloch Viewer Loaded: loaded (/etc/systemd/system/molochviewer-selks.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2020-05-12 15:03:37 BST; 30min ago Main PID: 23261 (sh) Tasks: 12 (limit: 4915) Memory: 35.4M CGroup: /system.slice/molochviewer-selks.service ├─23261 /bin/sh -c /data/moloch/bin/node viewer.js -c /data/moloch/etc/config.ini >> /data/moloch/logs/viewer.log 2>&1 └─23262 /data/moloch/bin/node viewer.js -c /data/moloch/etc/config.ini
May 12 15:03:37 inf8603.swe6.unibet.com systemd[1]: molochviewer-selks.service: Service RestartSec=1min 30s expired, scheduling restart. May 12 15:03:37 inf8603.swe6.unibet.com systemd[1]: molochviewer-selks.service: Scheduled restart job, restart counter is at 4. May 12 15:03:37 inf8603.swe6.unibet.com systemd[1]: Stopped Moloch Viewer. May 12 15:03:37 inf8603.swe6.unibet.com systemd[1]: Started Moloch Viewer. May 12 15:09:06 inf8603.swe6.unibet.com systemd[1]: molochviewer-selks.service: Current command vanished from the unit file, execution of the command list won't be resumed. ● molochpcapread-selks.service - Moloch Pcap Read Loaded: loaded (/etc/systemd/system/molochpcapread-selks.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Tue 2020-05-12 15:15:10 BST; 19min ago Process: 25005 ExecStart=/bin/sh -c /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m --copy --delete -R /data/nsm/ >> /data/moloch/logs/capture.log 2>&1 (code=exited, status=1/FAILURE) Main PID: 25005 (code=exited, status=1/FAILURE)
May 12 15:15:10 inf8603.swe6.unibet.com systemd[1]: molochpcapread-selks.service: Service RestartSec=1min 30s expired, scheduling restart. May 12 15:15:10 inf8603.swe6.unibet.com systemd[1]: molochpcapread-selks.service: Scheduled restart job, restart counter is at 4. May 12 15:15:10 inf8603.swe6.unibet.com systemd[1]: Stopped Moloch Pcap Read. May 12 15:15:10 inf8603.swe6.unibet.com systemd[1]: molochpcapread-selks.service: Start request repeated too quickly. May 12 15:15:10 inf8603.swe6.unibet.com systemd[1]: molochpcapread-selks.service: Failed with result 'exit-code'. May 12 15:15:10 inf8603.swe6.unibet.com systemd[1]: Failed to start Moloch Pcap Read. scirius RUNNING pid 24207, uptime 0:29:58 ii elasticsearch 6.8.8 all Elasticsearch is a distributed RESTful search engine built for the cloud. Reference documentation can be found at https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html and the 'Elasticsearch: The Definitive Guide' book can be found at https://www.elastic.co/guide/en/elasticsearch/guide/current/index.html ii elasticsearch-curator 5.8.1 amd64 Have indices in Elasticsearch? This is the tool for you!\n\nLike a museum curator manages the exhibits and collections on display, \nElasticsearch Curator helps you curate, or manage your indices. ii evebox 1:0.11.1 amd64 no description given ii kibana 6.8.8 amd64 Explore and visualize your Elasticsearch data ii kibana-dashboards-stamus 2019081202 amd64 Kibana 6 dashboard templates. ii logstash 1:6.8.8-1 all An extensible logging pipeline hi moloch 2.1.2-1 amd64 Moloch Full Packet System ii scirius 3.4.0-5 amd64 Django application to manage Suricata ruleset ii suricata 1:2020033001-0stamus0 amd64 Suricata open source multi-thread IDS/IPS/NSM system. Filesystem Type Size Used Avail Use% Mounted on udev devtmpfs 32G 0 32G 0% /dev tmpfs tmpfs 6.3G 677M 5.7G 11% /run /dev/sdc1 ext4 54G 30G 21G 59% / tmpfs tmpfs 32G 0 32G 0% /dev/shm tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs tmpfs 32G 0 32G 0% /sys/fs/cgroup tmpfs tmpfs 6.3G 0 6.3G 0% /run/user/1000
pevma
commented
4 years ago What are the last 20-30 lines of
/data/moloch/logs/capture.log ?
firion4ik
commented
4 years ago these are the only lines in /data/moloch/logs/capture.log
pevma
commented
4 years ago Sorry - what lines are those?
firion4ik
commented
4 years ago These ones, I put them in my comment above, just before the health check logs
cat /data/moloch/logs/capture.log May 12 15:09:06 http.c:304 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/_template/sessions2_template?filter_path=**._meta&include_type_name=true 0/80 0ms 195ms May 12 15:09:06 http.c:304 moloch_http_send_sync(): 1/1 SYNC 404 http://localhost:9200/sequence/sequence/fn-inf8603 0/76 0ms 1ms May 12 15:09:07 http.c:304 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/files/file/_search?size=1&sort=num:desc&q=node:inf8603 0/137 0ms 1078ms May 12 15:09:07 http.c:304 moloch_http_send_sync(): 1/1 SYNC 404 http://localhost:9200/stats/stat/inf8603 0/66 0ms 1ms May 12 15:09:07 http.c:304 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/fields/field/_search?size=3000 0/13200 0ms 5ms May 12 15:09:07 config.c:819 moloch_config_monitor_file(): Couldn't stat country file file /data/moloch/etc/GeoLite2-Country.mmdb error No such file or directory May 12 15:10:38 http.c:304 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/_template/sessions2_template?filter_path=**._meta&include_type_name=true 0/80 0ms 2ms May 12 15:10:38 http.c:304 moloch_http_send_sync(): 1/1 SYNC 404 http://localhost:9200/sequence/sequence/fn-inf8603 0/76 0ms 0ms May 12 15:10:38 http.c:304 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/files/file/_search?size=1&sort=num:desc&q=node:inf8603 0/134 0ms 1ms May 12 15:10:38 http.c:304 moloch_http_send_sync(): 1/1 SYNC 404 http://localhost:9200/stats/stat/inf8603 0/66 0ms 0ms May 12 15:10:38 http.c:304 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/fields/field/_search?size=3000 0/13200 0ms 1ms May 12 15:10:38 config.c:819 moloch_config_monitor_file(): Couldn't stat country file file /data/moloch/etc/GeoLite2-Country.mmdb error No such file or directory May 12 15:12:09 http.c:304 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/_template/sessions2_template?filter_path=**._meta&include_type_name=true 0/80 0ms 689ms May 12 15:12:09 http.c:304 moloch_http_send_sync(): 1/1 SYNC 404 http://localhost:9200/sequence/sequence/fn-inf8603 0/76 0ms 0ms May 12 15:12:09 http.c:304 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/files/file/_search?size=1&sort=num:desc&q=node:inf8603 0/134 0ms 1ms May 12 15:12:09 http.c:304 moloch_http_send_sync(): 1/1 SYNC 404 http://localhost:9200/stats/stat/inf8603 0/66 0ms 0ms May 12 15:12:09 http.c:304 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/fields/field/_search?size=3000 0/13200 0ms 1ms May 12 15:12:09 config.c:819 moloch_config_monitor_file(): Couldn't stat country file file /data/moloch/etc/GeoLite2-Country.mmdb error No such file or directory May 12 15:13:39 http.c:304 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/_template/sessions2_template?filter_path=**._meta&include_type_name=true 0/80 0ms 802ms May 12 15:13:39 http.c:304 moloch_http_send_sync(): 1/1 SYNC 404 http://localhost:9200/sequence/sequence/fn-inf8603 0/76 0ms 1ms May 12 15:13:39 http.c:304 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/files/file/_search?size=1&sort=num:desc&q=node:inf8603 0/134 0ms 2ms May 12 15:13:39 http.c:304 moloch_http_send_sync(): 1/1 SYNC 404 http://localhost:9200/stats/stat/inf8603 0/66 0ms 7ms May 12 15:13:39 http.c:304 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/fields/field/_search?size=3000 0/13200 0ms 6ms May 12 15:13:39 config.c:819 moloch_config_monitor_file(): Couldn't stat country file file /data/moloch/etc/GeoLite2-Country.mmdb error No such file or directory
pevma
commented
4 years ago Can you rerun first time set up script but also make sure you have internet connectivity please?
(also apt unhold the moloch package before that - at the moment it says it is on hold hi)
firion4ik
commented
4 years ago The first time setup script exited with an error (see below). This could be due to lack of free space on / , as the molochpcapread-selks.service is responsible for pcap rotation and it was not running, so now the disk is full of pcaps, 0% space available on it.
/home/selks-user# df -h Filesystem Size Used Avail Use% Mounted on udev 32G 0 32G 0% /dev tmpfs 6.3G 677M 5.7G 11% /run /dev/sdc1 54G 51G 1.9M 100% / tmpfs 32G 0 32G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 32G 0 32G 0% /sys/fs/cgroup tmpfs 6.3G 0 6.3G 0% /run/user/1000
more /opt/selks/log/selks-first-time-setup_stamus.log
Download GEO files? (yes or no) [yes] Moloch - Downloading GEO files WARNING: timestamping does nothing in combination with -O. See the manual for details.
Username/Password Authentication Failed. WARNING: timestamping does nothing in combination with -O. See the manual for details.
Username/Password Authentication Failed. 2020-05-13 10:34:36 URL:https://raw.githubusercontent.com/wireshark/wireshark/master/manuf [1704042/1704042] -> "oui.txt" [1]
Moloch - Configured - Now continue with step 4 in /data/moloch/README.txt
/sbin/start elasticsearch # for upstart/Centos 6/Ubuntu 14.04
systemctl start elasticsearch.service # for systemd/Centos 7/Ubuntu 16.045) Initialize/Upgrade Elasticsearch Moloch configuration a) If this is the first install, or want to delete all data /data/moloch/db/db.pl http://ESHOST:9200 init b) If this is an update to moloch package /data/moloch/db/db.pl http://ESHOST:9200 upgrade 6) Add an admin user if a new install or after an init /data/moloch/bin/moloch_add_user.sh admin "Admin User" THEPASSWORD --admin 7) Start everything a) If using upstart (Centos 6 or sometimes Ubuntu 14.04): /sbin/start molochcapture /sbin/start molochviewer b) If using systemd (Centos 7 or Ubuntu 16.04 or sometimes Ubuntu 14.04) systemctl start molochcapture.service systemctl start molochviewer.service 8) Look at log files for errors /data/moloch/logs/viewer.log /data/moloch/logs/capture.log 9) Visit http://MOLOCHHOST:8005 with your favorite browser. user: admin password: THEPASSWORD from step #6
Any configuration changes can be made to /data/moloch/etc/config.ini See https://molo.ch/faq#moloch-is-not-working for issues
Additional information can be found at:
Would you like to setup a retention policy now? (y/n)
Please specify the maximum file size in Gigabytes. The disk should have room for at least 10 times the specified value. (default is 12)
Setting maxFileSizeG to 1 Gigabyte.
Please specify the maximum rotation time in minutes. (default is none)
Setting maxFileTimeM to 60 minutes.
Elastic search error { Error: [cluster_block_exception] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
at respond (/data/moloch/node_modules/elasticsearch/src/lib/transport.js:308:15)
at checkRespForFailure (/data/moloch/node_modules/elasticsearch/src/lib/transport.js:267:7)
at HttpConnector.
pevma
commented
4 years ago I think you may have a full disk
Please see below for more info:
https://discuss.elastic.co/t/forbidden-12-index-read-only-allow-delete-api/110282/2
firion4ik
commented
4 years ago I think you may have a full disk
Thanks pevma Yes, as I said, the disk is full, and before putting the Elasticsearch back from read-only mode I need to free up the space. The biggest files are indices under /var/lib/elasticsearch/nodes/0/ and /var/log/suricata/eve.json [6 GB]
Do you know which files/folders could be safely cleaned?
pevma
commented
4 years ago I think you should clean up older ES data not the eve.json. there is a script under /opt/ (in crontab) that cleans old indices ...you can maybe change the time form 14 to 3 days and clean up ?
firion4ik
commented
4 years ago /etc/crontab contains only 1 entry:
0 3 * * * root ( /data/moloch/db/db.pl http://127.0.0.1:9200 expire daily 14 )
is it not under /opt/ though...is it that one?
pevma
commented
4 years ago you should have
0 4 * * * root /opt/selks/delete-old-logs.sh
yes, this file exists on the file system, but it wasn't in crontab (may be due to unfinished first time setup...not sure)
I changed 14 days to 1 day and ran it, but got this error
Fatal Python error: initfsencoding: Unable to get the locale encoding Traceback (most recent call last): File "/opt/python/3.7.4/lib/python3.7/encodings/__init__.py", line 31, in <module> zipimport.ZipImportError: can't decompress data; zlib not available ./delete-old-logs.sh: line 18: 5427 Aborted (core dumped) /opt/elasticsearch-curator/curator_cli delete_indices --filter_list ' [ { "filtertype": "age", "source": "creation_date", "direction": "older", "unit": "days", "unit_count": 1 }, { "filtertype": "pattern", "kind": "prefix", "value": "logstash*" } ] '
pevma
commented
4 years ago I think you may have the old version of that , can you please check the file and make sure there is a line inside starting like so
/usr/bin/curator_cli aka we need curator_cli if not please edit it to look like /usr/bin/curator_cli and execute
The full script should be looking like so :
root@SELKS:~# cat /opt/selks/delete-old-logs.sh
#!/bin/bash
/usr/bin/curator_cli delete_indices --filter_list '
[
{
"filtertype": "age",
"source": "creation_date",
"direction": "older",
"unit": "days",
"unit_count": 14
},
{
"filtertype": "pattern",
"kind": "prefix",
"value": "logstash*"
}
]
'
I decided to go with SELKS v6 RC1 installation from scratch and it seems it is working now. Thank you for your help pevma.
Hello, I'm new to SELKS and just installed it. Did the first configuration script and upgrade script. But I can't figure out why molochpcapread-selks.service doesn't get started. I tried to unhold the package and run the upgrade script again, bu the moloch version didn't change. When I run the upgrade script, I'm getting a "jq error" (see below):
`root@inf8603:~# selks-upgrade_stamus NOTE: Depending on the size and how busy the system is the upgrade may take a while. Starting the upgrade sequence...
Hit:1 http://security.debian.org/debian-security stretch/updates InRelease Ign:2 http://ftp.se.debian.org/debian stretch InRelease Hit:3 http://ftp.se.debian.org/debian stretch-updates InRelease Hit:4 http://ftp.se.debian.org/debian stretch Release Hit:5 https://artifacts.elastic.co/packages/6.x/apt stable InRelease Hit:6 https://packages.elastic.co/curator/5/debian9 stable InRelease Hit:9 http://packages.stamus-networks.com/selks5/debian stretch InRelease Hit:7 http://evebox.org/files/debian stable InRelease Hit:10 http://packages.stamus-networks.com/selks5/debian-kernel stretch InRelease Reading package lists... Done Reading package lists... Done Building dependency tree Reading state information... Done selks-scripts-stamus is already the newest version (2019060301). 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. NOTE: Starting second stage upgrade sequence...
outputs.7.pcap-log.enabled = yes Hit:1 http://security.debian.org/debian-security stretch/updates InRelease Ign:2 http://ftp.se.debian.org/debian stretch InRelease Hit:3 http://ftp.se.debian.org/debian stretch-updates InRelease Hit:4 http://ftp.se.debian.org/debian stretch Release Hit:5 https://artifacts.elastic.co/packages/6.x/apt stable InRelease Hit:6 https://packages.elastic.co/curator/5/debian9 stable InRelease Hit:8 http://packages.stamus-networks.com/selks5/debian stretch InRelease Hit:10 http://packages.stamus-networks.com/selks5/debian-kernel stretch InRelease Hit:7 http://evebox.org/files/debian stable InRelease Reading package lists... Done Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. scirius: stopped scirius: started jq: error (at:1): Cannot index object with number
dpkg: error: --compare-versions takes three arguments:
Type dpkg --help for help about installing and deinstalling packages []; Use 'apt' or 'aptitude' for user-friendly package management; Type dpkg -Dhelp for a list of dpkg debug flag values; Type dpkg --force-help for a list of forcing options; Type dpkg-deb --help for help about manipulating .deb files;
Options marked [*] produce a lot of output - pipe it through 'less' or 'more' ! root@inf8603:~# `
This is the health check script output:
`● suricata.service - LSB: Next Generation IDS/IPS Loaded: loaded (/etc/init.d/suricata; generated; vendor preset: enabled) Active: active (running) since Wed 2020-04-29 14:07:58 BST; 31min ago Docs: man:systemd-sysv-generator(8) Process: 21639 ExecStop=/etc/init.d/suricata stop (code=exited, status=0/SUCCESS) Process: 21653 ExecStart=/etc/init.d/suricata start (code=exited, status=0/SUCCESS) Tasks: 30 (limit: 4915) CGroup: /system.slice/suricata.service └─21661 /usr/bin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid --af-packet -D -v --user=logstash
Apr 29 14:07:58 inf8603.swe6.unibet.com systemd[1]: Starting LSB: Next Generation IDS/IPS... Apr 29 14:07:58 inf8603.swe6.unibet.com suricata[21653]: Starting suricata in IDS (af-packet) mode... done. Apr 29 14:07:58 inf8603.swe6.unibet.com systemd[1]: Started LSB: Next Generation IDS/IPS. ● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2020-04-29 14:07:54 BST; 31min ago Docs: http://www.elastic.co Main PID: 21463 (java) Tasks: 116 (limit: 4915) CGroup: /system.slice/elasticsearch.service ├─21463 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.netwo…et └─21605 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
Apr 29 14:07:54 inf8603.swe6.unibet.com systemd[1]: Started Elasticsearch. Apr 29 14:07:54 inf8603.swe6.unibet.com elasticsearch[21463]: warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME ● logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2020-04-20 13:50:30 BST; 1 weeks 2 days ago Main PID: 566 (java) Tasks: 47 (limit: 4915) CGroup: /system.slice/logstash.service └─566 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-…sh
Apr 29 14:07:56 inf8603.swe6.unibet.com logstash[566]: [2020-04-29T14:07:56,936][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are…2} Apr 29 14:07:56 inf8603.swe6.unibet.com logstash[566]: [2020-04-29T14:07:56,973][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are…4} Apr 29 14:07:57 inf8603.swe6.unibet.com logstash[566]: [2020-04-29T14:07:57,464][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an e…"} Apr 29 14:07:58 inf8603.swe6.unibet.com logstash[566]: [2020-04-29T14:07:58,937][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are…4} Apr 29 14:08:00 inf8603.swe6.unibet.com logstash[566]: [2020-04-29T14:08:00,975][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are…8} Apr 29 14:08:02 inf8603.swe6.unibet.com logstash[566]: [2020-04-29T14:08:02,466][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an e…"} Apr 29 14:08:02 inf8603.swe6.unibet.com logstash[566]: [2020-04-29T14:08:02,939][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are…8} Apr 29 14:08:04 inf8603.swe6.unibet.com logstash[566]: [2020-04-29T14:08:04,370][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are…2} Apr 29 14:08:06 inf8603.swe6.unibet.com logstash[566]: [2020-04-29T14:08:06,371][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are…4} Apr 29 14:08:07 inf8603.swe6.unibet.com logstash[566]: [2020-04-29T14:08:07,480][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://127.0.0.1:9200/"} Hint: Some lines were ellipsized, use -l to show in full. ● kibana.service - Kibana Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2020-04-29 14:07:54 BST; 31min ago Main PID: 21470 (node) Tasks: 11 (limit: 4915) CGroup: /system.slice/kibana.service └─21470 /usr/share/kibana/bin/../node/bin/node --no-warnings --max-http-header-size=65536 /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml
Apr 29 14:08:08 inf8603.swe6.unibet.com kibana[21470]: {"type":"log","@timestamp":"2020-04-29T13:08:08Z","tags":["status","plugin:reporting@6.8.8","info"],"pid":21470,"state":"…connections"} Apr 29 14:08:08 inf8603.swe6.unibet.com kibana[21470]: {"type":"log","@timestamp":"2020-04-29T13:08:08Z","tags":["info","monitoring-ui","kibana-monitoring"],"pid":21470,"messag… collection"} Apr 29 14:08:08 inf8603.swe6.unibet.com kibana[21470]: {"type":"log","@timestamp":"2020-04-29T13:08:08Z","tags":["status","plugin:security@6.8.8","info"],"pid":21470,"state":"g…connections"} Apr 29 14:08:08 inf8603.swe6.unibet.com kibana[21470]: {"type":"log","@timestamp":"2020-04-29T13:08:08Z","tags":["status","plugin:maps@6.8.8","info"],"pid":21470,"state":"green…connections"} Apr 29 14:08:08 inf8603.swe6.unibet.com kibana[21470]: {"type":"log","@timestamp":"2020-04-29T13:08:08Z","tags":["license","info","xpack"],"pid":21470,"message":"Imported licen…tus: active"} Apr 29 14:08:08 inf8603.swe6.unibet.com kibana[21470]: {"type":"log","@timestamp":"2020-04-29T13:08:08Z","tags":["info","migrations"],"pid":21470,"message":"Migrating .kibana_2…o .kibana_3"} Apr 29 14:08:08 inf8603.swe6.unibet.com kibana[21470]: {"type":"log","@timestamp":"2020-04-29T13:08:08Z","tags":["info","migrations"],"pid":21470,"message":"Pointing alias .kib… .kibana_3."} Apr 29 14:08:08 inf8603.swe6.unibet.com kibana[21470]: {"type":"log","@timestamp":"2020-04-29T13:08:08Z","tags":["info","migrations"],"pid":21470,"message":"Finished in 860ms."} Apr 29 14:08:08 inf8603.swe6.unibet.com kibana[21470]: {"type":"log","@timestamp":"2020-04-29T13:08:08Z","tags":["listening","info"],"pid":21470,"message":"Server running at ht…alhost:5601"} Apr 29 14:08:08 inf8603.swe6.unibet.com kibana[21470]: {"type":"log","@timestamp":"2020-04-29T13:08:08Z","tags":["status","plugin:spaces@6.8.8","info"],"pid":21470,"state":"gre…connections"} Hint: Some lines were ellipsized, use -l to show in full. ● evebox.service - EveBox Server Loaded: loaded (/lib/systemd/system/evebox.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2020-04-29 09:31:50 BST; 5h 7min ago Main PID: 10843 (evebox) Tasks: 12 (limit: 4915) CGroup: /system.slice/evebox.service └─10843 /usr/bin/evebox server
Apr 29 09:31:50 inf8603 evebox[10843]: 2020-04-29 09:31:50 (server.go:308) -- Using ElasticSearch Index logstash.
Apr 29 09:31:50 inf8603 evebox[10843]: 2020-04-29 09:31:50 (elasticsearch.go:109) -- Event base index: logstash
Apr 29 09:31:50 inf8603 evebox[10843]: 2020-04-29 09:31:50 (elasticsearch.go:110) -- Event search index: logstash-*
Apr 29 09:31:50 inf8603 evebox[10843]: 2020-04-29 09:31:50 (server.go:338) -- Connected to Elastic Search (version: 6.7.0)
Apr 29 09:31:50 inf8603 evebox[10843]: 2020-04-29 09:31:50 (elasticsearch.go:177) -- Assuming Logstash style index
Apr 29 09:31:50 inf8603 evebox[10843]: 2020-04-29 09:31:50 (server.go:131) -- Session reaper started
Apr 29 09:31:50 inf8603 evebox[10843]: 2020-04-29 09:31:50 (server.go:165) -- Authentication disabled.
Apr 29 09:31:50 inf8603 evebox[10843]: 2020-04-29 09:31:50 (server.go:261) -- Listening on [127.0.0.1]:5636
Apr 29 11:46:38 inf8603.swe6.unibet.com evebox[10843]: 2020-04-29 11:46:38 (anonymous.go:64) -- Logging in anonymous user {selks-user} from 127.0.0.1:49210
Apr 29 12:46:50 inf8603.swe6.unibet.com evebox[10843]: 2020-04-29 12:46:50 (sessionstore.go:64) -- Expiring session -- username=selks-user addr=127.0.0.1:49210
● molochviewer-selks.service - Moloch Viewer
Loaded: loaded (/etc/systemd/system/molochviewer-selks.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2020-04-29 14:09:26 BST; 29min ago
Main PID: 21853 (sh)
Tasks: 11 (limit: 4915)
CGroup: /system.slice/molochviewer-selks.service
├─21853 /bin/sh -c /data/moloch/bin/node viewer.js -c /data/moloch/etc/config.ini >> /data/moloch/logs/viewer.log 2>&1
└─21854 /data/moloch/bin/node viewer.js -c /data/moloch/etc/config.ini
Apr 29 14:09:26 inf8603.swe6.unibet.com systemd[1]: Started Moloch Viewer. ● molochpcapread-selks.service - Moloch Pcap Read Loaded: loaded (/etc/systemd/system/molochpcapread-selks.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Wed 2020-04-29 14:13:56 BST; 25min ago Process: 21880 ExecStart=/bin/sh -c /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m --copy --delete -R /data/nsm/ >> /data/moloch/logs/capture.log 2>&1 (code=exited, status=1/FAILURE) Main PID: 21880 (code=exited, status=1/FAILURE)
Apr 29 14:12:26 inf8603.swe6.unibet.com systemd[1]: molochpcapread-selks.service: Failed with result 'exit-code'. Apr 29 14:13:56 inf8603.swe6.unibet.com systemd[1]: molochpcapread-selks.service: Service hold-off time over, scheduling restart. Apr 29 14:13:56 inf8603.swe6.unibet.com systemd[1]: Stopped Moloch Pcap Read. Apr 29 14:13:56 inf8603.swe6.unibet.com systemd[1]: molochpcapread-selks.service: Start request repeated too quickly. Apr 29 14:13:56 inf8603.swe6.unibet.com systemd[1]: Failed to start Moloch Pcap Read. Apr 29 14:13:56 inf8603.swe6.unibet.com systemd[1]: molochpcapread-selks.service: Unit entered failed state. Apr 29 14:13:56 inf8603.swe6.unibet.com systemd[1]: molochpcapread-selks.service: Failed with result 'exit-code'. scirius RUNNING pid 21548, uptime 0:31:30 ii elasticsearch 6.8.8 all Elasticsearch is a distributed RESTful search engine built for the cloud. Reference documentation can be found at https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html and the 'Elasticsearch: The Definitive Guide' book can be found at https://www.elastic.co/guide/en/elasticsearch/guide/current/index.html ii elasticsearch-curator 5.8.1 amd64 Have indices in Elasticsearch? This is the tool for you!\n\nLike a museum curator manages the exhibits and collections on display, \nElasticsearch Curator helps you curate, or manage your indices. ii evebox 1:0.11.1 amd64 no description given ii kibana 6.8.8 amd64 Explore and visualize your Elasticsearch data ii kibana-dashboards-stamus 2019030501 amd64 Kibana 6 dashboard templates. ii logstash 1:6.8.8-1 all An extensible logging pipeline ii moloch 1.7.1-1 amd64 Moloch Full Packet System ii scirius 3.2.0-1 amd64 Django application to manage Suricata ruleset ii suricata 1:2019121401-0stamus1 amd64 Suricata open source multi-thread IDS/IPS/NSM system. Filesystem Type Size Used Avail Use% Mounted on udev devtmpfs 32G 0 32G 0% /dev tmpfs tmpfs 6.3G 17M 6.3G 1% /run /dev/sdc1 ext4 54G 4.8G 46G 10% / tmpfs tmpfs 32G 0 32G 0% /dev/shm tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs tmpfs 32G 0 32G 0% /sys/fs/cgroup tmpfs tmpfs 6.3G 0 6.3G 0% /run/user/1000 `
And here is the /data/moloch/logs/capture.log
Apr 29 11:51:11 config.c:816 moloch_config_monitor_file(): Couldn't stat country file file /data/moloch/etc/GeoLite2-Country.mmdb error No such file or directory Apr 29 11:52:41 http.c:283 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/_template/sessions2_template?filter_path=**._meta 0/80 1ms 2ms Apr 29 11:52:41 http.c:283 moloch_http_send_sync(): 1/1 SYNC 404 http://localhost:9200/sequence/sequence/fn-inf8603 0/76 0ms 1ms Apr 29 11:52:41 http.c:283 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/files/file/_search?size=1&sort=num:desc&q=node:inf8603 0/134 0ms 2ms Apr 29 11:52:41 http.c:283 moloch_http_send_sync(): 1/1 SYNC 404 http://localhost:9200/stats/stat/inf8603 0/66 0ms 1ms Apr 29 11:52:41 http.c:283 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/fields/field/_search?size=3000 0/13191 0ms 3ms Apr 29 11:52:41 config.c:816 moloch_config_monitor_file(): Couldn't stat country file file /data/moloch/etc/GeoLite2-Country.mmdb error No such file or directory Apr 29 11:54:11 http.c:283 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/_template/sessions2_template?filter_path=**._meta 0/80 1ms 2ms Apr 29 11:54:11 http.c:283 moloch_http_send_sync(): 1/1 SYNC 404 http://localhost:9200/sequence/sequence/fn-inf8603 0/76 0ms 1ms Apr 29 11:54:11 http.c:283 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/files/file/_search?size=1&sort=num:desc&q=node:inf8603 0/134 0ms 2ms Apr 29 11:54:11 http.c:283 moloch_http_send_sync(): 1/1 SYNC 404 http://localhost:9200/stats/stat/inf8603 0/66 0ms 1ms Apr 29 11:54:11 http.c:283 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/fields/field/_search?size=3000 0/13191 0ms 2ms Apr 29 11:54:11 config.c:816 moloch_config_monitor_file(): Couldn't stat country file file /data/moloch/etc/GeoLite2-Country.mmdb error No such file or directory Apr 29 14:07:56 http.c:237 moloch_http_send_sync(): Retry http://localhost:9200/_template/sessions2_template?filter_path=**._meta error 'Couldn't connect to server' Apr 29 14:07:56 http.c:237 moloch_http_send_sync(): Retry http://localhost:9200/_template/sessions2_template?filter_path=**._meta error 'Couldn't connect to server' Apr 29 14:07:56 http.c:237 moloch_http_send_sync(): Retry http://localhost:9200/_template/sessions2_template?filter_path=**._meta error 'Couldn't connect to server' Apr 29 14:07:56 http.c:237 moloch_http_send_sync(): Retry http://localhost:9200/_template/sessions2_template?filter_path=**._meta error 'Couldn't connect to server' Apr 29 14:07:56 http.c:241 moloch_http_send_sync(): libcurl failure http://localhost:9200/_template/sessions2_template?filter_path=**._meta error 'Couldn't connect to server' Apr 29 14:07:56 db.c:1808 moloch_db_check(): ERROR - Couldn't load version information, database might be down or out of date. Run "db/db.pl host:port upgrade" Apr 29 14:09:26 http.c:283 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/_template/sessions2_template?filter_path=**._meta 0/80 1ms 8ms Apr 29 14:09:26 http.c:283 moloch_http_send_sync(): 1/1 SYNC 404 http://localhost:9200/sequence/sequence/fn-inf8603 0/76 0ms 2ms Apr 29 14:09:26 http.c:283 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/files/file/_search?size=1&sort=num:desc&q=node:inf8603 0/135 0ms 29ms Apr 29 14:09:26 http.c:283 moloch_http_send_sync(): 1/1 SYNC 404 http://localhost:9200/stats/stat/inf8603 0/66 0ms 2ms Apr 29 14:09:26 http.c:283 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/fields/field/_search?size=3000 0/13191 0ms 5ms Apr 29 14:09:26 config.c:816 moloch_config_monitor_file(): Couldn't stat country file file /data/moloch/etc/GeoLite2-Country.mmdb error No such file or directory Apr 29 14:10:56 http.c:283 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/_template/sessions2_template?filter_path=**._meta 0/80 1ms 5ms Apr 29 14:10:56 http.c:283 moloch_http_send_sync(): 1/1 SYNC 404 http://localhost:9200/sequence/sequence/fn-inf8603 0/76 0ms 2ms Apr 29 14:10:56 http.c:283 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/files/file/_search?size=1&sort=num:desc&q=node:inf8603 0/134 0ms 3ms Apr 29 14:10:56 http.c:283 moloch_http_send_sync(): 1/1 SYNC 404 http://localhost:9200/stats/stat/inf8603 0/66 0ms 1ms Apr 29 14:10:56 http.c:283 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/fields/field/_search?size=3000 0/13191 0ms 4ms Apr 29 14:10:56 config.c:816 moloch_config_monitor_file(): Couldn't stat country file file /data/moloch/etc/GeoLite2-Country.mmdb error No such file or directory Apr 29 14:12:26 http.c:283 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/_template/sessions2_template?filter_path=**._meta 0/80 1ms 4ms Apr 29 14:12:26 http.c:283 moloch_http_send_sync(): 1/1 SYNC 404 http://localhost:9200/sequence/sequence/fn-inf8603 0/76 0ms 1ms Apr 29 14:12:26 http.c:283 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/files/file/_search?size=1&sort=num:desc&q=node:inf8603 0/134 0ms 3ms Apr 29 14:12:26 http.c:283 moloch_http_send_sync(): 1/1 SYNC 404 http://localhost:9200/stats/stat/inf8603 0/66 0ms 1ms Apr 29 14:12:26 http.c:283 moloch_http_send_sync(): 1/1 SYNC 200 http://localhost:9200/fields/field/_search?size=3000 0/13191 0ms 4ms Apr 29 14:12:26 config.c:816 moloch_config_monitor_file(): Couldn't stat country file file /data/moloch/etc/GeoLite2-Country.mmdb error No such file or directoryAny ideas are appreciated! Thanks in advance!