Can't Upgrade from SELKS 5 to SELKS 6 -- Elasticsearch and Moloch errors

[P-Sandusky]

I have a SELKS installation here that I've successfully upgraded to SELKS 5, but now it's not making the transition to SELKS 6. Suricata and Scirius seem to have made the change, but Logstash hangs when I run the upgrade scripts, and Elasticsearch fails outright. Once Elastic fails, Moloch doesn't upgrade.

I've tried killing Logstash's processes directly, and that got the install script to run. Elastic promptly quit. At first, I noticed a number of what appeared to be Python dependencies missing, so I ran the dependency install script. At the most recent attempt, Logstash would not halt, so I did not see if installing the dependencies had an impact.

Here is my most recent health output:

selks-user@SELKS:~$ sudo selks-health-check_stamus ● suricata.service - LSB: Next Generation IDS/IPS Loaded: loaded (/etc/init.d/suricata; generated) Active: active (running) since Tue 2021-02-23 14:08:21 EST; 35min ago Docs: man:systemd-sysv-generator(8) Process: 846 ExecStart=/etc/init.d/suricata start (code=exited, status=0/SUCCESS) Tasks: 14 (limit: 4915) Memory: 1.4G CGroup: /system.slice/suricata.service └─909 /usr/bin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid --af-packet -D -v --user=logstash

Feb 23 14:08:21 SELKS systemd[1]: Starting LSB: Next Generation IDS/IPS... Feb 23 14:08:21 SELKS suricata[846]: Starting suricata in IDS (af-packet) mode... done. Feb 23 14:08:21 SELKS systemd[1]: Started LSB: Next Generation IDS/IPS. ● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Tue 2021-02-23 14:22:54 EST; 20min ago Docs: https://www.elastic.co Process: 3897 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE) Main PID: 3897 (code=exited, status=1/FAILURE)

Feb 23 14:22:54 SELKS systemd-entrypoint[3897]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) Feb 23 14:22:54 SELKS systemd-entrypoint[3897]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) Feb 23 14:22:54 SELKS systemd-entrypoint[3897]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) Feb 23 14:22:54 SELKS systemd-entrypoint[3897]: at org.elasticsearch.cli.Command.main(Command.java:79) Feb 23 14:22:54 SELKS systemd-entrypoint[3897]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) Feb 23 14:22:54 SELKS systemd-entrypoint[3897]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) Feb 23 14:22:54 SELKS systemd-entrypoint[3897]: For complete error details, refer to the log at /var/log/elasticsearch/elasticsearch.log Feb 23 14:22:54 SELKS systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE Feb 23 14:22:54 SELKS systemd[1]: elasticsearch.service: Failed with result 'exit-code'. Feb 23 14:22:54 SELKS systemd[1]: Failed to start Elasticsearch. ● logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled) Active: deactivating (stop-sigterm) since Tue 2021-02-23 14:12:36 EST; 31min ago Main PID: 497 (java) Tasks: 55 (limit: 4915) Memory: 985.1M CGroup: /system.slice/logstash.service └─497 /usr/share/logstash/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyO…

Feb 23 14:43:23 SELKS logstash[497]: [2021-02-23T14:43:23,532][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info… Feb 23 14:43:26 SELKS logstash[497]: [2021-02-23T14:43:26,752][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES inst… Feb 23 14:43:26 SELKS logstash[497]: [2021-02-23T14:43:26,923][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES inst… Feb 23 14:43:28 SELKS logstash[497]: [2021-02-23T14:43:28,605][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info… Feb 23 14:43:31 SELKS logstash[497]: [2021-02-23T14:43:31,759][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES inst… Feb 23 14:43:31 SELKS logstash[497]: [2021-02-23T14:43:31,932][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES inst… Feb 23 14:43:33 SELKS logstash[497]: [2021-02-23T14:43:33,698][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info… Feb 23 14:43:36 SELKS logstash[497]: [2021-02-23T14:43:36,766][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES inst… Feb 23 14:43:36 SELKS logstash[497]: [2021-02-23T14:43:36,942][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES inst… Feb 23 14:43:38 SELKS logstash[497]: [2021-02-23T14:43:38,777][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info… Hint: Some lines were ellipsized, use -l to show in full. ● kibana.service - Kibana Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2021-02-23 14:22:54 EST; 20min ago Docs: https://www.elastic.co Main PID: 4121 (node) Tasks: 11 (limit: 4915) Memory: 159.4M CGroup: /system.slice/kibana.service └─4121 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist --logging.dest=/var/log/kibana/kibana.log --pid.file=/run/kiba…

Feb 23 14:22:54 SELKS systemd[1]: Started Kibana. ● evebox.service - EveBox Server Loaded: loaded (/lib/systemd/system/evebox.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2021-02-23 14:08:20 EST; 35min ago Main PID: 491 (evebox) Tasks: 9 (limit: 4915) Memory: 19.2M CGroup: /system.slice/evebox.service └─491 /usr/bin/evebox server

Feb 23 14:08:20 SELKS systemd[1]: Started EveBox Server. Feb 23 14:08:20 SELKS evebox[491]: 2021-02-23 14:08:20 INFO evebox::version: This is EveBox version 0.12.0 (rev: ba9d586); x86_64-unknown-linux-musl Feb 23 14:08:20 SELKS evebox[491]: 2021-02-23 14:08:20 INFO evebox::server::main: Using temporary in-memory configuration database Feb 23 14:08:20 SELKS evebox[491]: 2021-02-23 14:08:20 INFO evebox::sqlite::configrepo: Initializing SQLite database Feb 23 14:08:20 SELKS evebox[491]: 2021-02-23 14:08:20 INFO evebox::sqlite::configrepo: Updating SQLite database to schema version 1 Feb 23 14:08:20 SELKS evebox[491]: 2021-02-23 14:08:20 ERROR evebox::server::main: Failed to get Elasticsearch version, things may not work righ…s error 111) Feb 23 14:08:20 SELKS evebox[491]: 2021-02-23 14:08:20 INFO evebox::server::main: Starting server on 127.0.0.1:5636, tls=false Hint: Some lines were ellipsized, use -l to show in full. ● molochviewer-selks.service - Moloch Viewer Loaded: loaded (/etc/systemd/system/molochviewer-selks.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Tue 2021-02-23 14:14:39 EST; 29min ago Process: 2085 ExecStart=/bin/sh -c /data/moloch/bin/node viewer.js -c /data/moloch/etc/config.ini >> /data/moloch/logs/viewer.log 2>&1 (code=exited, status=1/FAILURE) Main PID: 2085 (code=exited, status=1/FAILURE)

Feb 23 14:14:39 SELKS systemd[1]: molochviewer-selks.service: Service RestartSec=1min 30s expired, scheduling restart. Feb 23 14:14:39 SELKS systemd[1]: molochviewer-selks.service: Scheduled restart job, restart counter is at 4. Feb 23 14:14:39 SELKS systemd[1]: Stopped Moloch Viewer. Feb 23 14:14:39 SELKS systemd[1]: molochviewer-selks.service: Start request repeated too quickly. Feb 23 14:14:39 SELKS systemd[1]: molochviewer-selks.service: Failed with result 'exit-code'. Feb 23 14:14:39 SELKS systemd[1]: Failed to start Moloch Viewer. ● molochpcapread-selks.service - Moloch Pcap Read Loaded: loaded (/etc/systemd/system/molochpcapread-selks.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Tue 2021-02-23 14:14:21 EST; 29min ago Process: 2082 ExecStart=/bin/sh -c /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m --copy --delete -R /data/nsm/ >> /data/moloch/logs/capture.log 2>&1 (code=exited, status=1/FAILURE) Main PID: 2082 (code=exited, status=1/FAILURE)

Feb 23 14:14:21 SELKS systemd[1]: molochpcapread-selks.service: Service RestartSec=1min 30s expired, scheduling restart. Feb 23 14:14:21 SELKS systemd[1]: molochpcapread-selks.service: Scheduled restart job, restart counter is at 4. Feb 23 14:14:21 SELKS systemd[1]: Stopped Moloch Pcap Read. Feb 23 14:14:21 SELKS systemd[1]: molochpcapread-selks.service: Start request repeated too quickly. Feb 23 14:14:21 SELKS systemd[1]: molochpcapread-selks.service: Failed with result 'exit-code'. Feb 23 14:14:21 SELKS systemd[1]: Failed to start Moloch Pcap Read. scirius RUNNING pid 4135, uptime 0:20:46 ii elasticsearch 7.11.1 amd64 Distributed RESTful search engine built for the cloud ii elasticsearch-curator 5.8.3 amd64 Have indices in Elasticsearch? This is the tool for you!\n\nLike a museum curator manages the exhibits and collections on display, \nElasticsearch Curator helps you curate, or manage your indices. ii evebox 1:0.12.0 amd64 no description given ii kibana 7.11.1 amd64 Explore and visualize your Elasticsearch data ii kibana-dashboards-stamus 2020122001 amd64 Kibana 6 dashboard templates. ii logstash 1:7.11.1-1 amd64 An extensible logging pipeline hi moloch 2.2.3-1 amd64 Moloch Full Packet System ii scirius 3.5.0-3 amd64 Django application to manage Suricata ruleset ii suricata 1:2021012201-0stamus0 amd64 Suricata open source multi-thread IDS/IPS/NSM system. Filesystem Type Size Used Avail Use% Mounted on udev devtmpfs 3.9G 0 3.9G 0% /dev tmpfs tmpfs 798M 8.7M 790M 2% /run /dev/sda1 ext4 484G 12G 448G 3% / tmpfs tmpfs 3.9G 0 3.9G 0% /dev/shm tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup tmpfs tmpfs 798M 0 798M 0% /run/user/1000 selks-user@SELKS:~$

[pevma]

Did you use the upgrade script ?

-- Regards, Peter Manev

On 23 Feb 2021, at 20:45, P-Sandusky notifications@github.com wrote:

 I have a SELKS installation here that I've successfully upgraded to SELKS 5, but now it's not making the transition to SELKS 6. Suricata and Scirius seem to have made the change, but Logstash hangs when I run the upgrade scripts, and Elasticsearch fails outright. Once Elastic fails, Moloch doesn't upgrade.

I've tried killing Logstash's processes directly, and that got the install script to run. Elastic promptly quit. At first, I noticed a number of what appeared to be Python dependencies missing, so I ran the dependency install script. At the most recent attempt, Logstash would not halt, so I did not see if installing the dependencies had an impact.

Here is my most recent health output:

selks-user@SELKS:~$ sudo selks-health-check_stamus ● suricata.service - LSB: Next Generation IDS/IPS Loaded: loaded (/etc/init.d/suricata; generated) Active: active (running) since Tue 2021-02-23 14:08:21 EST; 35min ago Docs: man:systemd-sysv-generator(8) Process: 846 ExecStart=/etc/init.d/suricata start (code=exited, status=0/SUCCESS) Tasks: 14 (limit: 4915) Memory: 1.4G CGroup: /system.slice/suricata.service └─909 /usr/bin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid --af-packet -D -v --user=logstash

Feb 23 14:08:21 SELKS systemd[1]: Starting LSB: Next Generation IDS/IPS... Feb 23 14:08:21 SELKS suricata[846]: Starting suricata in IDS (af-packet) mode... done. Feb 23 14:08:21 SELKS systemd[1]: Started LSB: Next Generation IDS/IPS. ● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Tue 2021-02-23 14:22:54 EST; 20min ago Docs: https://www.elastic.co Process: 3897 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE) Main PID: 3897 (code=exited, status=1/FAILURE)

Feb 23 14:22:54 SELKS systemd-entrypoint[3897]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) Feb 23 14:22:54 SELKS systemd-entrypoint[3897]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) Feb 23 14:22:54 SELKS systemd-entrypoint[3897]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) Feb 23 14:22:54 SELKS systemd-entrypoint[3897]: at org.elasticsearch.cli.Command.main(Command.java:79) Feb 23 14:22:54 SELKS systemd-entrypoint[3897]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) Feb 23 14:22:54 SELKS systemd-entrypoint[3897]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) Feb 23 14:22:54 SELKS systemd-entrypoint[3897]: For complete error details, refer to the log at /var/log/elasticsearch/elasticsearch.log Feb 23 14:22:54 SELKS systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE Feb 23 14:22:54 SELKS systemd[1]: elasticsearch.service: Failed with result 'exit-code'. Feb 23 14:22:54 SELKS systemd[1]: Failed to start Elasticsearch. ● logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled) Active: deactivating (stop-sigterm) since Tue 2021-02-23 14:12:36 EST; 31min ago Main PID: 497 (java) Tasks: 55 (limit: 4915) Memory: 985.1M CGroup: /system.slice/logstash.service └─497 /usr/share/logstash/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyO…

Feb 23 14:43:23 SELKS logstash[497]: [2021-02-23T14:43:23,532][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info… Feb 23 14:43:26 SELKS logstash[497]: [2021-02-23T14:43:26,752][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES inst… Feb 23 14:43:26 SELKS logstash[497]: [2021-02-23T14:43:26,923][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES inst… Feb 23 14:43:28 SELKS logstash[497]: [2021-02-23T14:43:28,605][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info… Feb 23 14:43:31 SELKS logstash[497]: [2021-02-23T14:43:31,759][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES inst… Feb 23 14:43:31 SELKS logstash[497]: [2021-02-23T14:43:31,932][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES inst… Feb 23 14:43:33 SELKS logstash[497]: [2021-02-23T14:43:33,698][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info… Feb 23 14:43:36 SELKS logstash[497]: [2021-02-23T14:43:36,766][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES inst… Feb 23 14:43:36 SELKS logstash[497]: [2021-02-23T14:43:36,942][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES inst… Feb 23 14:43:38 SELKS logstash[497]: [2021-02-23T14:43:38,777][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info… Hint: Some lines were ellipsized, use -l to show in full. ● kibana.service - Kibana Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2021-02-23 14:22:54 EST; 20min ago Docs: https://www.elastic.co Main PID: 4121 (node) Tasks: 11 (limit: 4915) Memory: 159.4M CGroup: /system.slice/kibana.service └─4121 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist --logging.dest=/var/log/kibana/kibana.log --pid.file=/run/kiba…

Feb 23 14:22:54 SELKS systemd[1]: Started Kibana. ● evebox.service - EveBox Server Loaded: loaded (/lib/systemd/system/evebox.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2021-02-23 14:08:20 EST; 35min ago Main PID: 491 (evebox) Tasks: 9 (limit: 4915) Memory: 19.2M CGroup: /system.slice/evebox.service └─491 /usr/bin/evebox server

Feb 23 14:08:20 SELKS systemd[1]: Started EveBox Server. Feb 23 14:08:20 SELKS evebox[491]: 2021-02-23 14:08:20 INFO evebox::version: This is EveBox version 0.12.0 (rev: ba9d586); x86_64-unknown-linux-musl Feb 23 14:08:20 SELKS evebox[491]: 2021-02-23 14:08:20 INFO evebox::server::main: Using temporary in-memory configuration database Feb 23 14:08:20 SELKS evebox[491]: 2021-02-23 14:08:20 INFO evebox::sqlite::configrepo: Initializing SQLite database Feb 23 14:08:20 SELKS evebox[491]: 2021-02-23 14:08:20 INFO evebox::sqlite::configrepo: Updating SQLite database to schema version 1 Feb 23 14:08:20 SELKS evebox[491]: 2021-02-23 14:08:20 ERROR evebox::server::main: Failed to get Elasticsearch version, things may not work righ…s error 111) Feb 23 14:08:20 SELKS evebox[491]: 2021-02-23 14:08:20 INFO evebox::server::main: Starting server on 127.0.0.1:5636, tls=false Hint: Some lines were ellipsized, use -l to show in full. ● molochviewer-selks.service - Moloch Viewer Loaded: loaded (/etc/systemd/system/molochviewer-selks.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Tue 2021-02-23 14:14:39 EST; 29min ago Process: 2085 ExecStart=/bin/sh -c /data/moloch/bin/node viewer.js -c /data/moloch/etc/config.ini >> /data/moloch/logs/viewer.log 2>&1 (code=exited, status=1/FAILURE) Main PID: 2085 (code=exited, status=1/FAILURE)

Feb 23 14:14:39 SELKS systemd[1]: molochviewer-selks.service: Service RestartSec=1min 30s expired, scheduling restart. Feb 23 14:14:39 SELKS systemd[1]: molochviewer-selks.service: Scheduled restart job, restart counter is at 4. Feb 23 14:14:39 SELKS systemd[1]: Stopped Moloch Viewer. Feb 23 14:14:39 SELKS systemd[1]: molochviewer-selks.service: Start request repeated too quickly. Feb 23 14:14:39 SELKS systemd[1]: molochviewer-selks.service: Failed with result 'exit-code'. Feb 23 14:14:39 SELKS systemd[1]: Failed to start Moloch Viewer. ● molochpcapread-selks.service - Moloch Pcap Read Loaded: loaded (/etc/systemd/system/molochpcapread-selks.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Tue 2021-02-23 14:14:21 EST; 29min ago Process: 2082 ExecStart=/bin/sh -c /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m --copy --delete -R /data/nsm/ >> /data/moloch/logs/capture.log 2>&1 (code=exited, status=1/FAILURE) Main PID: 2082 (code=exited, status=1/FAILURE)

Feb 23 14:14:21 SELKS systemd[1]: molochpcapread-selks.service: Service RestartSec=1min 30s expired, scheduling restart. Feb 23 14:14:21 SELKS systemd[1]: molochpcapread-selks.service: Scheduled restart job, restart counter is at 4. Feb 23 14:14:21 SELKS systemd[1]: Stopped Moloch Pcap Read. Feb 23 14:14:21 SELKS systemd[1]: molochpcapread-selks.service: Start request repeated too quickly. Feb 23 14:14:21 SELKS systemd[1]: molochpcapread-selks.service: Failed with result 'exit-code'. Feb 23 14:14:21 SELKS systemd[1]: Failed to start Moloch Pcap Read. scirius RUNNING pid 4135, uptime 0:20:46 ii elasticsearch 7.11.1 amd64 Distributed RESTful search engine built for the cloud ii elasticsearch-curator 5.8.3 amd64 Have indices in Elasticsearch? This is the tool for you!\n\nLike a museum curator manages the exhibits and collections on display, \nElasticsearch Curator helps you curate, or manage your indices. ii evebox 1:0.12.0 amd64 no description given ii kibana 7.11.1 amd64 Explore and visualize your Elasticsearch data ii kibana-dashboards-stamus 2020122001 amd64 Kibana 6 dashboard templates. ii logstash 1:7.11.1-1 amd64 An extensible logging pipeline hi moloch 2.2.3-1 amd64 Moloch Full Packet System ii scirius 3.5.0-3 amd64 Django application to manage Suricata ruleset ii suricata 1:2021012201-0stamus0 amd64 Suricata open source multi-thread IDS/IPS/NSM system. Filesystem Type Size Used Avail Use% Mounted on udev devtmpfs 3.9G 0 3.9G 0% /dev tmpfs tmpfs 798M 8.7M 790M 2% /run /dev/sda1 ext4 484G 12G 448G 3% / tmpfs tmpfs 3.9G 0 3.9G 0% /dev/shm tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup tmpfs tmpfs 798M 0 798M 0% /run/user/1000 selks-user@SELKS:~$

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[P-Sandusky]

I used the one in /SELKS5-SELKS6/. No luck.

[P-Sandusky]

Here's the most recent (two?) runs from Elasticsearch.log:

[2021-02-23T14:08:40,776][INFO ][o.e.n.Node ] [SELKS] version[7.11.1], pid[849], build[default/deb/ff17057114c2199c9c1bbecc727003a907c0db7a/2021-02-15T13:44:09.394032Z], OS[Linux/4.19.0-14-amd64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9] [2021-02-23T14:08:40,800][INFO ][o.e.n.Node ] [SELKS] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true] [2021-02-23T14:08:40,801][INFO ][o.e.n.Node ] [SELKS] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-16348176929129466504, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms3988m, -Xmx3988m, -XX:MaxDirectMemorySize=2090860544, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true] [2021-02-23T14:08:50,572][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [aggs-matrix-stats] [2021-02-23T14:08:50,573][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [analysis-common] [2021-02-23T14:08:50,574][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [constant-keyword] [2021-02-23T14:08:50,574][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [flattened] [2021-02-23T14:08:50,575][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [frozen-indices] [2021-02-23T14:08:50,575][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [ingest-common] [2021-02-23T14:08:50,576][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [ingest-geoip] [2021-02-23T14:08:50,576][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [ingest-user-agent] [2021-02-23T14:08:50,577][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [kibana] [2021-02-23T14:08:50,578][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [lang-expression] [2021-02-23T14:08:50,578][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [lang-mustache] [2021-02-23T14:08:50,584][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [lang-painless] [2021-02-23T14:08:50,585][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [mapper-extras] [2021-02-23T14:08:50,587][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [mapper-version] [2021-02-23T14:08:50,587][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [parent-join] [2021-02-23T14:08:50,588][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [percolator] [2021-02-23T14:08:50,589][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [rank-eval] [2021-02-23T14:08:50,590][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [reindex] [2021-02-23T14:08:50,591][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [repositories-metering-api] [2021-02-23T14:08:50,591][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [repository-url] [2021-02-23T14:08:50,592][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [search-business-rules] [2021-02-23T14:08:50,592][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [searchable-snapshots] [2021-02-23T14:08:50,593][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [spatial] [2021-02-23T14:08:50,594][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [systemd] [2021-02-23T14:08:50,595][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [transform] [2021-02-23T14:08:50,596][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [transport-netty4] [2021-02-23T14:08:50,599][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [unsigned-long] [2021-02-23T14:08:50,600][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [vectors] [2021-02-23T14:08:50,600][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [wildcard] [2021-02-23T14:08:50,601][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-aggregate-metric] [2021-02-23T14:08:50,601][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-analytics] [2021-02-23T14:08:50,601][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-async] [2021-02-23T14:08:50,602][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-async-search] [2021-02-23T14:08:50,602][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-autoscaling] [2021-02-23T14:08:50,602][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-ccr] [2021-02-23T14:08:50,603][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-core] [2021-02-23T14:08:50,603][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-data-streams] [2021-02-23T14:08:50,603][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-deprecation] [2021-02-23T14:08:50,603][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-enrich] [2021-02-23T14:08:50,604][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-eql] [2021-02-23T14:08:50,604][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-fleet] [2021-02-23T14:08:50,604][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-graph] [2021-02-23T14:08:50,605][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-identity-provider] [2021-02-23T14:08:50,605][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-ilm] [2021-02-23T14:08:50,605][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-ingest] [2021-02-23T14:08:50,606][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-logstash] [2021-02-23T14:08:50,606][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-ml] [2021-02-23T14:08:50,606][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-monitoring] [2021-02-23T14:08:50,606][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-ql] [2021-02-23T14:08:50,607][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-rollup] [2021-02-23T14:08:50,607][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-runtime-fields] [2021-02-23T14:08:50,607][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-security] [2021-02-23T14:08:50,608][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-sql] [2021-02-23T14:08:50,608][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-stack] [2021-02-23T14:08:50,608][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-voting-only-node] [2021-02-23T14:08:50,609][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-watcher] [2021-02-23T14:08:50,610][INFO ][o.e.p.PluginsService ] [SELKS] no plugins loaded [2021-02-23T14:08:50,753][INFO ][o.e.e.NodeEnvironment ] [SELKS] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [447.3gb], net total_space [483.2gb], types [ext4] [2021-02-23T14:08:50,762][INFO ][o.e.e.NodeEnvironment ] [SELKS] heap size [3.8gb], compressed ordinary object pointers [true] [2021-02-23T14:08:51,570][INFO ][o.e.n.Node ] [SELKS] node name [SELKS], node ID [voxRMqkTTqGjtgSKpUG0FA], cluster name [elasticsearch], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest] [2021-02-23T14:09:02,384][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [SELKS] [controller/1147] [Main.cc@117] controller (64 bit): Version 7.11.1 (Build b7aec245e3d54f) Copyright (c) 2021 Elasticsearch BV [2021-02-23T14:09:03,919][INFO ][o.e.x.s.a.s.FileRolesStore] [SELKS] parsed [0] roles from file [/etc/elasticsearch/roles.yml] [2021-02-23T14:09:07,930][INFO ][o.e.t.NettyAllocator ] [SELKS] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}] [2021-02-23T14:09:08,164][INFO ][o.e.d.DiscoveryModule ] [SELKS] using discovery type [zen] and seed hosts providers [settings] [2021-02-23T14:09:09,493][INFO ][o.e.g.DanglingIndicesState] [SELKS] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually [2021-02-23T14:09:10,327][INFO ][o.e.n.Node ] [SELKS] initialized [2021-02-23T14:09:10,327][INFO ][o.e.n.Node ] [SELKS] starting ... [2021-02-23T14:09:10,374][INFO ][o.e.x.s.c.PersistentCache] [SELKS] persistent cache index loaded [2021-02-23T14:09:10,652][INFO ][o.e.t.TransportService ] [SELKS] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300} [2021-02-23T14:09:11,090][ERROR][o.e.b.Bootstrap ] [SELKS] Exception java.lang.IllegalStateException: The index [[logstash-fileinfo-2021.02.22/3gFBI1G8So-ctURgcbl92A]] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.11.1. at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.checkSupportedVersion(MetadataIndexUpgradeService.java:108) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:82) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:236) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:219) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:150) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.node.Node.start(Node.java:827) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:310) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:396) [elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.11.1.jar:7.11.1] at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.11.1.jar:7.11.1] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.11.1.jar:7.11.1] [2021-02-23T14:09:11,100][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [SELKS] uncaught exception in thread [main] org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: The index [[logstash-fileinfo-2021.02.22/3gFBI1G8So-ctURgcbl92A]] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.11.1. at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.11.1.jar:7.11.1] at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.11.1.jar:7.11.1] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.11.1.jar:7.11.1] Caused by: java.lang.IllegalStateException: The index [[logstash-fileinfo-2021.02.22/3gFBI1G8So-ctURgcbl92A]] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.11.1. at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.checkSupportedVersion(MetadataIndexUpgradeService.java:108) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:82) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:236) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:219) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:150) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.node.Node.start(Node.java:827) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:310) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:396) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.11.1.jar:7.11.1] ... 6 more [2021-02-23T14:09:11,107][INFO ][o.e.n.Node ] [SELKS] stopping ... [2021-02-23T14:09:11,135][INFO ][o.e.n.Node ] [SELKS] stopped [2021-02-23T14:09:11,136][INFO ][o.e.n.Node ] [SELKS] closing ... [2021-02-23T14:09:11,159][INFO ][o.e.n.Node ] [SELKS] closed [2021-02-23T14:09:11,162][INFO ][o.e.x.m.p.NativeController] [SELKS] Native controller process has stopped - no new native processes can be started [2021-02-23T14:22:35,579][INFO ][o.e.n.Node ] [SELKS] version[7.11.1], pid[3897], build[default/deb/ff17057114c2199c9c1bbecc727003a907c0db7a/2021-02-15T13:44:09.394032Z], OS[Linux/4.19.0-14-amd64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9] [2021-02-23T14:22:35,585][INFO ][o.e.n.Node ] [SELKS] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true] [2021-02-23T14:22:35,585][INFO ][o.e.n.Node ] [SELKS] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-11326985180863640013, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms3988m, -Xmx3988m, -XX:MaxDirectMemorySize=2090860544, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true] [2021-02-23T14:22:40,236][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [aggs-matrix-stats] [2021-02-23T14:22:40,237][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [analysis-common] [2021-02-23T14:22:40,237][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [constant-keyword] [2021-02-23T14:22:40,238][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [flattened] [2021-02-23T14:22:40,238][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [frozen-indices] [2021-02-23T14:22:40,238][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [ingest-common] [2021-02-23T14:22:40,239][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [ingest-geoip] [2021-02-23T14:22:40,239][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [ingest-user-agent] [2021-02-23T14:22:40,240][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [kibana] [2021-02-23T14:22:40,240][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [lang-expression] [2021-02-23T14:22:40,240][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [lang-mustache] [2021-02-23T14:22:40,241][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [lang-painless] [2021-02-23T14:22:40,241][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [mapper-extras] [2021-02-23T14:22:40,242][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [mapper-version] [2021-02-23T14:22:40,242][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [parent-join] [2021-02-23T14:22:40,243][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [percolator] [2021-02-23T14:22:40,243][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [rank-eval] [2021-02-23T14:22:40,244][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [reindex] [2021-02-23T14:22:40,244][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [repositories-metering-api] [2021-02-23T14:22:40,245][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [repository-url] [2021-02-23T14:22:40,246][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [search-business-rules] [2021-02-23T14:22:40,246][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [searchable-snapshots] [2021-02-23T14:22:40,247][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [spatial] [2021-02-23T14:22:40,247][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [systemd] [2021-02-23T14:22:40,247][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [transform] [2021-02-23T14:22:40,248][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [transport-netty4] [2021-02-23T14:22:40,248][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [unsigned-long] [2021-02-23T14:22:40,249][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [vectors] [2021-02-23T14:22:40,249][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [wildcard] [2021-02-23T14:22:40,250][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-aggregate-metric] [2021-02-23T14:22:40,250][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-analytics] [2021-02-23T14:22:40,250][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-async] [2021-02-23T14:22:40,251][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-async-search] [2021-02-23T14:22:40,251][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-autoscaling] [2021-02-23T14:22:40,251][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-ccr] [2021-02-23T14:22:40,252][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-core] [2021-02-23T14:22:40,252][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-data-streams] [2021-02-23T14:22:40,252][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-deprecation] [2021-02-23T14:22:40,253][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-enrich] [2021-02-23T14:22:40,253][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-eql] [2021-02-23T14:22:40,253][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-fleet] [2021-02-23T14:22:40,254][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-graph] [2021-02-23T14:22:40,254][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-identity-provider] [2021-02-23T14:22:40,255][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-ilm] [2021-02-23T14:22:40,255][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-ingest] [2021-02-23T14:22:40,256][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-logstash] [2021-02-23T14:22:40,256][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-ml] [2021-02-23T14:22:40,257][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-monitoring] [2021-02-23T14:22:40,257][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-ql] [2021-02-23T14:22:40,257][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-rollup] [2021-02-23T14:22:40,258][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-runtime-fields] [2021-02-23T14:22:40,258][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-security] [2021-02-23T14:22:40,258][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-sql] [2021-02-23T14:22:40,259][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-stack] [2021-02-23T14:22:40,259][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-voting-only-node] [2021-02-23T14:22:40,260][INFO ][o.e.p.PluginsService ] [SELKS] loaded module [x-pack-watcher] [2021-02-23T14:22:40,261][INFO ][o.e.p.PluginsService ] [SELKS] no plugins loaded [2021-02-23T14:22:40,353][INFO ][o.e.e.NodeEnvironment ] [SELKS] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [447.1gb], net total_space [483.2gb], types [ext4] [2021-02-23T14:22:40,354][INFO ][o.e.e.NodeEnvironment ] [SELKS] heap size [3.8gb], compressed ordinary object pointers [true] [2021-02-23T14:22:40,629][INFO ][o.e.n.Node ] [SELKS] node name [SELKS], node ID [voxRMqkTTqGjtgSKpUG0FA], cluster name [elasticsearch], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest] [2021-02-23T14:22:48,732][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [SELKS] [controller/4081] [Main.cc@117] controller (64 bit): Version 7.11.1 (Build b7aec245e3d54f) Copyright (c) 2021 Elasticsearch BV [2021-02-23T14:22:50,215][INFO ][o.e.x.s.a.s.FileRolesStore] [SELKS] parsed [0] roles from file [/etc/elasticsearch/roles.yml] [2021-02-23T14:22:52,161][INFO ][o.e.t.NettyAllocator ] [SELKS] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}] [2021-02-23T14:22:52,288][INFO ][o.e.d.DiscoveryModule ] [SELKS] using discovery type [zen] and seed hosts providers [settings] [2021-02-23T14:22:53,236][INFO ][o.e.g.DanglingIndicesState] [SELKS] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually [2021-02-23T14:22:54,171][INFO ][o.e.n.Node ] [SELKS] initialized [2021-02-23T14:22:54,171][INFO ][o.e.n.Node ] [SELKS] starting ... [2021-02-23T14:22:54,220][INFO ][o.e.x.s.c.PersistentCache] [SELKS] persistent cache index loaded [2021-02-23T14:22:54,351][INFO ][o.e.t.TransportService ] [SELKS] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300} [2021-02-23T14:22:54,801][ERROR][o.e.b.Bootstrap ] [SELKS] Exception java.lang.IllegalStateException: The index [[logstash-2021.02.22/lgzF808KSneDDI6whnU_4w]] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.11.1. at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.checkSupportedVersion(MetadataIndexUpgradeService.java:108) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:82) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:236) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:219) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:150) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.node.Node.start(Node.java:827) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:310) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:396) [elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.11.1.jar:7.11.1] at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.11.1.jar:7.11.1] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.11.1.jar:7.11.1] [2021-02-23T14:22:54,810][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [SELKS] uncaught exception in thread [main] org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: The index [[logstash-2021.02.22/lgzF808KSneDDI6whnU_4w]] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.11.1. at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.11.1.jar:7.11.1] at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.11.1.jar:7.11.1] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.11.1.jar:7.11.1] Caused by: java.lang.IllegalStateException: The index [[logstash-2021.02.22/lgzF808KSneDDI6whnU_4w]] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.11.1. at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.checkSupportedVersion(MetadataIndexUpgradeService.java:108) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:82) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:236) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:219) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:150) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.node.Node.start(Node.java:827) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:310) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:396) ~[elasticsearch-7.11.1.jar:7.11.1] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.11.1.jar:7.11.1] ... 6 more [2021-02-23T14:22:54,816][INFO ][o.e.n.Node ] [SELKS] stopping ... [2021-02-23T14:22:54,846][INFO ][o.e.n.Node ] [SELKS] stopped [2021-02-23T14:22:54,847][INFO ][o.e.n.Node ] [SELKS] closing ... [2021-02-23T14:22:54,879][INFO ][o.e.n.Node ] [SELKS] closed [2021-02-23T14:22:54,883][INFO ][o.e.x.m.p.NativeController] [SELKS] Native controller process has stopped - no new native processes can be started selks-user@SELKS:~$

[P-Sandusky]

Not to spam overmuch, but... Running updates directly from apt didn't set things to rights, either.

Earlier, Elastic was giving some indexing errors. I found an earlier issue (https://github.com/StamusNetworks/SELKS/issues/238) that I hoped would get around the problem, but there hasn't been much happiness to be found where Elastic is concerned.

[P-Sandusky]

Here's what I'm getting when I run the install script (I killed Logstash before I ran this, else it would hang):

selks-user@SELKS:~/SELKS/scripts$ ls -l total 12 drwxr-xr-x 2 root root 4096 Feb 22 17:04 SELKS3-SELKS4 drwxr-xr-x 2 root root 4096 Feb 22 17:04 SELKS4-SELKS5 drwxr-xr-x 2 root root 4096 Feb 22 17:04 SELKS5-SELKS6 selks-user@SELKS:~/SELKS/scripts$ cd SELKS5-SELKS6 selks-user@SELKS:~/SELKS/scripts/SELKS5-SELKS6$ ls -l total 20 -rw-r--r-- 1 root root 102 Feb 22 17:04 README.rst -rwxr-xr-x 1 root root 12853 Feb 22 17:04 SN-S5-S6-Upgrade.sh selks-user@SELKS:~/SELKS/scripts/SELKS5-SELKS6$ sudo ./SN-S5-S6-Upgrade.sh

• (( 0 != 0 ))
• mkdir -p /opt/selks/preupgrade/elasticsearch/etc/default /opt/selks/preupgrade/elasticsearch/etc/elasticsearch
• mv /etc/alternatives/desktop-background /opt/selks/preupgrade mv: cannot stat '/etc/alternatives/desktop-background': No such file or directory
• /bin/systemctl stop elasticsearch
• /bin/systemctl stop kibana
• /bin/systemctl stop logstash
• /bin/systemctl stop suricata
• /usr/bin/supervisorctl stop scirius scirius: stopped
• '[' -f /etc/apt/sources.list.d/elastic-6.x.list ']'
• cat
• '[' -f /etc/apt/sources.list.d/curator5.list ']'
• mv /etc/apt/sources.list.d/curator5.list /opt/selks/preupgrade/curator5.list.orig
• cat
• '[' -f /etc/nginx/sites-available/default ']'
• rm -rf /etc/nginx/sites-enabled/default
• '[' -f /etc/nginx/sites-available/selks5.conf ']'
• rm -rf /etc/nginx/sites-available/selks5.conf
• rm -rf /etc/nginx/sites-enabled/selks5.conf
• '[' -f /etc/nginx/sites-available/selks5.conf ']'
• cat
• ln -s /etc/nginx/sites-available/selks6.conf /etc/nginx/sites-enabled/selks6.conf ln: failed to create symbolic link '/etc/nginx/sites-enabled/selks6.conf': File exists
• /bin/systemctl restart nginx Job for nginx.service failed because the control process exited with error code. See "systemctl status nginx.service" and "journalctl -xe" for details.
• '[' -f /etc/logstash/conf.d/logstash.conf ']'
• mv /etc/logstash/conf.d/logstash.conf /opt/selks/preupgrade/logstash.conf.orig
• cat
• '[' -f /etc/logstash/elasticsearch6-template.json ']'
• cat
• '[' -f /etc/apt/sources.list.d/selks5.list ']'
• cat
• wget -qO - http://packages.stamus-networks.com/packages.selks6.stamus-networks.com.gpg.key
• apt-key add - OK
• /bin/systemctl stop kibana
• '[' -f /usr/lib/systemd/system/elasticsearch.service ']'
• cp /usr/lib/systemd/system/elasticsearch.service /opt/selks/preupgrade/elasticsearch.service.orig
• cp -r /etc/elasticsearch/elasticsearch.keystore /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/jvm.options /etc/elasticsearch/jvm.options.d /etc/elasticsearch/log4j2.properties /etc/elasticsearch/role_mapping.yml /etc/elasticsearch/roles.yml /etc/elasticsearch/users /etc/elasticsearch/users_roles /opt/selks/preupgrade/elasticsearch/etc/
• cp /etc/default/elasticsearch /opt/selks/preupgrade/elasticsearch/etc/default/
• sed -i s/stretch/buster/g /etc/apt/sources.list
• apt-get update Hit:1 http://security.debian.org/debian-security buster/updates InRelease Hit:2 http://packages.stamus-networks.com/selks4/debian stretch InRelease Hit:3 https://packages.elastic.co/curator/5/debian9 stable InRelease Ign:4 https://artifacts.elastic.co/packages/5.x/apt stable InRelease Hit:5 http://files.evebox.org/evebox/debian stable InRelease Hit:6 http://packages.stamus-networks.com/selks4/debian-kernel stretch InRelease Hit:7 http://packages.stamus-networks.com/selks6/debian buster InRelease Hit:8 https://artifacts.elastic.co/packages/7.x/apt stable InRelease Hit:9 http://debian.csail.mit.edu/debian buster InRelease Hit:10 http://packages.stamus-networks.com/selks6/debian-kernel buster InRelease Hit:11 http://debian.csail.mit.edu/debian buster-updates InRelease Hit:12 https://artifacts.elastic.co/packages/5.x/apt stable Release Reading package lists... Done W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12 W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12 W: Target Translations (main/i18n/Translation-en_US) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12 W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12 W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2 W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2 W: Target Translations (main/i18n/Translation-en_US) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2 W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2 W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13 W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13 W: Target Translations (main/i18n/Translation-en_US) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13 W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13 W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2 W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2 W: Target Translations (main/i18n/Translation-en_US) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2 W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2 W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12 W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12 W: Target Translations (main/i18n/Translation-en_US) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12 W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12 W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13 W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13 W: Target Translations (main/i18n/Translation-en_US) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13 W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13
• DEBIAN_FRONTEND=noninteractive
• apt-get -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef -y dist-upgrade Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
• exit_status=0
• [[ 0 -ne 0 ]]
• chown root:elasticsearch /etc/default/elasticsearch
• '[' -f /usr/lib/systemd/system/elasticsearch.service.dpkg-new ']'
• '[' -f /etc/default/elasticsearch.dpkg-new ']'
• chown -R kibana /usr/share/kibana/optimize/ chown: cannot access '/usr/share/kibana/optimize/': No such file or directory
• /bin/systemctl restart elasticsearch Job for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
• /bin/systemctl restart kibana
• /usr/share/logstash/bin/logstash-plugin install logstash-filter-geoip Using bundled JDK: /usr/share/logstash/jdk OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. Validating logstash-filter-geoip Installing logstash-filter-geoip Installation successful
• /bin/systemctl restart logstash
• chown logstash -R /data/nsm/
• /bin/systemctl restart suricata
• /usr/bin/supervisorctl restart scirius scirius: ERROR (not running) scirius: started
• sleep 30
• apt-get update Hit:1 http://debian.csail.mit.edu/debian buster InRelease Hit:2 http://packages.stamus-networks.com/selks4/debian stretch InRelease Hit:3 https://packages.elastic.co/curator/5/debian9 stable InRelease Hit:4 http://debian.csail.mit.edu/debian buster-updates InRelease Ign:5 https://artifacts.elastic.co/packages/5.x/apt stable InRelease Hit:6 http://packages.stamus-networks.com/selks4/debian-kernel stretch InRelease Hit:7 http://files.evebox.org/evebox/debian stable InRelease Hit:8 http://packages.stamus-networks.com/selks6/debian buster InRelease Hit:9 http://security.debian.org/debian-security buster/updates InRelease Hit:10 https://artifacts.elastic.co/packages/7.x/apt stable InRelease Hit:11 http://packages.stamus-networks.com/selks6/debian-kernel buster InRelease Hit:12 https://artifacts.elastic.co/packages/5.x/apt stable Release Reading package lists... Done W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12 W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12 W: Target Translations (main/i18n/Translation-en_US) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12 W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12 W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2 W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2 W: Target Translations (main/i18n/Translation-en_US) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2 W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2 W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13 W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13 W: Target Translations (main/i18n/Translation-en_US) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13 W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13 W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2 W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2 W: Target Translations (main/i18n/Translation-en_US) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2 W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/elastic-7.x.list:1 and /etc/apt/sources.list.d/elastic-7.x.list:2 W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12 W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12 W: Target Translations (main/i18n/Translation-en_US) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12 W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/selks6.list:5 and /etc/apt/sources.list.d/selks6.list:12 W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13 W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13 W: Target Translations (main/i18n/Translation-en_US) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13 W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/selks6.list:6 and /etc/apt/sources.list.d/selks6.list:13
• apt-get -y install elasticsearch-curator Reading package lists... Done Building dependency tree Reading state information... Done elasticsearch-curator is already the newest version (5.8.3). 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
• mv /opt/selks/delete-old-logs.sh /opt/selks/preupgrade/delete-old-logs.sh
• cat
• mkdir -p /opt/molochtmp
• cd /opt/molochtmp/
• apt-get -y install libwww-perl libjson-perl libyaml-dev libcrypto++6 Reading package lists... Done Building dependency tree Reading state information... Done libcrypto++6 is already the newest version (5.6.4-8). libjson-perl is already the newest version (4.02000-1). libwww-perl is already the newest version (6.36-2). libyaml-dev is already the newest version (0.2.1-1). 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
• wget https://files.molo.ch/builds/ubuntu-18.04/moloch_2.2.3-1_amd64.deb --2021-02-23 15:14:42-- https://files.molo.ch/builds/ubuntu-18.04/moloch_2.2.3-1_amd64.deb Resolving files.molo.ch (files.molo.ch)... 52.85.79.27, 52.85.79.30, 52.85.79.126, ... Connecting to files.molo.ch (files.molo.ch)|52.85.79.27|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 82686856 (79M) [application/x-debian-package] Saving to: ‘moloch_2.2.3-1_amd64.deb’

moloch_2.2.3-1_amd64.deb 100%[=============================================================================>] 78.86M 10.9MB/s in 7.4s

2021-02-23 15:14:50 (10.7 MB/s) - ‘moloch_2.2.3-1_amd64.deb’ saved [82686856/82686856]

• dpkg -i moloch_2.2.3-1_amd64.deb dpkg: warning: files list file for package 'python-minimal' missing; assuming package has no files currently installed dpkg: warning: files list file for package 'python2-minimal' missing; assuming package has no files currently installed (Reading database ... 203043 files and directories currently installed.) Preparing to unpack moloch_2.2.3-1_amd64.deb ... Unpacking moloch (2.2.3-1) over (2.2.3-1) ... Setting up moloch (2.2.3-1) ... READ /data/moloch/README.txt and RUN /data/moloch/bin/Configure
• cd /opt/
• rm /opt/molochtmp -r
• apt-mark hold moloch moloch set on hold.
• echo '0 3 * root ( /data/moloch/db/db.pl http://127.0.0.1:9200 expire daily 14 )'
• sed -i 's/ELASTICSEARCH_VERSION = 6/ELASTICSEARCH_VERSION = 7/g' /etc/scirius/local_settings.py
• sed -i 's/KIBANA_VERSION = 6/KIBANA_VERSION = 7/g' /etc/scirius/local_settings.py
• sed -i 's/KIBANA_INDEX = "kibana-int"/KIBANA_INDEX = ".kibana"/g' /etc/scirius/local_settings.py
• sed -i 's/KIBANA6_DASHBOARDS_PATH = "\/opt\/selks\/kibana6-dashboards\/"/KIBANA6_DASHBOARDS_PATH = "\/opt\/selks\/kibana7-dashboards\/"/g' /etc/scirius/local_settings.py
• echo 'ELASTICSEARCH_KEYWORD = "keyword"'
• echo 'USE_MOLOCH = True'
• echo 'MOLOCH_URL = "http://localhost:8005"'
• /usr/bin/supervisorctl restart scirius scirius: stopped scirius: started
• curl -XDELETE 'http://localhost:9200/.kibana*' curl: (7) Failed to connect to localhost port 9200: Connection refused
• /bin/systemctl restart kibana
• sleep 20
• selks-first-time-setup_stamus START of first time setup script - Tue 23 Feb 2021 03:16:58 PM EST

Setting up sniffing interface

Please supply a network interface(s) to set up SELKS Suricata IDPS thread detection on 0: ens32 1: ens33 2: lo Please type in interface or space delimited interfaces below and hit "Enter". Example: eth1 OR Example: eth1 eth2 eth3

Configure threat detection for INTERFACE(S): ens33

The supplied network interface(s): ens33

DONE! FPC - Full Packet Capture. Suricata will rotate and delete the pcap captured files. FPC_Retain - Full Packet Capture with having Moloch's pcap retention/rotation. Keeps the pcaps as long as there is space available. None - disable packet capture

1) FPC 2) FPC_Retain 3) NONE Please choose an option. Type in a number and hit "Enter" 2 Enable Full Pcacket Capture with pcap retaining

Starting Moloch DB set up

% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 9200: Connection refused Traceback (most recent call last): File "bin/manage.py", line 10, in execute_from_command_line(sys.argv) File "/usr/share/python/scirius/local/lib/python2.7/site-packages/django/core/management/init.py", line 364, in execute_from_command_line utility.execute() File "/usr/share/python/scirius/local/lib/python2.7/site-packages/django/core/management/init.py", line 356, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/usr/share/python/scirius/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv self.execute(*args, cmd_options) File "/usr/share/python/scirius/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute output = self.handle(*args, *options) File "/usr/share/python/scirius/local/lib/python2.7/site-packages/rules/management/commands/kibana_reset.py", line 38, in handle self.kibana_reset() File "/usr/share/python/scirius/local/lib/python2.7/site-packages/rules/es_data.py", line 1972, in kibana_reset self._create_kibana_mappings() File "/usr/share/python/scirius/local/lib/python2.7/site-packages/rules/es_data.py", line 1852, in _create_kibana_mappings if not self.client.indices.exists('.kibana'): File "/usr/share/python/scirius/local/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped return func(args, params=params, kwargs) File "/usr/share/python/scirius/local/lib/python2.7/site-packages/elasticsearch/client/indices.py", line 268, in exists return self.transport.perform_request("HEAD", _make_path(index), params=params) File "/usr/share/python/scirius/local/lib/python2.7/site-packages/elasticsearch/transport.py", line 358, in perform_request timeout=timeout, File "/usr/share/python/scirius/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 224, in perform_request raise ConnectionError("N/A", str(e), e) elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7ff516b25710>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7ff516b25710>: Failed to establish a new connection: [Errno 111] Connection refused) Dashboards loading set up job failed...Exiting...

Exited with ERROR

FINISH of first time setup script - Tue 23 Feb 2021 03:17:58 PM EST

Exited with FAILED Full log located at - /opt/selks/log/selks-first-time-setup_stamus.log Press enter to continue

selks-user@SELKS:~/SELKS/scripts/SELKS5-SELKS6$

[pevma]

It seems Elastic is not up?

[P-Sandusky]

Still isn't. Now it's showing as "dead."

I tried a fresh install from the 6.0 build, hoping to put back anything that might be missing and to start again as fresh as possible. Here's the latest health check report:

selks-user@SELKS:~$ sudo selks-health-check_stamus -l ● suricata.service - LSB: Next Generation IDS/IPS Loaded: loaded (/etc/init.d/suricata; generated) Active: active (exited) since Tue 2021-02-23 20:49:01 EST; 11h ago Docs: man:systemd-sysv-generator(8) Process: 865 ExecStart=/etc/init.d/suricata start (code=exited, status=0/SUCCESS)

Feb 23 20:49:01 SELKS systemd[1]: Starting LSB: Next Generation IDS/IPS... Feb 23 20:49:01 SELKS suricata[865]: Starting suricata in IDS (af-packet) mode... done. Feb 23 20:49:01 SELKS systemd[1]: Started LSB: Next Generation IDS/IPS. ● elasticsearch.service - LSB: Starts elasticsearch Loaded: loaded (/etc/init.d/elasticsearch; generated) Active: inactive (dead) Docs: man:systemd-sysv-generator(8) ● logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2021-02-23 20:49:00 EST; 11h ago Main PID: 516 (java) Tasks: 58 (limit: 4915) Memory: 1000.5M CGroup: /system.slice/logstash.service └─516 /usr/share/logstash/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=…

Feb 24 08:29:50 SELKS logstash[516]: [2021-02-24T08:29:50,891][WARN ][logstash.outputs.elasticsearch][main] Attempted to re… Feb 24 08:29:51 SELKS logstash[516]: [2021-02-24T08:29:51,021][WARN ][logstash.outputs.elasticsearch][main] Attempted to re… Feb 24 08:29:55 SELKS logstash[516]: [2021-02-24T08:29:55,894][WARN ][logstash.outputs.elasticsearch][main] Attempted to re… Feb 24 08:29:56 SELKS logstash[516]: [2021-02-24T08:29:56,026][WARN ][logstash.outputs.elasticsearch][main] Attempted to re… Feb 24 08:30:00 SELKS logstash[516]: [2021-02-24T08:30:00,898][WARN ][logstash.outputs.elasticsearch][main] Attempted to re… Feb 24 08:30:01 SELKS logstash[516]: [2021-02-24T08:30:01,030][WARN ][logstash.outputs.elasticsearch][main] Attempted to re… Feb 24 08:30:05 SELKS logstash[516]: [2021-02-24T08:30:05,901][WARN ][logstash.outputs.elasticsearch][main] Attempted to re… Feb 24 08:30:06 SELKS logstash[516]: [2021-02-24T08:30:06,035][WARN ][logstash.outputs.elasticsearch][main] Attempted to re… Feb 24 08:30:10 SELKS logstash[516]: [2021-02-24T08:30:10,904][WARN ][logstash.outputs.elasticsearch][main] Attempted to re… Feb 24 08:30:11 SELKS logstash[516]: [2021-02-24T08:30:11,040][WARN ][logstash.outputs.elasticsearch][main] Attempted to re… Hint: Some lines were ellipsized, use -l to show in full. ● kibana.service - Kibana Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Tue 2021-02-23 20:49:11 EST; 11h ago Docs: https://www.elastic.co Process: 892 ExecStart=/usr/share/kibana/bin/kibana --logging.dest=/var/log/kibana/kibana.log --pid.file=/run/kibana/kibana.pid (code=exited, status=200/CHDIR) Main PID: 892 (code=exited, status=200/CHDIR)

Feb 23 20:49:11 SELKS systemd[1]: kibana.service: Service RestartSec=3s expired, scheduling restart. Feb 23 20:49:11 SELKS systemd[1]: kibana.service: Scheduled restart job, restart counter is at 3. Feb 23 20:49:11 SELKS systemd[1]: Stopped Kibana. Feb 23 20:49:11 SELKS systemd[1]: kibana.service: Start request repeated too quickly. Feb 23 20:49:11 SELKS systemd[1]: kibana.service: Failed with result 'exit-code'. Feb 23 20:49:11 SELKS systemd[1]: Failed to start Kibana. Unit evebox.service could not be found. ● molochviewer-selks.service - Moloch Viewer Loaded: loaded (/etc/systemd/system/molochviewer-selks.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Tue 2021-02-23 20:55:02 EST; 11h ago Process: 1031 ExecStart=/bin/sh -c /data/moloch/bin/node viewer.js -c /data/moloch/etc/config.ini >> /data/moloch/logs/viewer.log 2>&1 (code=exited, status=200/CHDIR) Main PID: 1031 (code=exited, status=200/CHDIR)

Feb 23 20:55:02 SELKS systemd[1]: molochviewer-selks.service: Service RestartSec=1min 30s expired, scheduling restart. Feb 23 20:55:02 SELKS systemd[1]: molochviewer-selks.service: Scheduled restart job, restart counter is at 4. Feb 23 20:55:02 SELKS systemd[1]: Stopped Moloch Viewer. Feb 23 20:55:02 SELKS systemd[1]: molochviewer-selks.service: Start request repeated too quickly. Feb 23 20:55:02 SELKS systemd[1]: molochviewer-selks.service: Failed with result 'exit-code'. Feb 23 20:55:02 SELKS systemd[1]: Failed to start Moloch Viewer. ● molochpcapread-selks.service - Moloch Pcap Read Loaded: loaded (/etc/systemd/system/molochpcapread-selks.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Tue 2021-02-23 20:55:01 EST; 11h ago Process: 1029 ExecStart=/bin/sh -c /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m --copy --delete -R /data/nsm/ >> /data/moloch/logs/capture.log 2>&1 (code=exited, status=127) Main PID: 1029 (code=exited, status=127)

Feb 23 20:55:01 SELKS systemd[1]: molochpcapread-selks.service: Service RestartSec=1min 30s expired, scheduling restart. Feb 23 20:55:01 SELKS systemd[1]: molochpcapread-selks.service: Scheduled restart job, restart counter is at 4. Feb 23 20:55:01 SELKS systemd[1]: Stopped Moloch Pcap Read. Feb 23 20:55:01 SELKS systemd[1]: molochpcapread-selks.service: Start request repeated too quickly. Feb 23 20:55:01 SELKS systemd[1]: molochpcapread-selks.service: Failed with result 'exit-code'. Feb 23 20:55:01 SELKS systemd[1]: Failed to start Moloch Pcap Read. /usr/bin/selks-health-check_stamus: line 30: /usr/bin/supervisorctl: No such file or directory rc elasticsearch 7.11.1 amd64 Distributed RESTful search engine built for the cloud ii elasticsearch-curator 5.8.3 amd64 Have indices in Elasticsearch? This is the tool for you!\n\nLike a museum curator manages the exhibits and collections on display, \nElasticsearch Curator helps you curate, or manage your indices. rc evebox 1:0.12.0 amd64 no description given rc kibana 7.11.1 amd64 Explore and visualize your Elasticsearch data ii kibana-dashboards-stamus 2020122001 amd64 Kibana 6 dashboard templates. ii logstash 1:7.11.1-1 amd64 An extensible logging pipeline rc scirius 3.5.0-3 amd64 Django application to manage Suricata ruleset rc suricata 1:2021012201-0stamus0 amd64 Suricata open source multi-thread IDS/IPS/NSM system. Filesystem Type Size Used Avail Use% Mounted on udev devtmpfs 3.9G 0 3.9G 0% /dev tmpfs tmpfs 798M 21M 778M 3% /run /dev/sda1 ext4 484G 23G 437G 5% / tmpfs tmpfs 3.9G 0 3.9G 0% /dev/shm tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup tmpfs tmpfs 798M 0 798M 0% /run/user/1000 selks-user@SELKS:~$

[pevma]

Ok - so if you tail the elasticsearch logs there should be some info > /var/log/elasticsearch/elasticsearch.log ?

[P-Sandusky]

Elastic hasn't even built out a folder for itself in /var/log. Kibana and Logstash have folders, but Elastic has not generated one.

[P-Sandusky]

I had to install an earlier version of Elastic and then upgrade it to the latest version to get it to behave itself semi-appropriately. Now it's failing again, but it has generated a log file. Here's what I got:

selks-user@SELKS:~$ sudo tail -l var/log/elasticsearch/elasticsearch.log
tail: cannot open 'var/log/elasticsearch/elasticsearch.log' for reading: No such file or directory
selks-user@SELKS:~$ sudo tail -l  /var/log/elasticsearch/elasticsearch.log
        at org.elasticsearch.node.Node.start(Node.java:827) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:310) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:396) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.11.1.jar:7.11.1]
        ... 6 more
[2021-02-24T16:38:29,805][INFO ][o.e.n.Node               ] [SELKS] stopping ...
[2021-02-24T16:38:29,838][INFO ][o.e.n.Node               ] [SELKS] stopped
[2021-02-24T16:38:29,839][INFO ][o.e.n.Node               ] [SELKS] closing ...
[2021-02-24T16:38:29,872][INFO ][o.e.n.Node               ] [SELKS] closed
[2021-02-24T16:38:29,876][INFO ][o.e.x.m.p.NativeController] [SELKS] Native controller process has stopped - no new native processes can be started
selks-user@SELKS:~$

[pevma]

It seems incomplete - can you tail the last 200 lines please?

[P-Sandusky]

Here's something a little more complete...

selks-user@SELKS:~$ sudo tail -200 /var/log/elasticsearch/elasticsearch.log
[2021-02-24T16:34:40,469][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [transport-netty4]
[2021-02-24T16:34:40,470][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [unsigned-long]
[2021-02-24T16:34:40,470][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [vectors]
[2021-02-24T16:34:40,471][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [wildcard]
[2021-02-24T16:34:40,471][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-aggregate-metric]
[2021-02-24T16:34:40,471][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-analytics]
[2021-02-24T16:34:40,472][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-async]
[2021-02-24T16:34:40,472][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-async-search]
[2021-02-24T16:34:40,473][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-autoscaling]
[2021-02-24T16:34:40,473][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-ccr]
[2021-02-24T16:34:40,474][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-core]
[2021-02-24T16:34:40,474][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-data-streams]
[2021-02-24T16:34:40,474][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-deprecation]
[2021-02-24T16:34:40,475][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-enrich]
[2021-02-24T16:34:40,475][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-eql]
[2021-02-24T16:34:40,476][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-fleet]
[2021-02-24T16:34:40,476][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-graph]
[2021-02-24T16:34:40,476][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-identity-provider]
[2021-02-24T16:34:40,477][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-ilm]
[2021-02-24T16:34:40,477][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-ingest]
[2021-02-24T16:34:40,478][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-logstash]
[2021-02-24T16:34:40,478][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-ml]
[2021-02-24T16:34:40,478][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-monitoring]
[2021-02-24T16:34:40,479][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-ql]
[2021-02-24T16:34:40,479][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-rollup]
[2021-02-24T16:34:40,480][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-runtime-fields]
[2021-02-24T16:34:40,480][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-security]
[2021-02-24T16:34:40,480][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-sql]
[2021-02-24T16:34:40,481][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-stack]
[2021-02-24T16:34:40,481][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-voting-only-node]
[2021-02-24T16:34:40,481][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-watcher]
[2021-02-24T16:34:40,482][INFO ][o.e.p.PluginsService     ] [SELKS] no plugins loaded
[2021-02-24T16:34:40,619][INFO ][o.e.e.NodeEnvironment    ] [SELKS] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [435.5gb], net total_space [483.2gb], types [ext4]
[2021-02-24T16:34:40,620][INFO ][o.e.e.NodeEnvironment    ] [SELKS] heap size [3.8gb], compressed ordinary object pointers [true]
[2021-02-24T16:34:42,016][INFO ][o.e.n.Node               ] [SELKS] node name [SELKS], node ID [voxRMqkTTqGjtgSKpUG0FA], cluster name [elasticsearch], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2021-02-24T16:34:54,040][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [SELKS] [controller/1262] [Main.cc@117] controller (64 bit): Version 7.11.1 (Build b7aec245e3d54f) Copyright (c) 2021 Elasticsearch BV
[2021-02-24T16:34:55,539][INFO ][o.e.x.s.a.s.FileRolesStore] [SELKS] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2021-02-24T16:34:58,809][INFO ][o.e.t.NettyAllocator     ] [SELKS] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2021-02-24T16:34:58,953][INFO ][o.e.d.DiscoveryModule    ] [SELKS] using discovery type [zen] and seed hosts providers [settings]
[2021-02-24T16:35:00,031][INFO ][o.e.g.DanglingIndicesState] [SELKS] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-02-24T16:35:01,359][INFO ][o.e.n.Node               ] [SELKS] initialized
[2021-02-24T16:35:01,360][INFO ][o.e.n.Node               ] [SELKS] starting ...
[2021-02-24T16:35:01,441][INFO ][o.e.x.s.c.PersistentCache] [SELKS] persistent cache index loaded
[2021-02-24T16:35:01,776][INFO ][o.e.t.TransportService   ] [SELKS] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2021-02-24T16:35:03,218][ERROR][o.e.b.Bootstrap          ] [SELKS] Exception
java.lang.IllegalStateException: The index [[logstash-dns-2021.02.22/w_xhgPARSamErzGJlcQQEA]] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.11.1.
        at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.checkSupportedVersion(MetadataIndexUpgradeService.java:108) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:82) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:236) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:219) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:150) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.node.Node.start(Node.java:827) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:310) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:396) [elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.11.1.jar:7.11.1]
        at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.11.1.jar:7.11.1]
[2021-02-24T16:35:03,232][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [SELKS] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: The index [[logstash-dns-2021.02.22/w_xhgPARSamErzGJlcQQEA]] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.11.1.
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.11.1.jar:7.11.1]
        at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.11.1.jar:7.11.1]
Caused by: java.lang.IllegalStateException: The index [[logstash-dns-2021.02.22/w_xhgPARSamErzGJlcQQEA]] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.11.1.
        at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.checkSupportedVersion(MetadataIndexUpgradeService.java:108) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:82) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:236) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:219) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:150) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.node.Node.start(Node.java:827) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:310) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:396) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.11.1.jar:7.11.1]
        ... 6 more
[2021-02-24T16:35:03,240][INFO ][o.e.n.Node               ] [SELKS] stopping ...
[2021-02-24T16:35:03,268][INFO ][o.e.n.Node               ] [SELKS] stopped
[2021-02-24T16:35:03,268][INFO ][o.e.n.Node               ] [SELKS] closing ...
[2021-02-24T16:35:03,295][INFO ][o.e.n.Node               ] [SELKS] closed
[2021-02-24T16:35:03,298][INFO ][o.e.x.m.p.NativeController] [SELKS] Native controller process has stopped - no new native processes can be started
[2021-02-24T16:38:12,347][INFO ][o.e.n.Node               ] [SELKS] version[7.11.1], pid[3238], build[default/deb/ff17057114c2199c9c1bbecc727003a907c0db7a/2021-02-15T13:44:09.394032Z], OS[Linux/4.19.0-14-amd64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2021-02-24T16:38:12,353][INFO ][o.e.n.Node               ] [SELKS] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2021-02-24T16:38:12,354][INFO ][o.e.n.Node               ] [SELKS] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-14297637706770411412, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms3988m, -Xmx3988m, -XX:MaxDirectMemorySize=2090860544, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [aggs-matrix-stats]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [analysis-common]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [constant-keyword]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [flattened]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [frozen-indices]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [ingest-common]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [ingest-geoip]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [ingest-user-agent]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [kibana]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [lang-expression]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [lang-mustache]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [lang-painless]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [mapper-extras]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [mapper-version]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [parent-join]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [percolator]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [rank-eval]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [reindex]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [repositories-metering-api]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [repository-url]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [search-business-rules]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [searchable-snapshots]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [spatial]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [systemd]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [transform]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [transport-netty4]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [unsigned-long]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [vectors]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [wildcard]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-aggregate-metric]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-analytics]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-async]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-async-search]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-autoscaling]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-ccr]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-core]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-data-streams]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-deprecation]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-enrich]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-eql]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-fleet]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-graph]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-identity-provider]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-ilm]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-ingest]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-logstash]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-ml]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-monitoring]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-ql]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-rollup]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-runtime-fields]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-security]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-sql]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-stack]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-voting-only-node]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-watcher]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] no plugins loaded
[2021-02-24T16:38:17,795][INFO ][o.e.e.NodeEnvironment    ] [SELKS] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [435.4gb], net total_space [483.2gb], types [ext4]
[2021-02-24T16:38:17,796][INFO ][o.e.e.NodeEnvironment    ] [SELKS] heap size [3.8gb], compressed ordinary object pointers [true]
[2021-02-24T16:38:18,026][INFO ][o.e.n.Node               ] [SELKS] node name [SELKS], node ID [voxRMqkTTqGjtgSKpUG0FA], cluster name [elasticsearch], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2021-02-24T16:38:25,224][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [SELKS] [controller/3424] [Main.cc@117] controller (64 bit): Version 7.11.1 (Build b7aec245e3d54f) Copyright (c) 2021 Elasticsearch BV
[2021-02-24T16:38:26,122][INFO ][o.e.x.s.a.s.FileRolesStore] [SELKS] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2021-02-24T16:38:27,651][INFO ][o.e.t.NettyAllocator     ] [SELKS] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2021-02-24T16:38:27,746][INFO ][o.e.d.DiscoveryModule    ] [SELKS] using discovery type [zen] and seed hosts providers [settings]
[2021-02-24T16:38:28,543][INFO ][o.e.g.DanglingIndicesState] [SELKS] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-02-24T16:38:29,355][INFO ][o.e.n.Node               ] [SELKS] initialized
[2021-02-24T16:38:29,356][INFO ][o.e.n.Node               ] [SELKS] starting ...
[2021-02-24T16:38:29,423][INFO ][o.e.x.s.c.PersistentCache] [SELKS] persistent cache index loaded
[2021-02-24T16:38:29,554][INFO ][o.e.t.TransportService   ] [SELKS] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2021-02-24T16:38:29,794][ERROR][o.e.b.Bootstrap          ] [SELKS] Exception
java.lang.IllegalStateException: The index [[logstash-http-2021.02.22/GCm01ePnSx6Ypg84hnKdxQ]] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.11.1.
        at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.checkSupportedVersion(MetadataIndexUpgradeService.java:108) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:82) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:236) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:219) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:150) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.node.Node.start(Node.java:827) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:310) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:396) [elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.11.1.jar:7.11.1]
        at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.11.1.jar:7.11.1]
[2021-02-24T16:38:29,799][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [SELKS] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: The index [[logstash-http-2021.02.22/GCm01ePnSx6Ypg84hnKdxQ]] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.11.1.
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.11.1.jar:7.11.1]
        at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.11.1.jar:7.11.1]
Caused by: java.lang.IllegalStateException: The index [[logstash-http-2021.02.22/GCm01ePnSx6Ypg84hnKdxQ]] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.11.1.
        at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.checkSupportedVersion(MetadataIndexUpgradeService.java:108) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:82) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:236) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:219) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:150) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.node.Node.start(Node.java:827) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:310) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:396) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.11.1.jar:7.11.1]
        ... 6 more
[2021-02-24T16:38:29,805][INFO ][o.e.n.Node               ] [SELKS] stopping ...
[2021-02-24T16:38:29,838][INFO ][o.e.n.Node               ] [SELKS] stopped
[2021-02-24T16:38:29,839][INFO ][o.e.n.Node               ] [SELKS] closing ...
[2021-02-24T16:38:29,872][INFO ][o.e.n.Node               ] [SELKS] closed
[2021-02-24T16:38:29,876][INFO ][o.e.x.m.p.NativeController] [SELKS] Native controller process has stopped - no new native processes can be started
selks-user@SELKS:~$

[P-Sandusky]

Does the Elasticsearch migration assistant run as part of the upgrade scripts?

[pevma]

So that here is really the issue it seems - was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.11.1. You need to migrate form ES 5 to ES 6 to ES7.