Open dcbeckett opened 3 years ago
Thanks for trying out SELKS. Did you do the first time setup after the install - https://github.com/StamusNetworks/SELKS/wiki/First-time-setup ?
I thought it would make more sense upgrading everything on first powerup as all the packages were a year old before creating the configurations etc with the setup wizard to avoid having to upgrade all the older version configs during the uprade, but perhaps that logic was wrong, I didn't look at that read me file unfortunately as I was already familiar with SELK installs in the past.
Running it my way around
selks-upgrade_stamus - gives original error above and upgrade crashes
followed by
selks-first-time-setup_stamus
gives
Traceback (most recent call last):
File "bin/manage.py", line 10, in
FINISH of first time setup script - Tue 13 Apr 2021 09:19:52 AM EDT
Exited with FAILED Full log located at - /opt/selks/log/selks-first-time-setup_stamus.log Press enter to continue
I have now done it the way described in the readme
Fresh install from iso selks-first-time-setup_stamus - works fine (on fresh install packages) I did this earlier this morning and it's been analysing traffic fine etc
Note I did set FPC to None - disable packet capture during first time setup
I've now tried a selks-upgrade_stamus and I get the same error Upgrading Moloch..
{"cluster_name":"elasticsearch","status":"green","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":14,"active_shards":14,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":1,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_n(Reading database ... 239918 files and directories currently installed.) Preparing to unpack .../moloch_2.7.1-1_amd64.deb ... Unpacking moloch (2.7.1-1) over (2.2.3-1) ... Setting up moloch (2.7.1-1) ... READ /data/moloch/README.txt and RUN /data/moloch/bin/Configure It is STRONGLY recommended that you stop ALL Arkime captures and viewers before proceeding. Use 'db.pl http://localhost:9200 backup' to backup db first.
There is 1 elastic search data node, if you expect more please fix first before proceeding.
This is a fresh Arkime install Looks like Arkime wasn't installed, must do init at /data/moloch/db/db.pl line 2827.
Starting Moloch SELKS services..
Failed to start molochpcapread-selks.service: Unit molochpcapread-selks.service not found. Failed to start molochviewer-selks.service: Unit molochviewer-selks.service not found. selks-user@SELKS:~$
ok , thank you for the update. I think this switch is not handled correctly. there is a fix here - https://github.com/StamusNetworks/selks-scripts/pull/9/commits/8e5d29afa0d0d6494cbc86a7ca8ff16af924f028 that I will cook a pkg for.
Hello!
i have a similar problem. i do a fresh install everything is fine i do the update and everything is fine until i reboot. after reboot elastic wont start - where can i find some logs to show you?
it is running in a VM and i have a few snapshots. its always the same. whatever i try.
no problems after upgrade, runs for hours. Reboot -> no elastic
please help. thx
What exactly is the issue - Arkime is not working or something else too?
elasticsearch.service - Elasticsearch Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled) Active: failed (Result: timeout) since Mon 2021-04-19 13:55:04 CEST; 1h 18min ago Docs: https://www.elastic.co Process: 1022 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=killed, signal=TERM) Main PID: 1022 (code=killed, signal=TERM)
I try to start it manually: sudo -i service elasticsearch start No luck
Job for elasticsearch.service failed because a timeout was exceeded. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
Apr 19 15:17:45 SELKS logstash[657]: [2021-04-19T15:17:45,003][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://127.0.0.1:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Poo Apr 19 15:17:50 SELKS logstash[657]: [2021-04-19T15:17:50,007][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://127.0.0.1:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Poo Apr 19 15:17:50 SELKS logstash[657]: [2021-04-19T15:17:50,010][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://127.0.0.1:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Poo Apr 19 15:17:53 SELKS sudo[2017]: selks-user : TTY=pts/0 ; PWD=/home/selks-user ; USER=root ; COMMAND=/usr/bin/journalctl -xe Apr 19 15:17:53 SELKS sudo[2017]: pam_unix(sudo:session): session opened for user root by selks-user(uid=0)
Thx
Von: Peter Manev @.> Gesendet: Montag, 19. April 2021 13:34 An: StamusNetworks/SELKS @.> Cc: snakeaj @.>; Comment @.> Betreff: Re: [StamusNetworks/SELKS] Clean SELKS 6 install fails on upgrade due to moloch/arkime (#305)
What exactly is the issue - Arkime is not working or something else too?
- You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FStamusNetworks%2FSELKS%2Fissues%2F305%23issuecomment-822397234&data=04%7C01%7C%7Cbc3092ba41354626e31a08d90326f770%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637544288155623165%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=mWso6LXpbBcwobFbk3s4e85rI43xJ%2F3zJ31mz9K8vBE%3D&reserved=0, or unsubscribehttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FATXWUPVEO6IABDRPZE76RDDTJQII5ANCNFSM4227RQEQ&data=04%7C01%7C%7Cbc3092ba41354626e31a08d90326f770%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637544288155633164%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=BdyP4bQ7xCbGcxRH7FJ51JWwEbf7LLx0dSG%2FJsaGCUo%3D&reserved=0.
Elasticsearch: Job for elasticsearch.service failed - Stack Overflowhttps://stackoverflow.com/questions/58656747/elasticsearch-job-for-elasticsearch-service-failed
Reconfigured Posts/Hosts and RAM - now it works again.
Von: Peter Manev @.> Gesendet: Montag, 19. April 2021 13:34 An: StamusNetworks/SELKS @.> Cc: snakeaj @.>; Comment @.> Betreff: Re: [StamusNetworks/SELKS] Clean SELKS 6 install fails on upgrade due to moloch/arkime (#305)
What exactly is the issue - Arkime is not working or something else too?
- You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FStamusNetworks%2FSELKS%2Fissues%2F305%23issuecomment-822397234&data=04%7C01%7C%7Cbc3092ba41354626e31a08d90326f770%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637544288155623165%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=mWso6LXpbBcwobFbk3s4e85rI43xJ%2F3zJ31mz9K8vBE%3D&reserved=0, or unsubscribehttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FATXWUPVEO6IABDRPZE76RDDTJQII5ANCNFSM4227RQEQ&data=04%7C01%7C%7Cbc3092ba41354626e31a08d90326f770%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637544288155633164%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=BdyP4bQ7xCbGcxRH7FJ51JWwEbf7LLx0dSG%2FJsaGCUo%3D&reserved=0.
It seems Es did not start.
What is the output of the last 50 lines in /var/log/elasticsearch/elasticsearch.log
?
[2021-04-20T06:54:10,482][INFO ][o.e.c.m.MetadataMappingService] [SELKS] [logstash-http-2021.04.20/rkgT5-zMRpCXfIWlCaFB0Q] update_mapping [_doc] [2021-04-20T06:54:10,491][INFO ][o.e.c.m.MetadataMappingService] [SELKS] [logstash-fileinfo-2021.04.20/PbcqiZ9YTiy6TIWCqZRHfg] update_mapping [_doc] [2021-04-20T06:54:10,606][INFO ][o.e.c.m.MetadataMappingService] [SELKS] [logstash-http-2021.04.20/rkgT5-zMRpCXfIWlCaFB0Q] update_mapping [_doc] [2021-04-20T08:09:20,143][INFO ][o.e.c.m.MetadataMappingService] [SELKS] [logstash-http-2021.04.20/rkgT5-zMRpCXfIWlCaFB0Q] update_mapping [_doc] [2021-04-20T08:18:55,447][INFO ][o.e.n.Node ] [SELKS] stopping ... [2021-04-20T08:19:00,860][INFO ][o.e.x.w.WatcherService ] [SELKS] stopping watch service, reason [shutdown initiated] [2021-04-20T08:19:01,376][INFO ][o.e.x.w.WatcherLifeCycleService] [SELKS] watcher has stopped and shutdown [2021-04-20T08:19:01,969][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [SELKS] [controller/2147] @.*** ML controller exiting [2021-04-20T08:19:02,040][INFO ][o.e.x.m.p.NativeController] [SELKS] Native controller process has stopped - no new native processes can be started
[2021-04-20T08:19:24,331][INFO ][o.e.n.Node ] [SELKS] stopped [2021-04-20T08:19:24,370][INFO ][o.e.n.Node ] [SELKS] closing ... [2021-04-20T08:19:29,923][INFO ][o.e.n.Node ] [SELKS] closed
After 5-10 minutes i can manually start it using "sudo -i service elasticsearch start"
Log:
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.kibana_task_manager_7.12.0_001][0]]. Consider using allow_partial_search_results
setting to bypass this error.
at org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:211) ~[elasticsearch-7.12.0.jar:7.12.0]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:397) [elasticsearch-7.12.0.jar:7.12.0]
... 81 more
[2021-04-20T08:50:00,620][INFO ][o.e.t.LoggingTaskListener] [SELKS] 179 finished with response BulkByScrollResponse[took=715.9ms,timed_out=false,sliceId=null,updated=10,created=0,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],sear$
[2021-04-20T08:50:02,167][INFO ][o.e.t.LoggingTaskListener] [SELKS] 180 finished with response BulkByScrollResponse[took=2.3s,timed_out=false,sliceId=null,updated=460,created=0,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search$
[2021-04-20T08:50:05,686][INFO ][o.e.c.r.a.AllocationService] [SELKS] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[logstash-alert-2021.04.15][0]]]).
[2021-04-20T08:50:36,088][INFO ][o.e.c.m.MetadataMappingService] [SELKS] [logstash-http-2021.04.20/rkgT5-zMRpCXfIWlCaFB0Q] update_mapping [_doc]
[2021-04-20T08:50:36,230][INFO ][o.e.c.m.MetadataMappingService] [SELKS] [logstash-fileinfo-2021.04.20/PbcqiZ9YTiy6TIWCqZRHfg] update_mapping [_doc]
[2021-04-20T08:50:36,277][INFO ][o.e.c.m.MetadataMappingService] [SELKS] [logstash-alert-2021.04.20/f0sNiA8XTROZ13zsAnVZjQ] update_mapping [_doc]
Von: Peter Manev @.> Gesendet: Montag, 19. April 2021 20:58 An: StamusNetworks/SELKS @.> Cc: snakeaj @.>; Comment @.> Betreff: Re: [StamusNetworks/SELKS] Clean SELKS 6 install fails on upgrade due to moloch/arkime (#305)
It seems Es did not start. What is the output of the last 50 lines in /var/log/elasticsearch/elasticsearch.log ?
- You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FStamusNetworks%2FSELKS%2Fissues%2F305%23issuecomment-822700984&data=04%7C01%7C%7C9f3ed6e779914182b08c08d903650304%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637544554636997106%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=e8uvBZ38fVv%2FTO%2B6nPuzEDhwyRHe1qo4KqBdDK%2FkvVI%3D&reserved=0, or unsubscribehttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FATXWUPXMI6BQOTYRAG3YQG3TJR4KNANCNFSM4227RQEQ&data=04%7C01%7C%7C9f3ed6e779914182b08c08d903650304%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637544554636997106%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=vPZ9MEf9nd66XqQlLr4t7ikQtsENMCMh6WdOCHXXfUw%3D&reserved=0.
Did you do the first time install and it finished properly/successfully ? ( https://github.com/StamusNetworks/SELKS/wiki/First-time-setup#first-time-setup )
Yes It is running fine now but I have to start elastic manally afterwards
Von: Peter Manev @.> Gesendet: Dienstag, 20. April 2021 16:27 An: StamusNetworks/SELKS @.> Cc: snakeaj @.>; Comment @.> Betreff: Re: [StamusNetworks/SELKS] Clean SELKS 6 install fails on upgrade due to moloch/arkime (#305)
When you reboot - does the problem persist, could be just service start delay.
Yes. When i reboot elastic wont start automaticly. I need to start it manually after a few minutes. Somtimes i have to do that 2-3 times.
Von: Peter Manev @.> Gesendet: Donnerstag, 22. April 2021 08:06 An: StamusNetworks/SELKS @.> Cc: snakeaj @.>; Comment @.> Betreff: Re: [StamusNetworks/SELKS] Clean SELKS 6 install fails on upgrade due to moloch/arkime (#305)
When you reboot - does the problem persist, could be just service start delay.
- You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FStamusNetworks%2FSELKS%2Fissues%2F305%23issuecomment-824565004&data=04%7C01%7C%7Ccf9241ab67eb44586a8508d90554a7a4%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637546683407938312%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=J7HVKwuqn2WM99LgftsVenWEHfDAV9pd6AgJQCptigg%3D&reserved=0, or unsubscribehttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FATXWUPSLEOI2DXOLO2NM3T3TJ64DFANCNFSM4227RQEQ&data=04%7C01%7C%7Ccf9241ab67eb44586a8508d90554a7a4%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637546683407938312%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=JJw1aJjnJup7vts6Ij8KM%2BhU2bIuei7pogUfpot6%2BDQ%3D&reserved=0.
Hi all! I have same issue, multiple times did fresh install and upgrade, always same problem, Elastic didnt start after reboot and have 2 Unassigned shards. and VM cant reboot long time, w8ing logstash process terminating
Also, in alert event from Kabana ill try move to FPC in Arkime, a have an error:
Also in Arkime no data at all.
For the VM not rebooting in a long time you might be hitting this maybe - https://github.com/StamusNetworks/SELKS/wiki/A-stop-job-running-for-logstash-takes-too-long-on-shutdown
Data in Arkime takes time to populate due to rotations and needs constant traffic flow - please have a look/confirm here for more information - https://github.com/StamusNetworks/SELKS/wiki/First-time-setup#full-packet-capture-fpc
With regards to ES is the version 7.12? There is no customization done on the elasticsearch part - just regular package install. Could be something similar to running https://www.elastic.co/guide/en/elasticsearch/reference/current/starting-elasticsearch.html#start-es-deb-init
Thanks for quick answer!
Your link help me to fix reboot.
/etc/suricata/selks6-addin.yaml same that in the manual, in first time configuration i check FPC_Retain, then i shuld 'dir:' to /data/moloch/raw/ ?
With ES (i use latest) link didnt helps me:
Looks like i fix problem with ES, just go thru this manual, who have same problem - https://sleeplessbeastie.eu/2020/02/29/how-to-prevent-systemd-service-start-operation-from-timing-out/
But still have problem with 2 shards, yellow indexes:
and one more problem with dashboards:
The FPC_Retain
is automatically taken care of in the setup script, no need to do anything.
Yellow shards in general do not mean ES is not functioning , not sure where they are from (on the shared screenshot) - https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html
Where i can find django config? Or how i can fix this error?
Forbidden (403)
CSRF verification failed. Request aborted.
You are seeing this message because this HTTPS site requires a 'Referer header' to be sent by your Web browser, but none was sent. This header is required for security reasons, to ensure that your browser is not being hijacked by third parties.
If you have configured your browser to disable 'Referer' headers, please re-enable them, at least for this site, or for HTTPS connections, or for 'same-origin' requests.
If you are using the <meta name="referrer" content="no-referrer"> tag or including the 'Referrer-Policy: no-referrer' header, please remove them. The CSRF protection requires the 'Referer' header to do strict referer checking. If you're concerned about privacy, use alternatives like <a rel="noreferrer" ...> for links to third-party sites.
Can you please describe what is the background of this error or what are you trying to do ?
Sure, that error appear when i open SELKS web via fqdn, redirect before from router
Did you adjust the hostname settings in the OS and double check the /etc/scirius/local_settings.py
- that it looks ok?
Did you adjust the hostname settings in the OS and double check the
/etc/scirius/local_settings.py
- that it looks ok?
Yes, /etc/scirius/local_settings.py
default settings, i can use SELKS via IP, but if i'll try open SELKS via ids.domain.com can try auth and then have error.
Maybe i must add fqdn in django settings?
Can you ping for example the fqdn form another place on the network ? (just a sanity check)
Can you ping for example the fqdn form another place on the network ? (just a sanity check)
Sure, all ok, as i say, i see auth page and after enter login\pass, have error
Have you installed something additional, sec tool or similar. It should work as described. It seems something else is stopping it.
Clean VM install of SELKS desktop from the public iso
I login onto the new system and run sudo selks-upgrade_stamus
Whilst accepting all default options. It fails when it gets to moloch due to Arkime not being properly initialised? It Exits the upgrade at this step and leaves the system in a bad state. I've pasted the last few lines of the upgrade below
`Processing triggers for initramfs-tools (0.133+deb10u1) ... update-initramfs: Generating /boot/initrd.img-4.19.0-16-amd64 Processing triggers for systemd (241-7~deb10u7) ... Processing triggers for ca-certificates (20200601~deb10u2) ... Updating certificates in /etc/ssl/certs... 0 added, 0 removed; done. Running hooks in /etc/ca-certificates/update.d...
done. done. scirius: stopped scirius: started
Upgrading Moloch..
{"cluster_name":"elasticsearch","status":"green","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":6,"active_shards":6,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,(Reading database ... 258667 files and directories currently installed.)ing_in_queue_millis":0,"active_shards_percent_as_number":100.0}(Reading database ... Preparing to unpack .../moloch_2.7.1-1_amd64.deb ... Unpacking moloch (2.7.1-1) over (2.2.3-1) ... Setting up moloch (2.7.1-1) ... READ /data/moloch/README.txt and RUN /data/moloch/bin/Configure It is STRONGLY recommended that you stop ALL Arkime captures and viewers before proceeding. Use 'db.pl http://localhost:9200 backup' to backup db first.
There is 1 elastic search data node, if you expect more please fix first before proceeding.
This is a fresh Arkime install Looks like Arkime wasn't installed, must do init at /data/moloch/db/db.pl line 2827.
Starting Moloch SELKS services..
Failed to start molochpcapread-selks.service: Unit molochpcapread-selks.service not found. Failed to start molochviewer-selks.service: Unit molochviewer-selks.service not found.`
It fails if I re-run the upgrade process or if I run first time setup it fails, the only solution was to wipe the system and re-install from iso.