search

Security-Onion-Solutions / securityonion

Security Onion is a free and open platform for threat hunting, enterprise security monitoring, and log management. It includes our own interfaces for alerting, dashboards, hunting, PCAP, detections, and case management. It also includes other tools such as osquery, CyberChef, Elasticsearch, Logstash, Kibana, Suricata, and Zeek.
https://securityonion.net
3.32k stars 515 forks source link

FIX: Intermittent soup errors causing soup to exit with failure message #13247

Closed jertel closed 2 months ago

jertel commented 5 months ago

  1. When shutting down salt-master, if it exits before the tail --pid ... command starts then the tail will error out and a message is echo'd to console that the salt-master did not shutdown within 30 seconds. That's not correct and should not be printed out in that situation. Ex:

    tail: invalid PID: ''
salt-master still running at 10:13:25.600201 after waiting 30s. We cannot kill due to systemd restart option.

  2. Soup needs to set +e before upgrading the elastic agent and then reset back to set -e after it returns. This will prevent soup from exiting if the first curl fails (it's in a retry loop). Ex:

    Checking to see if changes are needed.
parse error: Invalid numeric literal at line 1, column 7
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                             Dload  Upload   Total   Spent    Left  Speed
100    68  100    30  100    38   1917   2Kibana server is not ready yet428 --:--:-- --:--:-- --:--:--  4533
Checking if Elastic Agent update is necessary...
Executing command with retry support: curl --fail --retry 5 --retry-delay 15 -L 'https://repo.security((redacted)).net/file/so-repo/prod/2.4/elasticagent/elastic-agent_SO-8.10.4.tar.gz' --output '/nsm/elastic-fleet/artifacts/elastic-agent_SO-8.10.4.tar.gz'
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                             Dload  Upload   Total   Spent    Left  Speed
55 1389M   55  778M    0     0  10.2M      0  0:02:15  0:01:15  0:01:00  7597
curl: (92) HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)

    It's TBD where the parse error originated in this example.

m0duspwnens commented 2 months ago 
Executing command with retry support: curl --fail --retry 5 --retry-delay 15 -L 'https://repo.securityonion.net/file/so-repo/prod/2.4/elasticagent/elastic-agent_SO-8.14.3.tar.gz' --output '/nsm/elastic-fleet/artifacts/elastic-agent_SO-8.14.3.tar.gz'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:02:09 --:--:--     0
curl: (28) Failed to connect to repo.securityonion.net port 443: Connection timed out
Warning: Problem : timeout. Will retry in 15 seconds. 5 retries left.
  0     0    0     0    0     0      0      0 --:--:--  0:02:08 --:--:--     0
curl: (28) Failed to connect to repo.securityonion.net port 443: Connection timed out
Warning: Problem : timeout. Will retry in 15 seconds. 4 retries left.
100  851M  100  851M    0     0  38.2M      0  0:00:22  0:00:22 --:--:-- 43.6M
Results:  (0)
Executing command with retry support: curl --fail --retry 5 --retry-delay 15 -L 'https://repo.securityonion.net/file/so-repo/prod/2.4/elasticagent/elastic-agent_SO-8.14.3.md5' --output '/nsm/elastic-fleet/artifacts/elastic-agent_SO-8.14.3.md5'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    33  100    33    0     0     78      0 --:--:-- --:--:-- --:--:--    78
Results:  (0)
Source file and checksum are good.