Closed Linn1 closed 2 years ago
Linn1
commented
2 years ago I want to how to update the rules of suricata. I tried to use suricata-update but it seems not work.
So, it is Scirius or suricata-update manage the rules of suricata?
I know the ruleset web pages are controlled by Scirius.So I run suricata-update and the ruleset web pages are no change.
Linn1
commented
2 years ago I tried update the rule on ruleset web page, but failed. The window shows the error:
Error during update: Can not fetch data: [Errno 3] Source "ETOpen Ruleset" update failed: Connection error 'HTTPSConnectionPool(host='rules.emergingthreats.net', port=443): Max retries exceeded with url: /open/suricata-5.0/version.txt (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f4a4777fad0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))' Source "SSLBL abuse.ch" update failed: Connection error 'HTTPSConnectionPool(host='sslbl.abuse.ch', port=443): Max retries exceeded with url: /blacklist/sslblacklist_tls_cert.rules (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f4a47780b50>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))' Source "Suricata Traffic ID ruleset" update failed: Connection error 'HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /jasonish/suricata-trafficid/master/rules/traffic-id.rules (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f4a4778d690>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))'
So, now I lost my controll of the suri's rules.
Linn1
commented
2 years ago I tried to use scirius to update the rules. I run cd /usr/share/python/scirius && . /bin/activate && python bin/manage.py updatesuricata . But one rule source failed to update and the other two succeed.
Unable to update ruleset for suricata "SELKS": [Errno 1] Source "Suricata Traffic ID ruleset" update failed:
Connection error 'HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /jasonish/suricata-trafficid/master/rules/traffic-id.rules (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 503 Service Unavailable',)))'
Successfully pushed ruleset to suricata "SELKS"
The suricata.yaml says the rule-file is suricata.rules . I run suricata-update and the suricata.rules is updated.
I don't understand how the scirius manage the rules of suricata?
Linn1
commented
2 years ago I checked theselks6-addin.yaml, it says the rule-file is scirius.rules. I think it is Scirius managing the rules. But when I delete the alerts through the ruleset web pages. It failed. The error log : unknown error.
So, how can I disable some rules and delete these wrong alerts?
pevma
commented
2 years ago In Scirius, inside Management, you can select the Home tab, then select the signature you want to disable. After that on the left hand side (under Actions) you can select disable.
To enforce it go to the Suricata tab, and on the left hand side panel, select ruleset actions click update and push (all actions).
Linn1
commented
2 years ago I click update and push, the error log appears on the page.The error log is as shown last time.
I tried update the rule on ruleset web page, but failed. The window shows the error:
Error during update: Can not fetch data: [Errno 3] Source "ETOpen Ruleset" update failed: Connection error 'HTTPSConnectionPool(host='rules.emergingthreats.net', port=443): Max retries exceeded with url: /open/suricata-5.0/version.txt (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f4a4777fad0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))' Source "SSLBL abuse.ch" update failed: Connection error 'HTTPSConnectionPool(host='sslbl.abuse.ch', port=443): Max retries exceeded with url: /blacklist/sslblacklist_tls_cert.rules (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f4a47780b50>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))' Source "Suricata Traffic ID ruleset" update failed: Connection error 'HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /jasonish/suricata-trafficid/master/rules/traffic-id.rules (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f4a4778d690>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))'So, now I lost my controll of the suri's rules.
pevma
commented
2 years ago Maybe you can check if you have internet access /dns resolution from the box?
Linn1
commented
2 years ago I use proxy to access the internet. And just http、https、ftp can access the internet.
By the way, the elasticsearch did't work! I run curl http://localhost:9200, the response is "connection refused".
pevma
commented
2 years ago If you are using a proxy did you try specifying under System settings in Scirius management ? Click on “Use a proxy” option under Systems settings edition.
If this is SELKS ISO install - did the first time setup complete successfully ?
-- Regards, Peter Manev
On 24 Dec 2021, at 10:05, Linn1 @.***> wrote:
I use proxy to access the internet. And just http、https、ftp can access the internet. By the way, the elasticsearch did't work! I run curl http://localhost:9200, the response is "connection refused".
— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you commented.
Linn1
commented
2 years ago I run cd /usr/share/python/scirius/ && . bin/activate && python bin/manage.py updatesuricata && deactivate as root.
And after specifying under System settings in Scirius management, I update the sources on the web pages then it wored except some small "permission denied" errors !
I run curl http://localhost:9200 and the response is "connection refused".
I tried to run selks-first-set-up_stamus,but failed. When "Setting up Scirius/Moloch proxy user", appeas a error:
Elastic search error { Error: [circuit_breaking_exception] [parent] Data too large, data for [
pevma
commented
2 years ago What is the return of selks-health-status_stamus ?
-- Regards, Peter Manev
On 27 Dec 2021, at 10:20, Linn1 @.***> wrote:
I run cd /usr/share/python/scirius/ && . bin/activate && python bin/manage.py updatesuricata && deactivate as root. And after specifying under System settings in Scirius management, I update the sources on the web pages then it wored except some small "permission denied" errors ! I run curl http://localhost:9200 and the response is "connection refused". I tried to run selks-first-set-up_stamus,but failed. When "Setting up Scirius/Moloch proxy user", appeas a error:
Elastic search error { Error: [circuit_breaking_exception] [parent] Data too large, data for [
] would be [1025170800/977.6mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1025170152/977.6mb], new bytes reserved: [648/648b], usages [request=164232/160.3kb, fielddata=10326747/9.8mb, in_flight_requests=19676362/18.7mb, accounting=22255224/21.2mb], with { bytes_wanted=1025170800 & bytes_limit=1020054732 & durability="PERMANENT" } at respond (/data/moloch/node_modules/elasticsearch/src/lib/transport.js:308:15) at checkRespForFailure (/data/moloch/node_modules/elasticsearch/src/lib/transport.js:267:7) at HttpConnector. (/data/moloch/node_modules/elasticsearch/src/lib/connectors/http.js:166:7) at IncomingMessage.wrapper (/data/moloch/node_modules/lodash/lodash.js:4929:19) at IncomingMessage.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19) status: 429, displayName: 'TooManyRequests', message: '[circuit_breaking_exception] [parent] Data too large, data for [ ] would be [1025170800/977.6mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1025170152/977.6mb], new bytes reserved: [648/648b], usages [request=164232/160.3kb, fielddata=10326747/9.8mb, in_flight_requests=19676362/18.7mb, accounting=22255224/21.2mb], with { bytes_wanted=1025170800 & bytes_limit=1020054732 & durability="PERMANENT" }', path: '/users/user/moloch', query: { refresh: 'true', timeout: '10m' }, body: '{"userId":"moloch","userName":"moloch","passStore":"2780034eab8e28307400c5536cdabd3eff8edfc96a60a16272336af774a73cf0cc784297f74c2591204389d5fd74a8c1","enabled":true,"webEnabled":true,"headerAuthEnabled":true,"emailSearch":false,"createEnabled":true,"removeEnabled":false,"packetSearch":false,"welcomeMsgNum":0,"settings":{}}', statusCode: 429, response: '{"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [ ] would be [1025170800/977.6mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1025170152/977.6mb], new bytes reserved: [648/648b], usages [request=164232/160.3kb, fielddata=10326747/9.8mb, in_flight_requests=19676362/18.7mb, accounting=22255224/21.2mb]","bytes_wanted":1025170800,"bytes_limit":1020054732,"durability":"PERMANENT"}],"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [ ] would be [1025170800/977.6mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1025170152/977.6mb], new bytes reserved: [648/648b], usages [request=164232/160.3kb, fielddata=10326747/9.8mb, in_flight_requests=19676362/18.7mb, accounting=22255224/21.2mb]","bytes_wanted":1025170800,"bytes_limit":1020054732,"durability":"PERMANENT"},"status":429}', toString: [Function], toJSON: [Function] } Traceback (most recent call last): File "bin/manage.py", line 10, in execute_from_command_line(sys.argv) File "/usr/share/python/scirius/local/lib/python2.7/site-packages/django/core/management/init.py", line 364, in execute_from_command_line utility.execute() File "/usr/share/python/scirius/local/lib/python2.7/site-packages/django/core/management/init.py", line 356, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/usr/share/python/scirius/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv self.execute(*args, cmd_options) File "/usr/share/python/scirius/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute output = self.handle(*args, *options) File "/usr/share/python/scirius/local/lib/python2.7/site-packages/rules/management/commands/kibana_reset.py", line 38, in handle self.kibana_reset() File "/usr/share/python/scirius/local/lib/python2.7/site-packages/rules/es_data.py", line 1987, in kibana_reset self._kibana_inject(_type, _file) File "/usr/share/python/scirius/local/lib/python2.7/site-packages/rules/es_data.py", line 1872, in _kibana_inject self.client.delete(index='.kibana', doc_type=doc_type, id=name, refresh=True) File "/usr/share/python/scirius/local/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped return func(args, params=params, kwargs) File "/usr/share/python/scirius/local/lib/python2.7/site-packages/elasticsearch/client/init.py", line 1444, in delete "DELETE", _make_path(index, doc_type, id), params=params File "/usr/share/python/scirius/local/lib/python2.7/site-packages/elasticsearch/transport.py", line 358, in perform_request timeout=timeout, File "/usr/share/python/scirius/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 231, in perform_request self._raise_error(response.status, raw_data) File "/usr/share/python/scirius/local/lib/python2.7/site-packages/elasticsearch/connection/base.py", line 230, in _raise_error status_code, error_message, additional_info elasticsearch.exceptions.RequestError: RequestError(400, u'illegal_argument_exception', u'Rejecting mapping update to [.kibana_4] as the final mapping would have more than 1 type: [_doc, doc]') Dashboards loading set up job failed...Exiting... Exited with ERROR
— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you commented.
Linn1
commented
2 years ago I run selks-health-check_stamus, here is the return:
root@selks:~# selks-health-check_stamus
● suricata.service - LSB: Next Generation IDS/IPS
Loaded: loaded (/etc/init.d/suricata; generated)
Active: active (running) since Mon 2021-12-27 16:52:27 CST; 15h ago
Docs: man:systemd-sysv-generator(8)
Tasks: 54 (limit: 9830)
Memory: 18.2G
CGroup: /system.slice/suricata.service
└─13766 /usr/bin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid --af-packet -D -v --user=logsta…
12月 27 16:52:27 selks systemd[1]: Starting LSB: Next Generation IDS/IPS... 12月 27 16:52:27 selks suricata[13755]: Starting suricata in IDS (af-packet) mode... done. 12月 27 16:52:27 selks systemd[1]: Started LSB: Next Generation IDS/IPS. ● elasticsearch.service - Elasticsearch Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2021-12-27 18:22:22 CST; 14h ago Docs: https://www.elastic.co Main PID: 25320 (java) Tasks: 382 (limit: 9830) Memory: 6.3G CGroup: /system.slice/elasticsearch.service ├─25320 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.ne… └─25551 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
12月 27 18:21:58 selks systemd[1]: Starting Elasticsearch... 12月 27 18:22:22 selks systemd[1]: Started Elasticsearch. ● logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2021-12-24 23:02:29 CST; 3 days ago Main PID: 56800 (java) Tasks: 112 (limit: 9830) Memory: 1.6G CGroup: /system.slice/logstash.service └─56800 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccup…
12月 28 08:21:23 selks logstash[56800]: [2021-12-28T08:21:23,365][INFO ][logstash.outputs.elasticsearch][main][e55f734d663b7fb7ca2… 12月 28 08:21:23 selks logstash[56800]: [2021-12-28T08:21:23,365][INFO ][logstash.outputs.elasticsearch][main][e55f734d663b7fb7ca2… 12月 28 08:21:23 selks logstash[56800]: [2021-12-28T08:21:23,366][INFO ][logstash.outputs.elasticsearch][main][e55f734d663b7fb7ca2… 12月 28 08:21:23 selks logstash[56800]: [2021-12-28T08:21:23,366][INFO ][logstash.outputs.elasticsearch][main][e55f734d663b7fb7ca2… 12月 28 08:21:23 selks logstash[56800]: [2021-12-28T08:21:23,367][INFO ][logstash.outputs.elasticsearch][main][e55f734d663b7fb7ca2… 12月 28 08:21:23 selks logstash[56800]: [2021-12-28T08:21:23,367][INFO ][logstash.outputs.elasticsearch][main][e55f734d663b7fb7ca2… 12月 28 08:21:23 selks logstash[56800]: [2021-12-28T08:21:23,368][INFO ][logstash.outputs.elasticsearch][main][e55f734d663b7fb7ca2… 12月 28 08:21:23 selks logstash[56800]: [2021-12-28T08:21:23,368][INFO ][logstash.outputs.elasticsearch][main][e55f734d663b7fb7ca2… 12月 28 08:21:23 selks logstash[56800]: [2021-12-28T08:21:23,369][INFO ][logstash.outputs.elasticsearch][main][e55f734d663b7fb7ca2… 12月 28 08:21:23 selks logstash[56800]: [2021-12-28T08:21:23,372][INFO ][logstash.outputs.elasticsearch][main][e55f734d66…ount=>31} Hint: Some lines were ellipsized, use -l to show in full. ● kibana.service - Kibana Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2021-12-20 16:00:10 CST; 1 weeks 0 days ago Main PID: 783 (node) Tasks: 11 (limit: 9830) Memory: 293.0M CGroup: /system.slice/kibana.service └─783 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml
12月 28 08:30:31 selks kibana[783]: {"type":"log","@timestamp":"2021-12-28T00:30:31Z","tags":["status","plugin:grokdebugger@7.7.0"… 12月 28 08:30:31 selks kibana[783]: {"type":"log","@timestamp":"2021-12-28T00:30:31Z","tags":["status","plugin:logstash@7.7.0","in… 12月 28 08:30:31 selks kibana[783]: {"type":"log","@timestamp":"2021-12-28T00:30:31Z","tags":["status","plugin:beats_management@7.… 12月 28 08:30:31 selks kibana[783]: {"type":"log","@timestamp":"2021-12-28T00:30:31Z","tags":["status","plugin:index_lifecycle_man… 12月 28 08:30:31 selks kibana[783]: {"type":"log","@timestamp":"2021-12-28T00:30:31Z","tags":["status","plugin:rollup@7.7.0","info… 12月 28 08:30:31 selks kibana[783]: {"type":"log","@timestamp":"2021-12-28T00:30:31Z","tags":["status","plugin:cross_cluster_repli… 12月 28 08:30:31 selks kibana[783]: {"type":"log","@timestamp":"2021-12-28T00:30:31Z","tags":["status","plugin:reporting@7.7.0","i… 12月 28 08:30:31 selks kibana[783]: {"type":"log","@timestamp":"2021-12-28T00:30:31Z","tags":["status","plugin:security@7.7.0","in… 12月 28 08:30:31 selks kibana[783]: {"type":"log","@timestamp":"2021-12-28T00:30:31Z","tags":["status","plugin:spaces@7.7.0","info… 12月 28 08:35:03 selks kibana[783]: {"type":"log","@timestamp":"2021-12-28T00:35:03Z","tags":["error","reporting","esqueue","queue… Hint: Some lines were ellipsized, use -l to show in full. ● evebox.service - EveBox Server Loaded: loaded (/lib/systemd/system/evebox.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2021-12-20 16:00:10 CST; 1 weeks 0 days ago Main PID: 781 (evebox) Tasks: 51 (limit: 9830) Memory: 90.3M CGroup: /system.slice/evebox.service └─781 /usr/bin/evebox server
12月 24 11:26:16 selks evebox[781]: 2021-12-24 11:26:16 (anonymous.go:64)
12月 24 23:02:29 selks systemd[1]: Started Moloch Viewer. 12月 27 16:21:31 selks systemd[1]: molochviewer-selks.service: Current command vanished from the unit file, execution of … resumed. Hint: Some lines were ellipsized, use -l to show in full. ● molochpcapread-selks.service - Moloch Pcap Read Loaded: loaded (/etc/systemd/system/molochpcapread-selks.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2021-12-24 23:02:29 CST; 3 days ago Main PID: 56805 (sh) Tasks: 5 (limit: 9830) Memory: 1.1G CGroup: /system.slice/molochpcapread-selks.service ├─56805 /bin/sh -c /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m -s -R /data/nsm/ >> /data/moloch/l… └─56806 /data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini -m -s -R /data/nsm/
12月 24 23:02:29 selks systemd[1]: Started Moloch Pcap Read. 12月 27 16:21:31 selks systemd[1]: molochpcapread-selks.service: Current command vanished from the unit file, execution o… resumed. Hint: Some lines were ellipsized, use -l to show in full. scirius RUNNING pid 1488, uptime 7 days, 16:35:59 ii elasticsearch 7.7.0 amd64 Distributed RESTful search engine built for the cloud ii elasticsearch-curator 5.8.1 amd64 Have indices in Elasticsearch? This is the tool for you!\n\nLike a museum curator manages the exhibits and collections on display, \nElasticsearch Curator helps you curate, or manage your indices. ii evebox 1:0.11.1 amd64 no description given ii kibana 7.7.0 amd64 Explore and visualize your Elasticsearch data ii kibana-dashboards-stamus 2020042401 amd64 Kibana 6 dashboard templates. ii logstash 1:7.7.0-1 all An extensible logging pipeline ii moloch 2.2.3-1 amd64 Moloch Full Packet System ii scirius 3.5.0-3 amd64 Django application to manage Suricata ruleset ii suricata 1:2020050401-0stamus0 amd64 Suricata open source multi-thread IDS/IPS/NSM system. 文件系统 类型 容量 已用 可用 已用% 挂载点 udev devtmpfs 16G 0 16G 0% /dev tmpfs tmpfs 3.2G 283M 2.9G 9% /run /dev/sda1 ext4 518G 177G 315G 37% / tmpfs tmpfs 16G 0 16G 0% /dev/shm tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs tmpfs 16G 0 16G 0% /sys/fs/cgroup tmpfs tmpfs 3.2G 8.0K 3.2G 1% /run/user/0 tmpfs tmpfs 3.2G 4.0K 3.2G 1% /run/user/115 tmpfs tmpfs 3.2G 0 3.2G 0% /run/user/1000
Linn1
commented
2 years ago I seem to know the reason why failed to curl http://localhost:9200.
Because I use the proxy then when I "curl" something ,it will send request to the proxy instead of localhost.
So, I want to use the proxy to regular update rule. But it will lead to the failure of "delete-old-log.sh".
How can I fix this?
Linn1
commented
2 years ago The web page said the elasticsearch status is yellow!
I want to try to curl http://localhost:9200 without using proxy.
pevma
commented
2 years ago Did you set up the proxy in the GUI as mentioned here https://github.com/StamusNetworks/SELKS/issues/370#issuecomment-1000769160 ?
pevma
commented
2 years ago For delete-old-log.sh you can maybe edit the script with the correct xget request?
Linn1
commented
2 years ago Did you set up the proxy in the GUI as mentioned here #370 (comment) ? Yes. Not only in the GUI, also in the nsswitch.conf. I set up the http/https/ftp proxy in the
/etc/profile. If I set up the proxy in the GUI , don't I need to set it in the configure file?For
delete-old-log.shyou can maybe edit the script with the correct xget request? I set a cron job: rundelete-old-log.shevery night and write the output todelete.log. I just checked the log file and found that it work well. But when I runcurl http://localhost:9200orsh delete-old-log.sh, the response is connection refused.
Linn1
commented
2 years ago I try to use "evebox" to search events with some key words in msg. But it doesn't work. I don't know if it's my input or elasticsearch. The web page said the elasticsearch status is yellow because the unassigned shards is 5! How to make it turn to green?
I tried a right input on the "evebox", but it didn't work! I am sure that it is the es! I can't search any event...T^T
pevma
commented
2 years ago Did you set up the proxy in the GUI as mentioned here #370 (comment) ? Yes. Not only in the GUI, also in the nsswitch.conf. I set up the http/https/ftp proxy in the
/etc/profile. If I set up the proxy in the GUI , don't I need to set it in the configure file?
no you should be good, just the gui should be enough for getting intel. How did you set it up - you can share it with private email if you will or can hit me up in our discord server channel https://github.com/StamusNetworks/SELKS/wiki/Getting-Help
For
delete-old-log.shyou can maybe edit the script with the correct xget request? I set a cron job: rundelete-old-log.shevery night and write the output todelete.log. I just checked the log file and found that it work well. But when I runcurl http://localhost:9200orsh delete-old-log.sh, the response is connection refused.
This can be related to the proxy. You can setup retention through the ES ILM too - https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html
As the title say, I want to disable suri's some rules . I tried the
disable.confbut it didn't work. So, how can I disable these rules?