search

StamusNetworks / scirius

Scirius is a web application for Suricata ruleset management and threat hunting.
GNU General Public License v3.0
635 stars 150 forks source link

USE_KIBANA/ELASTICSEARCH=0 ignored, still tries to connect #269

Open brsolomon-deloitte opened 2 years ago

brsolomon-deloitte commented 2 years ago

Using local docker image scirius:test-1 built from scirius at commit 0b8fd6d240b54602679e07b550ff3e2c4dc459b0 fails in trying to connect to Elasticsearch, even though local_settings.py defaults to USE_ELASTICSEARCH=False and USE_KIBANA=False.

It appears that docker/scirius/bin/reset_dashboards.sh doesn't respect these variables whatsoever and calls python manage.py kibana_reset indiscriminately:

https://github.com/StamusNetworks/scirius/blob/0b8fd6d240b54602679e07b550ff3e2c4dc459b0/docker/scirius/bin/reset_dashboards.sh#L25

$ docker container run -e SECRET_KEY=$(openssl rand -hex 16) --expose 8000 scirius:test-1
Migrations for 'accounts':
  accounts/migrations/0007_auto_20220421_1743.py
    - Alter field timezone on sciriususer
Operations to perform:
  Apply all migrations: accounts, auth, authtoken, contenttypes, rules, sessions, suricata
Running migrations:
  Applying rules.0001_initial... OK
  Applying rules.0002_auto_20141207_1824... OK
  Applying rules.0003_auto_20141210_1421... OK
  Applying rules.0004_auto_20141210_1525... OK
  Applying rules.0005_auto_20141210_1734... OK
  Applying rules.0006_auto_20141210_1846... OK
  Applying rules.0007_auto_20141210_2037... OK
  Applying rules.0008_auto_20141210_2057... OK
  Applying rules.0009_auto_20141214_1203... OK
  Applying rules.0010_auto_20141222_1209... OK
  Applying rules.0011_auto_20141222_1304... OK
  Applying rules.0012_auto_20141222_1306... OK
  Applying rules.0013_auto_20141229_1527... OK
  Applying rules.0014_auto_20141229_1528... OK
  Applying rules.0015_auto_20141229_1610... OK
  Applying rules.0016_auto_20141229_1629... OK
  Applying rules.0017_auto_20141229_1716... OK
  Applying rules.0018_auto_20141229_1716... OK
  Applying rules.0019_auto_20141229_1719... OK
  Applying rules.0020_auto_20141229_1852... OK
  Applying rules.0021_auto_20141229_1853... OK
  Applying rules.0022_auto_20141229_1858... OK
  Applying rules.0023_auto_20141229_1903... OK
  Applying rules.0024_auto_20141229_2204... OK
  Applying rules.0025_auto_20141230_0812... OK
  Applying rules.0026_auto_20141231_0948... OK
  Applying rules.0027_auto_20141231_0953... OK
  Applying rules.0028_auto_20150101_2305... OK
  Applying rules.0029_auto_20150102_1212... OK
  Applying rules.0030_auto_20150103_1136... OK
  Applying rules.0031_auto_20150103_1138... OK
  Applying rules.0032_auto_20150103_1255... OK
  Applying rules.0033_auto_20150109_2319... OK
  Applying rules.0034_auto_20150111_2200... OK
  Applying rules.0035_auto_20150202_0937... OK
  Applying rules.0036_auto_20150203_1421... OK
  Applying rules.0037_auto_20150407_2040... OK
  Applying rules.0038_auto_20150516_0912... OK
  Applying rules.0039_auto_20150805_1737... OK
  Applying rules.0040_ruleset_rules_count... OK
  Applying rules.0041_source_authkey... OK
  Applying rules.0042_rule_state_in_source... OK
  Applying rules.0043_threshold... OK
  Applying rules.0044_flowbit_type... OK
  Applying rules.0045_auto_20160405_1300... OK
  Applying rules.0046_source_cert_verif... OK
  Applying rules.0047_proxy_validation... OK
  Applying rules.0048_custom_es... OK
  Applying rules.0049_auto_20161121_2342... OK
  Applying contenttypes.0001_initial... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying rules.0050_auto_20161128_2110... OK
  Applying rules.0051_auto_20161207_0758... OK
  Applying auth.0001_initial... OK
  Applying rules.0052_useraction_user... OK
  Applying rules.0053_unique_none_rules... OK
  Applying rules.0054_login_action... OK
  Applying rules.0055_auto_20180213_1723... OK
  Applying rules.0056_auto_20180223_0823... OK
  Applying rules.0057_auto_20180302_1312... OK
  Applying rules.0058_source_public_source... OK
  Applying rules.0059_auto_20180309_2012... OK
  Applying rules.0060_auto_20180403_0921... OK
  Applying rules.0061_auto_20180507_1410... OK
  Applying rules.0062_useraction_username... OK
  Applying rules.0063_ruleprocessingfilter_ruleprocessingfilterdef... OK
  Applying rules.0064_ruleprocessingfilter_rulesets... OK
  Applying rules.0061_auto_20180503_2200... OK
  Applying rules.0063_merge_20180718_0118... OK
  Applying rules.0065_merge_20180719_1505... OK
  Applying rules.0066_auto_20180807_1428... OK
  Applying rules.0067_source_use_iprep... OK
  Applying rules.0068_auto_20180818_2204... OK
  Applying rules.0069_auto_20190220_1500... OK
  Applying rules.0070_ruleprocessingfilterdef_full_string... OK
  Applying rules.0071_filterset... OK
  Applying rules.0072_send_mail... OK
  Applying rules.0073_filterset_description... OK
  Applying rules.0074_redlights_useraction... OK
  Applying rules.0075_suppress_validator... OK
  Applying rules.0075_custom_es_no_empty... OK
  Applying rules.0076_merge_20190926_1233... OK
  Applying rules.0077_auto_20191002_0820... OK
  Applying rules.0078_auto_20200206_1648... OK
  Applying rules.0079_source_remove_choice... OK
  Applying rules.0080_source_version... OK
  Applying rules.0081_django-2... OK
  Applying rules.0082_source_use_sys_proxy... OK
  Applying rules.0083_multi_es_validation... OK
  Applying rules.0084_fakepermissionmodel... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying auth.0010_alter_group_name_max_length... OK
  Applying auth.0011_update_proxy_permissions... OK
  Applying accounts.0001_initial... OK
  Applying accounts.0002_auto_20151110_1657... OK
  Applying accounts.0003_timezone... OK
  Applying accounts.0004_group... OK
  Applying rules.0085_roles_migrations... OK
  Applying accounts.0005_remove_user_flags... OK
  Applying accounts.0006_fix_role_default_priority... OK
  Applying accounts.0007_auto_20220421_1743... OK
  Applying authtoken.0001_initial... OK
  Applying authtoken.0002_auto_20160226_1747... OK
  Applying authtoken.0003_tokenproxy... OK
  Applying rules.0086_ruleset_suppressed_sids... OK
  Applying rules.0087_systemsettings_use_proxy_for_es... OK
  Applying rules.0088_ruleprocessingfilter_import_member... OK
  Applying rules.0089_ruleprocessingfilter_event_type... OK
  Applying rules.0090_useraction_ip... OK
  Applying rules.0091_useraction_missing_ruleset... OK
  Applying sessions.0001_initial... OK
  Applying suricata.0001_initial... OK
  Applying suricata.0002_auto_20151110_1657... OK
  Applying suricata.0003_suricata_yaml_file... OK
  Applying suricata.0004_auto_20160316_0844... OK
  Applying suricata.0005_django-2... OK
from django.contrib.auth.models import User; User.objects.create_superuser(***)
Successfully created source "ETOpen Ruleset"
Successfully updated source "ETOpen Ruleset"
Successfully created source "SSLBL abuse.ch"
Successfully updated source "SSLBL abuse.ch"
Successfully created source "PT Research Ruleset"
Successfully updated source "PT Research Ruleset"
Successfully created default ruleset "Default ruleset"
Successfully removed "stream-events" from ruleset "Default ruleset"
Successfully created suricata "suricata"
Successfully pushed ruleset to suricata "suricata"

236 static files copied to '/static'.
Kibana dashboards reset: Elasticsearch not ready, retrying in 10 seconds.
Kibana dashboards reset: Elasticsearch not ready, retrying in 10 seconds.
Kibana dashboards reset: Elasticsearch not ready, retrying in 10 seconds.
Kibana dashboards reset: Elasticsearch not ready, retrying in 10 seconds.
Kibana dashboards reset: Elasticsearch not ready, retrying in 10 seconds.
Kibana dashboards reset: Elasticsearch not ready, retrying in 10 seconds.
Kibana dashboards reset: Elasticsearch not ready, retrying in 10 seconds.
brsolomon-deloitte commented 2 years ago

Same thing with https://github.com/StamusNetworks/scirius/blob/master/docker/scirius/bin/create_ILM_policy.sh which tries to contact Elasticsearch indiscriminately.

brsolomon-deloitte commented 2 years ago

It would be easy for kibana_reset in rules/es_data.py to actually check for USE_KIBANA before trying to make a bunch of API calls to it.