search

splunk / splunk-ansible

Ansible playbooks for configuring and managing Splunk Enterprise and Universal Forwarder deployments
353 stars 185 forks source link

Splunk Enterprise Security application can't be installed #707

Closed yaroslav-nakonechnikov closed 1 year ago

yaroslav-nakonechnikov commented 1 year ago

Hello,

it looks like it is impossible to install Splunk Enterprise Security thru ansible step:

TASK [splunk_deployer : Determine installed apps] ******************************
ok: [localhost] => (item=splunk_archiver)
ok: [localhost] => (item=splunk_enterprise_on_docker)
ok: [localhost] => (item=splunk_gdi)
ok: [localhost] => (item=splunk_monitoring_console)
ok: [localhost] => (item=SplunkForwarder)
ok: [localhost] => (item=splunk_app_soar)
ok: [localhost] => (item=config_explorer)
ok: [localhost] => (item=splunk_essentials_9_0)
ok: [localhost] => (item=DA-ESS-ContentUpdate)
ok: [localhost] => (item=sample_app)
ok: [localhost] => (item=splunk_httpinput)
ok: [localhost] => (item=splunk_internal_metrics)
ok: [localhost] => (item=user-prefs)
ok: [localhost] => (item=launcher)
ok: [localhost] => (item=lookup_editor)
ok: [localhost] => (item=alert_webhook)
ok: [localhost] => (item=journald_input)
ok: [localhost] => (item=splunk_metrics_workspace)
ok: [localhost] => (item=legacy)
ok: [localhost] => (item=splunk_instrumentation)
ok: [localhost] => (item=learned)
ok: [localhost] => (item=phantom)
ok: [localhost] => (item=alert_logevent)
ok: [localhost] => (item=splunk-dashboard-studio)
ok: [localhost] => (item=search)
ok: [localhost] => (item=splunk_rapid_diag)
ok: [localhost] => (item=SplunkLightForwarder)
ok: [localhost] => (item=appsbrowser)
ok: [localhost] => (item=python_upgrade_readiness_app)
ok: [localhost] => (item=SplunkEnterpriseSecuritySuite)
ok: [localhost] => (item=Splunk_TA_snow)
ok: [localhost] => (item=introspection_generator_addon)
ok: [localhost] => (item=splunk_assist)
ok: [localhost] => (item=splunk_datasets_addon)
ok: [localhost] => (item=splunk_secure_gateway)
ok: [localhost] => (item=Splunk_Security_Essentials)
Thursday 02 February 2023  15:39:52 +0000 (0:00:00.622)       0:04:21.774 *****
included: /opt/ansible/roles/splunk_common/tasks/premium_apps/configure_ess.yml for localhost
Thursday 02 February 2023  15:39:52 +0000 (0:00:00.078)       0:04:21.852 *****

TASK [splunk_deployer : Enable SplunkEnterpriseSecuriteSuite app] **************
changed: [localhost]
Thursday 02 February 2023  15:39:58 +0000 (0:00:05.795)       0:04:27.648 *****

TASK [splunk_deployer : Get ESS version] ***************************************
ok: [localhost]
Thursday 02 February 2023  15:40:00 +0000 (0:00:02.376)       0:04:30.024 *****

FAILED - RETRYING: Run ESS post-install setup (10 retries left).
FAILED - RETRYING: Run ESS post-install setup (9 retries left).
FAILED - RETRYING: Run ESS post-install setup (8 retries left).
FAILED - RETRYING: Run ESS post-install setup (7 retries left).
FAILED - RETRYING: Run ESS post-install setup (6 retries left).
FAILED - RETRYING: Run ESS post-install setup (5 retries left).
FAILED - RETRYING: Run ESS post-install setup (4 retries left).
FAILED - RETRYING: Run ESS post-install setup (3 retries left).
FAILED - RETRYING: Run ESS post-install setup (2 retries left).
FAILED - RETRYING: Run ESS post-install setup (1 retries left).

TASK [splunk_deployer : Run ESS post-install setup] ****************************
fatal: [localhost]: FAILED! => {
    "attempts": 10,
    "changed": true,
    "cmd": [
        "/opt/splunk/bin/splunk",
        "search",
        "| essinstall --ssl_enablement auto --deployment_type shc_deployer",
        "-auth",
        "admin:f3hrlrFo0YcyODY4FtZc8GVo"
    ],
    "delta": "0:00:01.105343",
    "end": "2023-02-02 15:45:37.479420",
    "rc": 17,
    "start": "2023-02-02 15:45:36.374077"
}

STDERR:

WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.
FATAL: Error in 'essinstall' command: Automatic SSL enablement is not permitted on the deployer

MSG:

non-zero return code

PLAY RECAP *********************************************************************
localhost                  : ok=161  changed=17   unreachable=0    failed=1    skipped=82   rescued=0    ignored=0

Thursday 02 February 2023  15:45:37 +0000 (0:05:36.953)       0:10:06.978 *****
===============================================================================
splunk_deployer : Run ESS post-install setup -------------------------- 336.95s
splunk_common : Restart the splunkd service - Via CLI ------------------ 62.28s
splunk_deployer : Install app via REST --------------------------------- 32.39s
splunk_common : Start Splunk via CLI ----------------------------------- 24.45s
splunk_deployer : Download remote app ---------------------------------- 12.55s
splunk_common : Get Splunk status --------------------------------------- 9.39s
splunk_common : Set options in role_rbi_proxy_user ---------------------- 8.01s
splunk_deployer : Check app contents ------------------------------------ 6.95s
splunk_common : Set options in roleMap_SAML ----------------------------- 6.53s
splunk_deployer : Install app via REST ---------------------------------- 6.29s
splunk_common : Set options in saml ------------------------------------- 5.88s
splunk_deployer : Enable SplunkEnterpriseSecuriteSuite app -------------- 5.82s
splunk_deployer : Install app via REST ---------------------------------- 5.68s
splunk_deployer : Install app via REST ---------------------------------- 4.70s
splunk_deployer : Install app via REST ---------------------------------- 3.35s
splunk_deployer : Install app via REST ---------------------------------- 3.12s
splunk_deployer : Check local app --------------------------------------- 3.05s
splunk_deployer : Install app via REST ---------------------------------- 2.63s
splunk_deployer : Check local app --------------------------------------- 2.55s
splunk_deployer : Install app via REST ---------------------------------- 2.55s

in UI it is said: Unable to initialize modular input "es_identity_export" defined in the app "SplunkEnterpriseSecuritySuite": Introspecting scheme=es_identity_export: script running failed (PID 6543 exited with code 1)

in documentation: https://splunk.github.io/splunk-ansible/ADVANCED.html and https://splunk.github.io/splunk-ansible/advanced/default.yml.spec.html#spec no special settings described, so i guess it should work.

yaroslav-nakonechnikov commented 1 year ago 
[splunk@splunk-shc-e-deployer-0 splunk]$ /opt/splunk/bin/splunk search '| essinstall --ssl_enablement auto --deployment_type shc_deployer' -auth admin:SECRET
WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.
FATAL: Error in 'essinstall' command: Automatic SSL enablement is not permitted on the deployer

i guess this should be catched somehow...

yaroslav-nakonechnikov commented 1 year ago

can it be related to #416?

kumarajeet commented 1 year ago

@iaroslav-nakonechnikov thanks for sharing the issue.

The "auto" mode is not supported in the ES installation on a SHC (through deployer).

Please see the following setting for ES ssl enablement flag in https://splunk.github.io/splunk-ansible/ADVANCED.html

SPLUNK_ES_SSL_ENABLEMENT Set the ssl-enablement flag in ES. Valid values are 'auto', 'strict', and 'ignore'. Defaults to auto when present.

Please try the SPLUNK_ES_SSL_ENABLEMENT value as ignore or strict and check the ES installation again. [these values are described more in the Splunk ES doc https://docs.splunk.com/Documentation/ES/7.1.0/Install/InstallEnterpriseSecuritySHC]

Ignore mode would not check if splunkd in the SHC are ssl enabled. That is, it will ignore web.xml is : enableSplunkWebSSL = value in the SHC.

For strict mode, you will need to enable splunkd ssl through web.xml before installing ES. For this you will need to push web.xml from an app through deployer bundle push. The relevant setting in web.xml is : enableSplunkWebSSL =

yaroslav-nakonechnikov commented 1 year ago

@kumarajeet yes, thank you! i also finally found this setting, and managed to install it!

ps. SPLUNK_ES_SSL_ENABLEMENT is duplicated twice there in table: https://splunk.github.io/splunk-ansible/ADVANCED.html

yaroslav-nakonechnikov commented 1 year ago

nope, it was too fast.

i found, that ess 6.2.2 installs, but latest 7.1.0 can't, and i see that:

TASK [splunk_deployer : Get ESS version] ***************************************
ok: [localhost]
Monday 06 February 2023  12:46:39 +0000 (0:00:02.521)       0:06:00.470 *******
FAILED - RETRYING: Run ESS post-install setup (10 retries left).
FAILED - RETRYING: Run ESS post-install setup (9 retries left).
FAILED - RETRYING: Run ESS post-install setup (8 retries left).
FAILED - RETRYING: Run ESS post-install setup (7 retries left).
FAILED - RETRYING: Run ESS post-install setup (6 retries left).
FAILED - RETRYING: Run ESS post-install setup (5 retries left).
FAILED - RETRYING: Run ESS post-install setup (4 retries left).
FAILED - RETRYING: Run ESS post-install setup (3 retries left).
FAILED - RETRYING: Run ESS post-install setup (2 retries left).
FAILED - RETRYING: Run ESS post-install setup (1 retries left).

TASK [splunk_deployer : Run ESS post-install setup] ****************************
fatal: [localhost]: FAILED! => {
    "attempts": 10,
    "changed": true,
    "cmd": [
        "/opt/splunk/bin/splunk",
        "search",
        "| essinstall --ssl_enablement ignore --deployment_type shc_deployer",
        "-auth",
        "admin:QNejWN2qpBLndGh2puUYtuAm"
    ],
    "delta": "0:01:13.003950",
    "end": "2023-02-06 13:05:29.972898",
    "rc": 17,
    "start": "2023-02-06 13:04:16.968948"
}

STDERR:

WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.
FATAL: Error in 'essinstall' command: (InstallException) "install_apps" stage failed - Splunkd daemon is not responding: ('Error connecting to /services/apps/shc/es_deployer: The read operation timed out',)

MSG:

non-zero return code

PLAY RECAP *********************************************************************
localhost                  : ok=162  changed=37   unreachable=0    failed=1    skipped=81   rescued=0    ignored=0

Monday 06 February 2023  13:05:30 +0000 (0:18:50.078)       0:24:50.549 *******
===============================================================================
splunk_deployer : Run ESS post-install setup ------------------------- 1130.08s
splunk_deployer : Wait for SHC to be ready ----------------------------- 97.61s
splunk_common : Restart the splunkd service - Via CLI ------------------ 45.98s
splunk_deployer : Install app via REST --------------------------------- 34.87s
splunk_common : Check Splunk instance is running ----------------------- 28.25s
splunk_common : Start Splunk via CLI ----------------------------------- 19.55s
splunk_deployer : Download remote app ---------------------------------- 13.28s
splunk_common : Set options in role_rbi_proxy_user ---------------------- 8.74s
splunk_common : Set options in roleMap_SAML ----------------------------- 7.38s
splunk_deployer : Check app contents ------------------------------------ 7.05s
splunk_common : Set options in saml ------------------------------------- 6.46s
splunk_deployer : Enable SplunkEnterpriseSecuriteSuite app -------------- 4.86s
splunk_deployer : Install app via REST ---------------------------------- 3.75s
splunk_deployer : Install app via REST ---------------------------------- 3.36s
splunk_deployer : Install app via REST ---------------------------------- 3.03s
splunk_deployer : Check local app --------------------------------------- 2.97s
splunk_deployer : Install app via REST ---------------------------------- 2.56s
splunk_deployer : Install app via REST ---------------------------------- 2.50s
splunk_deployer : Get ESS version --------------------------------------- 2.50s
splunk_deployer : Check local app --------------------------------------- 2.40s

somehow it can't connect.

yaroslav-nakonechnikov commented 1 year ago

and with SPLUNK_HTTP_ENABLESSL=true and SPLUNK_ES_SSL_ENABLEMENT = strict setting:

TASK [splunk_deployer : Run ESS post-install setup] ****************************
fatal: [localhost]: FAILED! => {
    "attempts": 10,
    "changed": true,
    "cmd": [
        "/opt/splunk/bin/splunk",
        "search",
        "| essinstall --ssl_enablement strict --deployment_type shc_deployer",
        "-auth",
        "admin:kYHb9FJUgCNY0KT4rGle9lMm"
    ],
    "delta": "0:00:01.148483",
    "end": "2023-02-06 13:26:25.947112",
    "rc": 17,
    "start": "2023-02-06 13:26:24.798629"
}

STDERR:

WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.
FATAL: Error in 'essinstall' command: You must have SSL enabled to continue

MSG:

non-zero return code

PLAY RECAP *********************************************************************
localhost                  : ok=164  changed=38   unreachable=0    failed=1    skipped=83   rescued=0    ignored=0

Monday 06 February 2023  13:26:25 +0000 (0:05:17.067)       0:11:11.594 *******
===============================================================================
splunk_deployer : Run ESS post-install setup -------------------------- 317.07s
splunk_deployer : Wait for SHC to be ready ----------------------------- 87.17s
splunk_common : Restart the splunkd service - Via CLI ------------------ 45.29s
splunk_common : Check Splunk instance is running ----------------------- 40.21s
splunk_deployer : Install app via REST --------------------------------- 29.58s
splunk_common : Start Splunk via CLI ----------------------------------- 18.02s
splunk_deployer : Download remote app ---------------------------------- 17.38s
splunk_deployer : Check app contents ------------------------------------ 9.83s
splunk_common : Set options in role_rbi_proxy_user ---------------------- 7.97s
splunk_common : Set options in roleMap_SAML ----------------------------- 6.51s
splunk_common : Set options in saml ------------------------------------- 5.92s
splunk_deployer : Enable SplunkEnterpriseSecuriteSuite app -------------- 4.54s
splunk_deployer : Install app via REST ---------------------------------- 3.66s
splunk_deployer : Install app via REST ---------------------------------- 3.11s
splunk_deployer : Install app via REST ---------------------------------- 2.89s
splunk_deployer : Install app via REST ---------------------------------- 2.49s
splunk_deployer : Check local app --------------------------------------- 2.41s
splunk_common : Get Splunk status --------------------------------------- 2.26s
splunk_deployer : Check local app --------------------------------------- 2.15s
splunk_deployer : Get ESS version --------------------------------------- 2.11s
yaroslav-nakonechnikov commented 1 year ago

this command works:

[splunk@splunk-shc-e-deployer-0 splunk]$ /opt/splunk/bin/splunk search '| essinstall --dry-run --ssl_enablement ignore --deployment_type shc_deployer' -auth admin:ddd
WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.
                   app                                            filename                        action  disabled
------------------------------------------ ------------------------------------------------------ ------- --------
DA-ESS-AccessProtection                    DA-ESS-AccessProtection-7.1.0-39099.spl                upgrade False
DA-ESS-EndpointProtection                  DA-ESS-EndpointProtection-7.1.0-39099.spl              upgrade False
DA-ESS-IdentityManagement                  DA-ESS-IdentityManagement-7.1.0-39099.spl              upgrade False
DA-ESS-NetworkProtection                   DA-ESS-NetworkProtection-7.1.0-39099.spl               upgrade False
DA-ESS-ThreatIntelligence                  DA-ESS-ThreatIntelligence-7.1.0-39099.spl              upgrade False
SA-AccessProtection                        SA-AccessProtection-7.1.0-39099.spl                    upgrade False
SA-AuditAndDataProtection                  SA-AuditAndDataProtection-7.1.0-39099.spl              upgrade False
SA-EndpointProtection                      SA-EndpointProtection-7.1.0-39099.spl                  upgrade False
SA-IdentityManagement                      SA-IdentityManagement-7.1.0-39099.spl                  upgrade False
SA-NetworkProtection                       SA-NetworkProtection-7.1.0-39099.spl                   upgrade False
SA-ThreatIntelligence                      SA-ThreatIntelligence-7.1.0-39099.spl                  upgrade False
SA-UEBA                                    SA-UEBA-7.1.0-39099.spl                                upgrade True
SA-Utils                                   SA-Utils-7.1.0-39099.spl                               upgrade False
Splunk_ML_Toolkit                          Splunk_ML_Toolkit-5.3.1-1641570609576.tgz              upgrade False
Splunk_SA_CIM                              Splunk_SA_CIM-5.1.0-231.tgz                            upgrade False
Splunk_SA_Scientific_Python_linux_x86_64   Splunk_SA_Scientific_Python_linux_x86_64-3.0.2-0.tgz   upgrade False
Splunk_SA_Scientific_Python_windows_x86_64 Splunk_SA_Scientific_Python_windows_x86_64-3.0.2-0.tgz skip
Splunk_TA_ueba                             Splunk_TA_ueba-3.1.0-3156.spl                          install False
yaroslav-nakonechnikov commented 1 year ago

ok, looks like it finally started to work.

my solution was to run with bigger amount of cpu. Previously i was starting on 2 cores, and it failed. With 4 cores - it started. Which is a bit sad story.

splunk itself writes about 16 cores as minimum, where i can't agree for development environment.