UF Containers broken for all 9.x Images

mag-mkorn commented 1 year ago

Hi folks,

apparently all the UF Images for Version 9.x seem to be broken.

The ansible playbooks always get stuck at task Get Splunk status.

I tried to troubleshoot the problem, but wasn't able to find the problem, yet. Here are my findings so far:

The Warning "Attempting to revert the SPLUNK_HOME ownership" is written to user-seed.conf. It's not really nice, but doesn't seem to be the problem.
When playbook execution is paused before the failing task, I can manually execute the status command without any problems. Playbook execution can be continued after the manual execution as well and runs without problems.
When the playbook runs into the problematic state and I run the status command afterwards, splunk tries to perform an upgrade and fails.
The failing task performs some, but not all the actions that happen during the manual command execution. Certificates get created in both cases, but the failing task does not remove the ftr file.

I don't know why the same command fails when it is executed by ansible. All could image is that some environment variables could differ. I didn't check this, yet.

chipzzz commented 1 year ago

@alishamayor , has this been tested before it has been approved? Because it broke everything. https://github.com/splunk/splunk-ansible/commit/81b552d3ec0de6c199c3169d4f4e7248083cb989#commitcomment-87802600

Adam2Marsh commented 1 year ago

I've raised this to splunk via there support portal; case ref is 3200123

Will update this once I hear back.

Adam2Marsh commented 1 year ago

I got the following response; it's a known issue

This is a known Docker bug, its currently working in progress with Expected fix: 9.1.0 As of now the workaround is as follows

Workaround:

Remove: /opt/splunkforwarder/etc/myinstall/splunkd.xml file then start Splunk

Or this can be confgured:

Add the following to the compose file: splunk: tty: true

https://community.splunk.com/t5/Installation/Upgrading-Universal-Forwarder-8-x-x-to-9-x-x-does-not-work/m-p/619919_

I've asked for a date when version 9.1.0 will be released.

gergelydavidwork commented 1 year ago

I got the following response; it's a known issue

This is a known Docker bug, its currently working in progress with Expected fix: 9.1.0 As of now the workaround is as follows

Workaround:

Remove: /opt/splunkforwarder/etc/myinstall/splunkd.xml file then start Splunk

Or this can be confgured:

Add the following to the compose file: splunk: tty: true

https://community.splunk.com/t5/Installation/Upgrading-Universal-Forwarder-8-x-x-to-9-x-x-does-not-work/m-p/619919_

I've asked for a date when version 9.1.0 will be released.

Hi @Adam2Marsh , Did you get any response maybe when this will be released? Thank you!

Adam2Marsh commented 1 year ago

Hey @gergelydavidwork

This was fixed in 9.0.5; I haven't tested it myself but assured it is.

gergelydavidwork commented 1 year ago

Thanks @Adam2Marsh, but I have tried 9.0.5 and it is NOT fixed. That's why I asked if you got any response for the release date of 9.1.0

tnedeczki commented 1 year ago

I can't believe why it's taking so much time to fix this bug, which has been going on since September. Instead, you have to use 8.2.X versions which are quite risky from a security point of view.

The whole ansible process gets stuck when it comes to a task where it needs to call ./splunk.

For example:

get_splunk_status task: https://github.com/splunk/splunk-ansible/blob/develop/roles/splunk_common/tasks/get_splunk_status.yml#L10
if the following SPLUNK_BEFORE_START_CMD flag is used, this task will also be stuck: pre_splunk_start_commands https://github.com/splunk/splunk-ansible/blob/develop/roles/splunk_common/tasks/pre_splunk_start_commands.yml

If you go inside the container and kill of those process, you will get the following error message:

Error calling execve(): No such file or directory
Error launching  command: Invalid argument

Adam2Marsh commented 1 year ago

Hey All,

Re-raising back to splunk now... as I've doubled check and yep you still need the workaround to make 9.0.5

adityapinglesf commented 1 year ago

9.0.5 was updated yesterday. I was able to get the UF running with the following command -

 docker run -p 8000:8000 -p 8089:8089 -e "SPLUNK_PASSWORD=trypass@" -e "SPLUNK_START_ARGS=--accept-license" --name uf-1 -it splunk/universalforwarder:9.0.5

Can you try now with the newly released image?

tnedeczki commented 1 year ago

Hi @adityapinglesf

Thanks, it is working now.

Iammusa18 commented 1 year ago

Hi, @adityapinglesf @tnedeczki while it is encouraging to hear its working for you... i have ran the exact same command but 9.0.5 it still NOT working and it is stuck on below and never progresses beyond that...So no, 9.0.5 does not seem fixed at all. If someone has got this to work, please share any workaround you've employed. cheers

@Adam2Marsh can you please suggest how that workaround can be run as part of a Dockerfile etc..?

Thursday 13 July 2023  16:49:19 +0000 (0:00:00.062)       0:00:08.499 *********

TASK [splunk_common : include_tasks] *************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/get_splunk_status.yml for localhost
ok: [localhost]
Thursday 13 July 2023  16:49:19 +0000 (0:00:00.242)       0:00:08.742 *********

@splunk when will this be fixed? It does not make sense to release newer versions if your offical docker image does not support or is incompatible with said version? Thanks

bdalpe commented 1 year ago

@Iammusa18 double check you're not running an old cached version of the 9.0.5 image. A newer version was published with the same tag. Docker will not pull a new image if one is cached locally.

Run docker pull splunk/universalforwarder:9.0.5 to make sure it is up-to-date.

tnedeczki commented 1 year ago

Hi, @adityapinglesf @tnedeczki while it is encouraging to hear its working for you... i have ran the exact same command but 9.0.5 it still NOT working and it is stuck on below and never progresses beyond that...So no, 9.0.5 does not seem fixed at all. If someone has got this to work, please share any workaround you've employed. cheers

@Adam2Marsh can you please suggest how that workaround can be run as part of a Dockerfile etc..?
Thursday 13 July 2023  16:49:19 +0000 (0:00:00.062)       0:00:08.499 *********

TASK [splunk_common : include_tasks] *************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/get_splunk_status.yml for localhost
ok: [localhost]
Thursday 13 July 2023  16:49:19 +0000 (0:00:00.242)       0:00:08.742 *********
@splunk when will this be fixed? It does not make sense to release newer versions if your offical docker image does not support or is incompatible with said version? Thanks

yes, I ran into the same problem, but as @bdalpe mentions, you need to refresh the image.

➜  docker run -p 8000:8000 -p 8089:8089 -e "SPLUNK_PASSWORD=trypass@" -e "SPLUNK_START_ARGS=--accept-license" --name uf-1 -it splunk/universalforwarder:9.0.5
[DEPRECATION WARNING]: Ansible will require Python 3.8 or newer on the controller starting with Ansible 2.12. Current version: 3.7.16 (default, Jul 13 2023, 00:11:50) [GCC 8.5.0 20210514 (Red Hat 8.5.0-18)].
This feature will be removed from ansible-core in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: COMMAND_WARNINGS option, the command warnings feature is being removed. This feature will be removed from ansible-core in version 2.14. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: [defaults]callback_whitelist option, normalizing names to new standard, use callbacks_enabled instead. This feature will be removed from ansible-core in version 2.15. Deprecation
warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.

PLAY [Run default Splunk provisioning] **************************************************************************************************************************************************************************
Friday 14 July 2023  07:57:32 +0000 (0:00:00.322)       0:00:00.322 ***********

TASK [Gathering Facts] ******************************************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:35 +0000 (0:00:03.532)       0:00:03.855 ***********
Friday 14 July 2023  07:57:35 +0000 (0:00:00.136)       0:00:03.992 ***********

TASK [Provision role] *******************************************************************************************************************************************************************************************
Friday 14 July 2023  07:57:36 +0000 (0:00:00.230)       0:00:04.222 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/get_facts.yml for localhost
Friday 14 July 2023  07:57:36 +0000 (0:00:00.151)       0:00:04.374 ***********

TASK [splunk_common : Set privilege escalation user] ************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:36 +0000 (0:00:00.177)       0:00:04.551 ***********

TASK [splunk_common : Check for scloud] *************************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:37 +0000 (0:00:01.093)       0:00:05.644 ***********
Friday 14 July 2023  07:57:37 +0000 (0:00:00.096)       0:00:05.741 ***********
Friday 14 July 2023  07:57:37 +0000 (0:00:00.064)       0:00:05.806 ***********

TASK [splunk_common : Check for existing installation] **********************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:38 +0000 (0:00:00.679)       0:00:06.486 ***********

TASK [splunk_common : Set splunk install fact] ******************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:38 +0000 (0:00:00.162)       0:00:06.648 ***********

TASK [splunk_common : Check for existing splunk secret] *********************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:39 +0000 (0:00:00.613)       0:00:07.262 ***********

TASK [splunk_common : Set first run fact] ***********************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:39 +0000 (0:00:00.129)       0:00:07.391 ***********

TASK [splunk_common : Set splunk_build_type fact] ***************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/get_facts_build_type.yml for localhost
Friday 14 July 2023  07:57:39 +0000 (0:00:00.131)       0:00:07.523 ***********
Friday 14 July 2023  07:57:39 +0000 (0:00:00.077)       0:00:07.601 ***********
Friday 14 July 2023  07:57:39 +0000 (0:00:00.087)       0:00:07.688 ***********
Friday 14 July 2023  07:57:39 +0000 (0:00:00.084)       0:00:07.773 ***********
Friday 14 July 2023  07:57:39 +0000 (0:00:00.115)       0:00:07.889 ***********
Friday 14 July 2023  07:57:40 +0000 (0:00:00.134)       0:00:08.023 ***********

TASK [splunk_common : Set target version fact] ******************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/get_facts_target_version.yml for localhost
Friday 14 July 2023  07:57:40 +0000 (0:00:00.214)       0:00:08.238 ***********
Friday 14 July 2023  07:57:40 +0000 (0:00:00.073)       0:00:08.312 ***********
Friday 14 July 2023  07:57:40 +0000 (0:00:00.072)       0:00:08.384 ***********
Friday 14 July 2023  07:57:40 +0000 (0:00:00.100)       0:00:08.485 ***********

TASK [splunk_common : Find manifests] ***************************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:41 +0000 (0:00:00.974)       0:00:09.460 ***********

TASK [splunk_common : Set current version fact] *****************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:41 +0000 (0:00:00.211)       0:00:09.672 ***********

TASK [splunk_common : Setting upgrade fact] *********************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:41 +0000 (0:00:00.345)       0:00:10.017 ***********

TASK [splunk_common : Setting indexer cluster fact from config] *************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:42 +0000 (0:00:00.235)       0:00:10.253 ***********

TASK [splunk_common : Setting search head cluster fact from config] *********************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:42 +0000 (0:00:00.191)       0:00:10.444 ***********
Friday 14 July 2023  07:57:42 +0000 (0:00:00.109)       0:00:10.554 ***********
Friday 14 July 2023  07:57:42 +0000 (0:00:00.167)       0:00:10.722 ***********
Friday 14 July 2023  07:57:42 +0000 (0:00:00.130)       0:00:10.852 ***********

TASK [splunk_common : Detect service name] **********************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/get_facts_service_name.yml for localhost
Friday 14 July 2023  07:57:43 +0000 (0:00:00.268)       0:00:11.121 ***********

TASK [splunk_common : Setting service_name fact from config] ****************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:43 +0000 (0:00:00.216)       0:00:11.337 ***********
Friday 14 July 2023  07:57:43 +0000 (0:00:00.116)       0:00:11.454 ***********
Friday 14 July 2023  07:57:43 +0000 (0:00:00.141)       0:00:11.596 ***********
Friday 14 July 2023  07:57:43 +0000 (0:00:00.179)       0:00:11.776 ***********
Friday 14 July 2023  07:57:43 +0000 (0:00:00.236)       0:00:12.013 ***********
Friday 14 July 2023  07:57:44 +0000 (0:00:00.144)       0:00:12.157 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/change_splunk_directory_owner.yml for localhost
Friday 14 July 2023  07:57:44 +0000 (0:00:00.299)       0:00:12.456 ***********

TASK [splunk_common : Update Splunk directory owner] ************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:45 +0000 (0:00:01.288)       0:00:13.745 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/update_etc.yml for localhost
Friday 14 July 2023  07:57:46 +0000 (0:00:00.342)       0:00:14.087 ***********

TASK [splunk_common : Check if /sbin/updateetc.sh exists] *******************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:46 +0000 (0:00:00.810)       0:00:14.897 ***********

TASK [splunk_common : Update /opt/splunk/etc] *******************************************************************************************************************************************************************
changed: [localhost]
Friday 14 July 2023  07:57:48 +0000 (0:00:01.144)       0:00:16.042 ***********
Friday 14 July 2023  07:57:48 +0000 (0:00:00.124)       0:00:16.167 ***********
Friday 14 July 2023  07:57:48 +0000 (0:00:00.186)       0:00:16.354 ***********
Friday 14 July 2023  07:57:48 +0000 (0:00:00.117)       0:00:16.472 ***********
Friday 14 July 2023  07:57:48 +0000 (0:00:00.145)       0:00:16.618 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/remove_first_login.yml for localhost
Friday 14 July 2023  07:57:48 +0000 (0:00:00.153)       0:00:16.771 ***********

TASK [splunk_common : Create .ui_login] *************************************************************************************************************************************************************************
changed: [localhost]
Friday 14 July 2023  07:57:49 +0000 (0:00:00.838)       0:00:17.610 ***********
Friday 14 July 2023  07:57:49 +0000 (0:00:00.173)       0:00:17.784 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/set_splunk_secret.yml for localhost
Friday 14 July 2023  07:57:50 +0000 (0:00:00.265)       0:00:18.049 ***********
Friday 14 July 2023  07:57:50 +0000 (0:00:00.135)       0:00:18.185 ***********
Friday 14 July 2023  07:57:50 +0000 (0:00:00.121)       0:00:18.306 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/enable_admin_auth.yml for localhost
Friday 14 July 2023  07:57:50 +0000 (0:00:00.233)       0:00:18.540 ***********

TASK [splunk_common : Hash the password] ************************************************************************************************************************************************************************
changed: [localhost]
Friday 14 July 2023  07:57:52 +0000 (0:00:02.238)       0:00:20.778 ***********

TASK [splunk_common : Generate user-seed.conf (Linux)] **********************************************************************************************************************************************************
changed: [localhost] => (item=USERNAME)
changed: [localhost] => (item=HASHED_PASSWORD)
Friday 14 July 2023  07:57:54 +0000 (0:00:01.885)       0:00:22.664 ***********
Friday 14 July 2023  07:57:54 +0000 (0:00:00.203)       0:00:22.867 ***********
Friday 14 July 2023  07:57:55 +0000 (0:00:00.203)       0:00:23.071 ***********
Friday 14 July 2023  07:57:55 +0000 (0:00:00.131)       0:00:23.203 ***********
Friday 14 July 2023  07:57:55 +0000 (0:00:00.138)       0:00:23.341 ***********
Friday 14 July 2023  07:57:55 +0000 (0:00:00.070)       0:00:23.411 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/pre_splunk_start_commands.yml for localhost
Friday 14 July 2023  07:57:55 +0000 (0:00:00.187)       0:00:23.599 ***********
Friday 14 July 2023  07:57:55 +0000 (0:00:00.103)       0:00:23.703 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/enable_s2s.yml for localhost
Friday 14 July 2023  07:57:55 +0000 (0:00:00.223)       0:00:23.926 ***********
Friday 14 July 2023  07:57:56 +0000 (0:00:00.239)       0:00:24.166 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/s2s/configure_splunktcp.yml for localhost
Friday 14 July 2023  07:57:56 +0000 (0:00:00.396)       0:00:24.562 ***********

TASK [splunk_common : Enable splunktcp input] *******************************************************************************************************************************************************************
changed: [localhost]
Friday 14 July 2023  07:57:57 +0000 (0:00:00.813)       0:00:25.376 ***********

TASK [splunk_common : Remove splunktcp-ssl input] ***************************************************************************************************************************************************************
changed: [localhost]
Friday 14 July 2023  07:57:58 +0000 (0:00:00.749)       0:00:26.125 ***********

TASK [splunk_common : Remove input SSL settings] ****************************************************************************************************************************************************************
changed: [localhost]
Friday 14 July 2023  07:57:58 +0000 (0:00:00.759)       0:00:26.885 ***********

TASK [splunk_common : Reset root CA] ****************************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:57:59 +0000 (0:00:00.708)       0:00:27.594 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/trigger_restart.yml for localhost
Friday 14 July 2023  07:57:59 +0000 (0:00:00.151)       0:00:27.746 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/get_splunk_status.yml for localhost
Friday 14 July 2023  07:57:59 +0000 (0:00:00.202)       0:00:27.948 ***********

TASK [splunk_common : Restrict permissions on splunk.key for Status] ********************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/restrict_permissions.yml for localhost => (item=/opt/splunkforwarder/var/lib/splunk/kvstore/mongo/splunk.key)
Friday 14 July 2023  07:58:00 +0000 (0:00:00.541)       0:00:28.490 ***********

TASK [splunk_common : Check if /opt/splunkforwarder/var/lib/splunk/kvstore/mongo/splunk.key exists] *************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:01 +0000 (0:00:01.144)       0:00:29.634 ***********
Friday 14 July 2023  07:58:01 +0000 (0:00:00.073)       0:00:29.708 ***********

TASK [splunk_common : Get Splunk status] ************************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:02 +0000 (0:00:00.792)       0:00:30.501 ***********

TASK [splunk_common : Trigger restart] **************************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:03 +0000 (0:00:00.605)       0:00:31.106 ***********
Friday 14 July 2023  07:58:03 +0000 (0:00:00.078)       0:00:31.185 ***********
Friday 14 July 2023  07:58:03 +0000 (0:00:00.108)       0:00:31.294 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/set_mgmt_port.yml for localhost
Friday 14 July 2023  07:58:03 +0000 (0:00:00.156)       0:00:31.451 ***********

TASK [splunk_common : Set mgmt port] ****************************************************************************************************************************************************************************
changed: [localhost]
Friday 14 July 2023  07:58:04 +0000 (0:00:00.750)       0:00:32.201 ***********
Friday 14 July 2023  07:58:04 +0000 (0:00:00.064)       0:00:32.266 ***********
Friday 14 July 2023  07:58:04 +0000 (0:00:00.099)       0:00:32.365 ***********
Friday 14 July 2023  07:58:04 +0000 (0:00:00.103)       0:00:32.469 ***********
Friday 14 July 2023  07:58:04 +0000 (0:00:00.125)       0:00:32.595 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/enable_splunkd_ssl.yml for localhost
Friday 14 July 2023  07:58:04 +0000 (0:00:00.227)       0:00:32.823 ***********

TASK [splunk_common : Enable Splunkd SSL] ***********************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:05 +0000 (0:00:00.704)       0:00:33.527 ***********
Friday 14 July 2023  07:58:05 +0000 (0:00:00.064)       0:00:33.592 ***********
Friday 14 July 2023  07:58:05 +0000 (0:00:00.065)       0:00:33.657 ***********
Friday 14 July 2023  07:58:05 +0000 (0:00:00.066)       0:00:33.724 ***********
Friday 14 July 2023  07:58:05 +0000 (0:00:00.057)       0:00:33.782 ***********
Friday 14 July 2023  07:58:05 +0000 (0:00:00.134)       0:00:33.916 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/enable_forwarding.yml for localhost
Friday 14 July 2023  07:58:06 +0000 (0:00:00.251)       0:00:34.168 ***********
Friday 14 July 2023  07:58:06 +0000 (0:00:00.138)       0:00:34.306 ***********
Friday 14 July 2023  07:58:06 +0000 (0:00:00.168)       0:00:34.475 ***********
Friday 14 July 2023  07:58:06 +0000 (0:00:00.162)       0:00:34.637 ***********

TASK [splunk_common : Disable indexing on the current node] *****************************************************************************************************************************************************
changed: [localhost]
Friday 14 July 2023  07:58:07 +0000 (0:00:00.588)       0:00:35.226 ***********
Friday 14 July 2023  07:58:07 +0000 (0:00:00.114)       0:00:35.341 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/trigger_restart.yml for localhost
Friday 14 July 2023  07:58:07 +0000 (0:00:00.120)       0:00:35.461 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/get_splunk_status.yml for localhost
Friday 14 July 2023  07:58:07 +0000 (0:00:00.109)       0:00:35.571 ***********

TASK [splunk_common : Restrict permissions on splunk.key for Status] ********************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/restrict_permissions.yml for localhost => (item=/opt/splunkforwarder/var/lib/splunk/kvstore/mongo/splunk.key)
Friday 14 July 2023  07:58:07 +0000 (0:00:00.112)       0:00:35.683 ***********

TASK [splunk_common : Check if /opt/splunkforwarder/var/lib/splunk/kvstore/mongo/splunk.key exists] *************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:08 +0000 (0:00:00.756)       0:00:36.439 ***********
Friday 14 July 2023  07:58:08 +0000 (0:00:00.065)       0:00:36.505 ***********

TASK [splunk_common : Get Splunk status] ************************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:09 +0000 (0:00:00.575)       0:00:37.080 ***********

TASK [splunk_common : Trigger restart] **************************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:09 +0000 (0:00:00.525)       0:00:37.606 ***********
Friday 14 July 2023  07:58:09 +0000 (0:00:00.052)       0:00:37.659 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/start_splunk.yml for localhost
Friday 14 July 2023  07:58:09 +0000 (0:00:00.182)       0:00:37.841 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/get_splunk_status.yml for localhost
Friday 14 July 2023  07:58:09 +0000 (0:00:00.095)       0:00:37.937 ***********

TASK [splunk_common : Restrict permissions on splunk.key for Status] ********************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/restrict_permissions.yml for localhost => (item=/opt/splunkforwarder/var/lib/splunk/kvstore/mongo/splunk.key)
Friday 14 July 2023  07:58:10 +0000 (0:00:00.094)       0:00:38.032 ***********

TASK [splunk_common : Check if /opt/splunkforwarder/var/lib/splunk/kvstore/mongo/splunk.key exists] *************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:10 +0000 (0:00:00.588)       0:00:38.620 ***********
Friday 14 July 2023  07:58:10 +0000 (0:00:00.075)       0:00:38.696 ***********

TASK [splunk_common : Get Splunk status] ************************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:11 +0000 (0:00:00.608)       0:00:39.304 ***********

TASK [splunk_common : Cleanup Splunk runtime files] *************************************************************************************************************************************************************
ok: [localhost] => (item=/opt/splunkforwarder/var/run/splunk/splunkd.pid)
ok: [localhost] => (item=/opt/splunkforwarder/var/lib/splunk/kvstore/mongo/mongod.lock)
Friday 14 July 2023  07:58:12 +0000 (0:00:01.279)       0:00:40.584 ***********

TASK [splunk_common : Restrict permissions on splunk.key] *******************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/restrict_permissions.yml for localhost => (item=/opt/splunkforwarder/var/lib/splunk/kvstore/mongo/splunk.key)
Friday 14 July 2023  07:58:12 +0000 (0:00:00.113)       0:00:40.697 ***********

TASK [splunk_common : Check if /opt/splunkforwarder/var/lib/splunk/kvstore/mongo/splunk.key exists] *************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:13 +0000 (0:00:00.462)       0:00:41.160 ***********
Friday 14 July 2023  07:58:13 +0000 (0:00:00.054)       0:00:41.214 ***********

TASK [splunk_common : Start Splunk via CLI] *********************************************************************************************************************************************************************
changed: [localhost]
Friday 14 July 2023  07:58:17 +0000 (0:00:04.358)       0:00:45.573 ***********
Friday 14 July 2023  07:58:17 +0000 (0:00:00.097)       0:00:45.670 ***********
Friday 14 July 2023  07:58:17 +0000 (0:00:00.113)       0:00:45.783 ***********

TASK [splunk_common : Wait for splunkd management port] *********************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:19 +0000 (0:00:01.757)       0:00:47.541 ***********
Friday 14 July 2023  07:58:19 +0000 (0:00:00.030)       0:00:47.572 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/set_certificate_prefix.yml for localhost
Friday 14 July 2023  07:58:19 +0000 (0:00:00.416)       0:00:47.989 ***********

TASK [splunk_common : Test basic https endpoint] ****************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:21 +0000 (0:00:02.005)       0:00:49.994 ***********

TASK [splunk_common : Set url prefix for future REST calls] *****************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:22 +0000 (0:00:00.116)       0:00:50.111 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/clean_user_seed.yml for localhost
Friday 14 July 2023  07:58:22 +0000 (0:00:00.195)       0:00:50.306 ***********

TASK [splunk_common : Remove user-seed.conf] ********************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:22 +0000 (0:00:00.592)       0:00:50.898 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/add_splunk_license.yml for localhost
Friday 14 July 2023  07:58:23 +0000 (0:00:00.209)       0:00:51.108 ***********

TASK [splunk_common : Initialize licenses array] ****************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:23 +0000 (0:00:00.141)       0:00:51.249 ***********

TASK [splunk_common : Determine available licenses] *************************************************************************************************************************************************************
ok: [localhost] => (item=splunk.lic)
Friday 14 July 2023  07:58:23 +0000 (0:00:00.161)       0:00:51.411 ***********

TASK [splunk_common : Apply licenses] ***************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/apply_licenses.yml for localhost => (item=splunk.lic)
Friday 14 July 2023  07:58:23 +0000 (0:00:00.202)       0:00:51.614 ***********
Friday 14 July 2023  07:58:23 +0000 (0:00:00.204)       0:00:51.818 ***********
Friday 14 July 2023  07:58:24 +0000 (0:00:00.240)       0:00:52.059 ***********
Friday 14 July 2023  07:58:24 +0000 (0:00:00.099)       0:00:52.159 ***********

TASK [splunk_common : include_tasks] ****************************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/licenses/add_license.yml for localhost
Friday 14 July 2023  07:58:24 +0000 (0:00:00.178)       0:00:52.338 ***********
Friday 14 July 2023  07:58:24 +0000 (0:00:00.097)       0:00:52.435 ***********

TASK [splunk_common : Ensure license path] **********************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:25 +0000 (0:00:00.793)       0:00:53.229 ***********
Friday 14 July 2023  07:58:25 +0000 (0:00:00.077)       0:00:53.306 ***********
Friday 14 July 2023  07:58:25 +0000 (0:00:00.065)       0:00:53.371 ***********
Friday 14 July 2023  07:58:25 +0000 (0:00:00.024)       0:00:53.395 ***********
Friday 14 July 2023  07:58:25 +0000 (0:00:00.126)       0:00:53.521 ***********

TASK [splunk_universal_forwarder : include_tasks] ***************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_universal_forwarder/tasks/../../../roles/splunk_common/tasks/set_as_hec_receiver.yml for localhost
Friday 14 July 2023  07:58:25 +0000 (0:00:00.109)       0:00:53.631 ***********

TASK [splunk_universal_forwarder : Setup global HEC] ************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:27 +0000 (0:00:01.751)       0:00:55.382 ***********

TASK [splunk_universal_forwarder : Get existing HEC token] ******************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:28 +0000 (0:00:01.300)       0:00:56.682 ***********
Friday 14 July 2023  07:58:28 +0000 (0:00:00.072)       0:00:56.755 ***********
Friday 14 July 2023  07:58:28 +0000 (0:00:00.094)       0:00:56.850 ***********
Friday 14 July 2023  07:58:28 +0000 (0:00:00.092)       0:00:56.942 ***********
Friday 14 July 2023  07:58:28 +0000 (0:00:00.031)       0:00:56.974 ***********
Friday 14 July 2023  07:58:29 +0000 (0:00:00.071)       0:00:57.046 ***********
Friday 14 July 2023  07:58:29 +0000 (0:00:00.057)       0:00:57.103 ***********
Friday 14 July 2023  07:58:29 +0000 (0:00:00.057)       0:00:57.161 ***********
Friday 14 July 2023  07:58:29 +0000 (0:00:00.083)       0:00:57.244 ***********

TASK [splunk_universal_forwarder : include_tasks] ***************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_universal_forwarder/tasks/../../../roles/splunk_common/tasks/check_for_required_restarts.yml for localhost
Friday 14 July 2023  07:58:29 +0000 (0:00:00.108)       0:00:57.352 ***********

TASK [splunk_universal_forwarder : Check for required restarts] *************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:30 +0000 (0:00:01.416)       0:00:58.769 ***********
Friday 14 July 2023  07:58:30 +0000 (0:00:00.125)       0:00:58.895 ***********

TASK [Check all instances for required restarts] ****************************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/check_for_required_restarts.yml for localhost
Friday 14 July 2023  07:58:30 +0000 (0:00:00.122)       0:00:59.017 ***********

TASK [Check for required restarts] ******************************************************************************************************************************************************************************
ok: [localhost]
Friday 14 July 2023  07:58:32 +0000 (0:00:01.295)       0:01:00.312 ***********

PLAY RECAP ******************************************************************************************************************************************************************************************************
localhost                  : ok=81   changed=10   unreachable=0    failed=0    skipped=70   rescued=0    ignored=0

Friday 14 July 2023  07:58:32 +0000 (0:00:00.060)       0:01:00.373 ***********
===============================================================================
splunk_common : Start Splunk via CLI --------------------------------------------------------------------------------------------------------------------------------------------------------------------- 4.36s
Gathering Facts ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 3.53s
splunk_common : Hash the password ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2.24s
splunk_common : Test basic https endpoint ---------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.01s
splunk_common : Generate user-seed.conf (Linux) ---------------------------------------------------------------------------------------------------------------------------------------------------------- 1.89s
splunk_common : Wait for splunkd management port --------------------------------------------------------------------------------------------------------------------------------------------------------- 1.76s
splunk_universal_forwarder : Setup global HEC ------------------------------------------------------------------------------------------------------------------------------------------------------------ 1.75s
splunk_universal_forwarder : Check for required restarts ------------------------------------------------------------------------------------------------------------------------------------------------- 1.42s
splunk_universal_forwarder : Get existing HEC token ------------------------------------------------------------------------------------------------------------------------------------------------------ 1.30s
Check for required restarts ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 1.30s
splunk_common : Update Splunk directory owner ------------------------------------------------------------------------------------------------------------------------------------------------------------ 1.29s
splunk_common : Cleanup Splunk runtime files ------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.28s
splunk_common : Update /opt/splunk/etc ------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.14s
splunk_common : Check if /opt/splunkforwarder/var/lib/splunk/kvstore/mongo/splunk.key exists ------------------------------------------------------------------------------------------------------------- 1.14s
splunk_common : Check for scloud ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.09s
splunk_common : Find manifests --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.97s
splunk_common : Create .ui_login ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.84s
splunk_common : Enable splunktcp input ------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.81s
splunk_common : Check if /sbin/updateetc.sh exists ------------------------------------------------------------------------------------------------------------------------------------------------------- 0.81s
splunk_common : Ensure license path ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.79s
===============================================================================

Ansible playbook complete, will begin streaming var/log/splunk/splunkd_stderr.log

mag-mkorn commented 1 year ago

Thanks, the current 9.0.5 Image finally works again. I'll close the issue now.

adityapinglesf commented 1 year ago

Hi, @adityapinglesf @tnedeczki while it is encouraging to hear its working for you... i have ran the exact same command but 9.0.5 it still NOT working and it is stuck on below and never progresses beyond that...So no, 9.0.5 does not seem fixed at all. If someone has got this to work, please share any workaround you've employed. cheers

@Adam2Marsh can you please suggest how that workaround can be run as part of a Dockerfile etc..?
Thursday 13 July 2023  16:49:19 +0000 (0:00:00.062)       0:00:08.499 *********

TASK [splunk_common : include_tasks] *************************************************************************************************************************************************
included: /opt/ansible/roles/splunk_common/tasks/get_splunk_status.yml for localhost
ok: [localhost]
Thursday 13 July 2023  16:49:19 +0000 (0:00:00.242)       0:00:08.742 *********
@splunk when will this be fixed? It does not make sense to release newer versions if your offical docker image does not support or is incompatible with said version? Thanks

As suggested please try re-fetching the image 🙂

Iammusa18 commented 1 year ago

Yeah my bad. Re-fetching image has worked!! Silly me. thanks all

splunk / docker-splunk

UF Containers broken for all 9.x Images #557

https://community.splunk.com/t5/Installation/Upgrading-Universal-Forwarder-8-x-x-to-9-x-x-does-not-work/m-p/619919_

https://community.splunk.com/t5/Installation/Upgrading-Universal-Forwarder-8-x-x-to-9-x-x-does-not-work/m-p/619919_