Splunk SHC - 8.1 - Configuration

[Evotiona]

Hey team, I wrote about this a while ago with an issue in regards to Search Head Cluster deployment for 7.3 version. We are now on Splunk 8.1.1 and are hitting the same issue where we are unable to deploy a configured OR fresh copy of the app (1.0.30) to view the Configuration tab or get the supplied credentials from the Deployment server to take.

We have tried editing the ta_jira_service_desk_simple_addon_settings.conf from the cURL request as mentioned in the documentation, but the password supplied just stays clear text and no "passwords.conf" file is created even after reboot.

We receive the : jirarest.py line 113: NoneType object has no attribute 'find' error which we are imagining comes from there not being a correct password.conf file with the correctly hashed password.

Thoughts on this? We've tried the commenting out of the python.version=3 lines as mentioned before as well. Thanks!

[guilhemmarchand]

Hi @Evotiona !

Right, so, first the Addon is perfectly compatible with an SHC deployment, and its configuration will be automatically replicated to the SHC members, the Add-on configuration is covered here including SHC related mentions:

https://ta-jira-service-desk-simple-addon.readthedocs.io/en/latest/configuration.html

Now, to answer your question, tell me more about:

 to view the Configuration tab or get the supplied credentials from the Deployment server to take.

Are you saying that you are attempting to perform the configuration on the SHC deployer (the credentials and JIRA main settings like the JIRA URL and SSL related settings) then push from the deployer with the SHC bundle?

The supported configuration should be to push the out of the box application (so the normal package extracted with no local pre-configured) then perform the configuration on any member of the SHC.

[Evotiona]

Hello! Thanks for the quick response and help again! Yes exactly what you're describing. We have tried to do the following:

• download the app from Splunk base
• copy to deployer
• push to SHC
• navigate to the app - configuration and we just get the spinning loading icon and the tabs for Proxy/Logging/Additional Parameters with the URL and user and SSL items never load. They do load on a single instance/search head for us but nothing for the 3 heads in the cluster

We occasionally notice this activity on Splunk App Builder type apps but if we ensure there is no passwords.conf and reboot the search head it sometimes comes back but yeah for this one, we've done as you mentioned: no setup on deployer no input of additional parameters and just does not load that configuration page.

I just got done doing the commenting out of the python3 and repushed just for fun and no dice that time either.

[guilhemmarchand]

@Evotiona

Got it, I've seen this occasionnaly too, likely a Javascript cache issue, sometimes requires clearing your Browser cache or doing a bump against the search head, but could be linked to another thing.

I am planning to release a new version using a different and newer framework for the configuration side.

However, for now, can you try to go through the configuration via REST calls using curl and see how it goes?

https://ta-jira-service-desk-simple-addon.readthedocs.io/en/latest/configuration.html#configuring-via-rest-api

[Evotiona]

@guilhemmarchand Thanks for the other options to try!

We did the bump command to no avail, as well as the cache clearing in browser/tried another browser and another workstation.

We also did the REST calls with cURL just like mentioned in the guide. We successfully made a change to the ta_jira_service_desk_simple_addon_settings.conf file on all search heads, but the password just saved in plain text and never created the passwords.conf file after reboot of each search head.

Let us know if you need any logs or screenshots or anything, but looking forward to that different framework that may be what we need lest any other thing we can try. Thanks again for the help!

[guilhemmarchand]

@Evotiona

Let me verify this, to my knowledge the password should not have been created in clear text, the REST API call is 100% the same thing than what the UI does, then splunkd handle the call and does the stuff.

Will come back on this shortly.

[Evotiona]

Thank you! Really do appreciate it; this app is solid gold and can't wait to have it help our processes out!

[guilhemmarchand]

@Evotiona

So I did a check:

• Brand new deployment with SHC on 8.1.1
• The TA is deployed in v1.0.30 from the SHC deployer, there is no local directory in the SHC deployer
• Once the bundle is pushed, the following CURL command is performed, locally on the search head targetting itself on the loopback
• The relevant config is created, with the password encrypted automatically

sh1 is the first member:

[splunk@sh1 ~]$ curl -k -u admin:'splunklab' -X POST https://localhost:8089/servicesNS/nobody/TA-jira-service-desk-simple-addon/TA_jira_service_desk_simple_addon_settings/additional_parameters -d 'jira_url=myjira.mydomain.com:8443' -d 'jira_username=admin' -d 'jira_password=ch@ngeM3'^C
[splunk@sh1 ~]$ cat /opt/splunk/etc/apps/TA-jira-service-desk-simple-addon/local/
passwords.conf                                   ta_jira_service_desk_simple_addon_settings.conf
[splunk@sh1 ~]$ cat /opt/splunk/etc/apps/TA-jira-service-desk-simple-addon/local/passwords.conf
[credential:__REST_CREDENTIAL__#TA-jira-service-desk-simple-addon#configs/conf-ta_jira_service_desk_simple_addon_settings:additional_parameters``splunk_cred_sep``1:]
password = $7$YWjRglJ+beer5HoweEp1/yoZfvmabZ/CyAvUwWIH3P7tLU/ePhRe87KhcNX6qSWM5Vry8tBdB9boWLDwtw==

[credential:__REST_CREDENTIAL__#TA-jira-service-desk-simple-addon#configs/conf-ta_jira_service_desk_simple_addon_settings:additional_parameters``splunk_cred_sep``2:]
password = $7$fs1Pv1VvfexZOSKSXxnmqb6jYt9antj108bdazVQsz+Yyr2/+rhE9b0RL25WeRr8D2+lehuqTRRYGiOW4WcNVXLLFQkQnzMIHLEv9qnnXwlWlLsiMKZDzoPoZKD78KW/oRGrBH/3Xf5DnPEPThHROVEWBM0ZvAsyFoOV8yZR0q1h49pWUBWV7E3L9ze/sIvc+C8NIrAlnxHuPBLZbNWrKL8C0HgLwlNTHTu8
[splunk@sh1 ~]$ cat /opt/splunk/etc/apps/TA-jira-service-desk-simple-addon/local/ta_jira_service_desk_simple_addon_settings.conf
[additional_parameters]
jira_password = ********
jira_url = myjira.mydomain.com:8443
jira_username = admin
[splunk@sh1 ~]$

This is not what you have seen right?

[Evotiona]

That's what it looks like on a single search head for us. For cluster, we followed your instructions.

• Completely removed traces of Jira app / rebooted
• Downloaded and installed fresh 1.0.30 version on the Deployer. No local directory
• Pushed bundle and rolled search heads
• Attempted to cURL request with following request + error (we get that "additional_parameters does not exist" error often; if we change the "localhost" to the dns name of the cluser "splunk.domain.com:8089" then it does work, but again, saves in cleartext)

"curl -k -u admin:'' -X POST https://localhost:8089/servicesNS/nobody/TA-jira-service-desk-simple-addon/TA_jira_service_desk_simple_addon_settings/additional_parameters -d 'jira_url=servicedesk.website.com' -d 'jira_username=' -d 'jira_password='"

07-07-2021 14:26:21.359 -0500 ERROR AdminManagerExternal - Stack trace from python handler:\nTraceback (most recent call last):\n File "/opt/splunk/lib/python3.7/site-packages/splunk/admin.py", line 151, in init\n hand.execute(info)\n File "/opt/splunk/lib/python3.7/site-packages/splunk/admin.py", line 638, in execute\n if self.requestedAction == ACTION_EDIT: self.handleEdit(confInfo)\n File "/opt/splunk/etc/apps/TA-jira-service-desk-simple-addon/bin/ta_jira_service_desk_simple_addon/aob_py3/splunktaucclib/rest_handler/admin_external.py", line 40, in wrapper\n for entity in result:\n File "/opt/splunk/etc/apps/TA-jira-service-desk-simple-addon/bin/ta_jira_service_desk_simple_addon/aob_py3/splunktaucclib/rest_handler/handler.py", line 117, in wrapper\n for name, data, acl in meth(self, *args, **kwargs):\n File "/opt/splunk/etc/apps/TA-jira-service-desk-simple-addon/bin/ta_jira_service_desk_simple_addon/aob_py3/splunktaucclib/rest_handler/handler.py", line 86, in wrapper\n check_existing(self, name),\n File "/opt/splunk/etc/apps/TA-jira-service-desk-simple-addon/bin/ta_jira_service_desk_simple_addon/aob_py3/splunktaucclib/rest_handler/handler.py", line 68, in check_existing\n '"%s" does not exist' % name,\nsplunktaucclib.rest_handler.error.RestError: REST Error [404]: Not Found -- "additional_parameters" does not exist\n

[guilhemmarchand]

Hum that is very suspiscious, to me it's clearly something wrong on a SHC side. For example, talking to the localhost (or the lookback IP via 127.0.0.1) should always work as expected, when you say :

"dns name of the cluser "splunk.domain.com:8089"

This means you are you doing the REST call using your load balancer or VIP (as such Splunk does not have a cluster address, only members from the same clusters that you can reach independently via an LB normally) But that could introduce additional issues due to the LB like session stickyness, or URL handling by the LB etc, so running the curl locally against the localhost is basically how Splunk Web talks to splunkd.

This explains as well why the UI gets blocked, if the REST endoint cannot be reached, then the JS gets stuck forever.

I am not very sure what kind of misconfiguration could lead to that on your side, I had a user with a very similar issue that was caused by reaching some limits of the number of apps they can run on the same SHC, do you have many apps? (thousdands of?)

Otherwise I could think about some custom tuning that has been made overtime, some apps causing issues on Splunk, some OS related settings (like a bad definition in /etc/hosts of the localhost / loopback)

I would suggest that you use btool to review settings applied on the SHC and that are not from system/etc/default, focus on servers.conf, limits.conf, restmaps.conf Identify and review each settings to understand what was set.

This isn't a Splunk core bug as such, likely introduced by settings on your side in my opinion, splunkd.log certainly contain some pieces of errors that could highlight it

[Evotiona]

Thanks for the insight! We are going to look into that configuration now and see what's happening. Will keep you posted. Really do appreciate the help!

[Evotiona]

Hey @guilhemmarchand

Thanks again for all the suggestions and help! Turns out, we fell plague to the "App-Builder Bad Passwords.conf" issue (https://www.gnzlabs.io/gnzlabs-blog/splunk-aes-gcm-decryption-failed/ for more details). After a lot of troubleshooting and going down the route of the cluster being an issue, we had a bad passwords.conf from another app hiding in there. Clearing that helped us out and we are good to go.

Thanks again, the app is stellar and your work and persistence on it shows. Excellent work, friend! 5/5 stars, would recommend again!

[guilhemmarchand]

Ohhhhhh!!!!

@Evotiona Well done! I am glad you found it out ;-)

And guess what, now that you say it... I experienced this issue myself with a customer earlier with a wrong password affecting other UIs, I believe in the TA for Palo Alto something like that!

Damned I shall have thought about it, I will try to keep it in mind next time!

Appreciated!

[Evotiona]

Haha that was our app as well, must have carried it over from years ago. Again, truly appreciate the help!

[guilhemmarchand]

XD ;-)

Will definitvely add a note about that.