search

splunk / splunk-add-on-microsoft-azure

Splunk Add-on for Microsoft Azure
Apache License 2.0
11 stars 9 forks source link

Unable to obtain access token #24

Closed spijl closed 1 year ago

spijl commented 1 year ago

We use the TA for Azure Sign ins, Audit and Risk events for 20+ different Azure Tenants. After upgrading the app to 4.0.2. all inputs stopt working, when looking through the _internals we noticed these errors: ERROR pid=1579208 tid=MainThread file=base_modinput.py:log_error:316 | Splunk Unable to obtain access token

We checked the credentials with Postman but these worked fine and where able to obtain the data.

spijl commented 1 year ago

We just downgraded to 3.2.0 and the inputs started to work again. So it seems like a problem with the latest version of the TA.

JasonConger commented 1 year ago

Was this an in-place upgrade from version 3.x to 4.x?

The "unable to obtain access token" error occurs when the combination of client ID + client secret + tenant ID is incorrect. Since you have already verified the values, it's possible the client secret value(s) got corrupted during the upgrade. An issue with one of the libraries this add-on uses can lead to this behavior. The library has been patched and included in the add-on's next release (v4.0.3). https://github.com/splunk/splunk-add-on-microsoft-azure/blob/main/CHANGELOG.md.

brandon-gagliardi commented 1 year ago

@spijl , @JasonConger , I just emailed azure@splunk.com with what ended up being my solution:

We retained our existing inputs for AAD during an in place upgrade. The solution is to update every input you have for the new field (Endpoint) for the values, as v1 or beta was left as empty (null). I updated each with v1. IDK why I would use beta. This ended up being our issue. I’d recommend bubbling this up in the documentation or highlighting in an “if you’re upgrading” section assuming I didn’t just miss this note somewhere:

Errors we were seeing:

2022-10-03 10:46:54,519 ERROR pid=144691 tid=MainThread file=base_modinput.py:log_error:316 | Splunk Unable to obtain access token <--- This was my biggest red flag. 2022-10-05 13:03:53,246 ERROR pid=91330 tid=MainThread file=base_modinput.py:log_error:316 | Get error when collecting events. Traceback (most recent call last): File "/data/splunk/etc/apps/TA-MS-AAD/lib/splunktaucclib/modinput_wrapper/base_modinput.py", line 140, in stream_events self.collect_events(ew) File "/data/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_user.py", line 111, in collect_events response = azutils.get_items_batch_session(helper=helper, url=url, session=session) File "/data/splunk/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py", line 124, in get_items_batch_session raise e File "/data/splunk/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py", line 120, in get_items_batch_session r.raise_for_status() File "/data/splunk/etc/apps/TA-MS-AAD/lib/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://graph.microsoft.com/None/users

I'd say a big lesson learned on this is that when an upgrade leaves an input no longer working, disable the existing input and attempt to clone/recreate it and see what errors you get.