KQL help

[rolltidega]

I am hoping to get some help for the new KQL queries option. We are currently using the older Microsoft Log Analytics app and I am trying to test one of these Log Analytics queries in the new KQL input in the Azure add on. No matter what I try, I am getting this error in the logs:

Traceback (most recent call last): File "/opt/splunk/etc/apps/TA-MS-AAD/lib/splunktaucclib/modinput_wrapper/base_modinput.py", line 140, in stream_events self.collect_events(ew) File "/opt/splunk/etc/apps/TA-MS-AAD/bin/azure_kql.py", line 173, in collect_events helper.log_error("Splunk input_name=%s error geting data: %s") % (input_name, str(e)) TypeError: unsupported operand type(s) for %: 'NoneType' and 'tuple'

I have no idea what this means or what to do to get past it. Any help would be appreciated.

[JasonConger]

Can you share your inputs.conf section dealing with the KQL input? It should look something like this:

[azure_kql://kql]
azure_app_account = your_account
environment = public
index = your_index
index_stats = 1
interval = 600
kql_query = AzureActivity | where TimeGenerated between (now(-30d)..now())
source_type = azure:kql
tenant_id = your_tenant_id
workspace_id = your_workspace_id

[rolltidega]

[azure_kql://AzureActivityLog] azure_app_account = accountname environment = public index = test index_stats = false interval = 300 kql_query = AzureActivity | where ResourceProvider == "Microsoft.Compute" and ResourceGroup == "rg01" source_type = azure:kql tenant_id = --redacted-- workspace_id = --redacted--

[rolltidega]

I have tried it with just AzureActivity as the query with nothing after it.

[rolltidega]

Looks like I have more errors right before the original one, seems there is a read timeout? any idea what would cause that?

2022-07-27 13:52:41,611 ERROR pid=13233 tid=MainThread file=base_modinput.py:log_error:316 | Splunk input_name=%s error ge ting data: %s 2022-07-27 13:52:41,614 ERROR pid=13233 tid=MainThread file=base_modinput.py:log_error:316 | Get error when collecting event s. Traceback (most recent call last): File "/opt/splunk/etc/apps/TA-MS-AAD/lib/urllib3/connectionpool.py", line 449, in _make_request six.raise_from(e, None) File "", line 3, in raise_from File "/opt/splunk/etc/apps/TA-MS-AAD/lib/urllib3/connectionpool.py", line 444, in _make_request httplib_response = conn.getresponse() File "/opt/splunk/lib/python3.7/http/client.py", line 1373, in getresponse response.begin() File "/opt/splunk/lib/python3.7/http/client.py", line 319, in begin version, status, reason = self._read_status() File "/opt/splunk/lib/python3.7/http/client.py", line 280, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "/opt/splunk/lib/python3.7/socket.py", line 589, in readinto return self._sock.recv_into(b) File "/opt/splunk/lib/python3.7/ssl.py", line 1071, in recv_into return self.read(nbytes, buffer) File "/opt/splunk/lib/python3.7/ssl.py", line 929, in read return self._sslobj.read(len, buffer) socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/opt/splunk/etc/apps/TA-MS-AAD/lib/requests/adapters.py", line 499, in send timeout=timeout, File "/opt/splunk/etc/apps/TA-MS-AAD/lib/urllib3/connectionpool.py", line 788, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] File "/opt/splunk/etc/apps/TA-MS-AAD/lib/urllib3/util/retry.py", line 550, in increment raise six.reraise(type(error), error, _stacktrace) File "/opt/splunk/etc/apps/TA-MS-AAD/lib/urllib3/packages/six.py", line 770, in reraise raise value File "/opt/splunk/etc/apps/TA-MS-AAD/lib/urllib3/connectionpool.py", line 710, in urlopen chunked=chunked, File "/opt/splunk/etc/apps/TA-MS-AAD/lib/urllib3/connectionpool.py", line 451, in _make_request self._raise_timeout(err=e, url=url, timeout_value=read_timeout) File "/opt/splunk/etc/apps/TA-MS-AAD/lib/urllib3/connectionpool.py", line 341, in _raise_timeout self, url, "Read timed out. (read timeout=%s)" % timeout_value urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='api.loganalytics.io', port=443): Read timed out. (read timeout=5)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/opt/splunk/etc/apps/TA-MS-AAD/bin/azure_kql.py", line 124, in collect_events response = azutils.post_items_batch_session(helper=helper, url=url, headers=headers, data=json.dumps(data), session=session) File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py", line 135, in post_items_batch_session raise e File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py", line 130, in post_items_batch_session r = requests_retry_session(session=session).post(url=url, headers=headers, data=data, timeout=TIMEOUT) File "/opt/splunk/etc/apps/TA-MS-AAD/lib/requests/sessions.py", line 635, in post return self.request("POST", url, data=data, json=json, kwargs) File "/opt/splunk/etc/apps/TA-MS-AAD/lib/requests/sessions.py", line 587, in request resp = self.send(prep, send_kwargs) File "/opt/splunk/etc/apps/TA-MS-AAD/lib/requests/sessions.py", line 701, in send r = adapter.send(request, **kwargs) File "/opt/splunk/etc/apps/TA-MS-AAD/lib/requests/adapters.py", line 578, in send raise ReadTimeout(e, request=request) requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.loganalytics.io', port=443): Read timed out. (read timeout=5)

[rolltidega]

ran again with DEBUG logging. Seems like a 5sec timeout is too short?

2022-07-27 13:59:43,326 DEBUG pid=14522 tid=MainThread file=base_modinput.py:log_debug:298 | Splunk Getting proxy server. 2022-07-27 13:59:43,326 INFO pid=14522 tid=MainThread file=setup_util.py:log_info:142 | Proxy is not enabled! 2022-07-27 13:59:43,327 DEBUG pid=14522 tid=MainThread file=connectionpool.py:_new_conn:1007 | Starting new HTTPS connection (1): login.microsoftonline.com:443 2022-07-27 13:59:43,667 DEBUG pid=14522 tid=MainThread file=connectionpool.py:_make_request:465 | https://login.microsoftonl ine.com:443 "POST /guidhere/oauth2/token HTTP/1.1" 200 1416 2022-07-27 13:59:43,670 DEBUG pid=14522 tid=MainThread file=base_modinput.py:log_debug:298 | Splunk Getting proxy server. 2022-07-27 13:59:43,670 INFO pid=14522 tid=MainThread file=setup_util.py:log_info:142 | Proxy is not enabled! 2022-07-27 13:59:43,670 DEBUG pid=14522 tid=MainThread file=base_modinput.py:log_debug:298 | Splunk input_name=AzureActivi tyLog Collecting Log Analytics data. sourcetype='azure:kql' 2022-07-27 13:59:43,671 DEBUG pid=14522 tid=MainThread file=connectionpool.py:_new_conn:1007 | Starting new HTTPS connection (1): api.loganalytics.io:443 2022-07-27 13:59:48,744 DEBUG pid=14522 tid=MainThread file=base_modinput.py:log_debug:298 | Splunk filename: /opt/splunk/ etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py, module: post_items_batch_session, execution time: 5.073941946029663

[JasonConger]

The default timeout is 5 seconds. You can change this in $SPLUNK_HOME/etc/apps/TA-MS-AAD/bin/ta_auzre_utils/utils.py. Around line 29, change TIMEOUT = 5 #seconds.

I'll get this timeout moved to a user-configurable parameter in the UI.

[rolltidega]

Great the timeout worked. Can a start date field be added as well to the UI? The data that I pulled in for my test was from February 2022. I would need to pull in starting at a certain date and then when it runs the next time pull in the new data since the last pull. Unless there is a query parameter I can add to the KQL for this. I dont know KQL too well.

[SgtMoose]

We are seeing the same timeout error. I'll have to see if we can modify the timeout in the Python file mentioned via support since we are operating in Splunk cloud and don't have direct access. @rolltidega What timeout value did you end up using to get past the timeout issue? This seemed to break for us late July/early August and was working prior to that.