search

fastly / sigsci-splunk-app

Splunk app for Fastly (Signal Sciences)
https://splunkbase.splunk.com/app/3495
MIT License
9 stars 10 forks source link

SigSci Events throws KeyError when data collection is performed #32

Closed jeremy-cxf closed 1 year ago

jeremy-cxf commented 1 year ago

Reproduction steps:

Run Splunk Enterprise:

docker run -d -p 8000:8000 -e "SPLUNK_START_ARGS=--accept-license" -e "SPLUNK_PASSWORD=meep1234" --name splunk splunk/splunk:latest

Exec onto container. 

2023-06-06 20:09:02,873 ERROR pid=37632 tid=MainThread file=base_modinput.py:log_error:309 | Get error when collecting events.
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/sigsci_TA_for_splunk/bin/sigsci_ta_for_splunk/aob_py3/modinput_wrapper/base_modinput.py", line 128, in stream_events
    self.collect_events(ew)
  File "/opt/splunk/etc/apps/sigsci_TA_for_splunk/bin/SigsciEvent.py", line 64, in collect_events
    input_module.collect_events(self, ew)
  File "/opt/splunk/etc/apps/sigsci_TA_for_splunk/bin/input_module_SigsciEvent.py", line 222, in collect_events
    delta=int(time_deltas)
  File "/opt/splunk/etc/apps/sigsci_TA_for_splunk/bin/input_module_SigsciEvent.py", line 138, in pull_events
    all_events = get_results("Events", helper, config)
  File "/opt/splunk/etc/apps/sigsci_TA_for_splunk/bin/sigsci_helper.py", line 175, in get_results
    headers_out = data['headersOut']
KeyError: 'headersOut'

Where 'headersOut' is not found in the appropriate 'data' dictionary, resulting in a KeyError. This seems to be a regression of the changes in the latest release.

jeremy-cxf commented 1 year ago

Changing sigsci_helper to use the dictionary get method and changing the behaviour to where if the key is not found, an empty list [] is returned, allows the event to be populated, and I can see events in the index after:

            headers_out = data.get('headersOut', []) 
            headers_in = data.get('headersIn', [])
2023-06-06 21:14:27,388 INFO pid=3722 tid=MainThread file=base_modinput.py:log_info:295 | corp: jcocks
2023-06-06 21:14:27,388 INFO pid=3722 tid=MainThread file=base_modinput.py:log_info:295 | interval: 5
2023-06-06 21:14:27,388 INFO pid=3722 tid=MainThread file=base_modinput.py:log_info:295 | Run Type: Concurrent
2023-06-06 21:14:27,388 INFO pid=3722 tid=MainThread file=base_modinput.py:log_info:295 | site: chungus
2023-06-06 21:14:27,389 INFO pid=3722 tid=MainThread file=splunk_rest_client.py:_request_handler:99 | Use HTTP connection pooling
2023-06-06 21:14:27,404 INFO pid=3722 tid=MainThread file=base_modinput.py:log_info:295 | last_run_until: 1686086062
2023-06-06 21:14:27,436 INFO pid=3722 tid=MainThread file=base_modinput.py:log_info:295 | SiteName: chungus
2023-06-06 21:14:27,436 INFO pid=3722 tid=MainThread file=base_modinput.py:log_info:295 | Start Period: 2023-06-06 21:14:22
2023-06-06 21:14:27,436 INFO pid=3722 tid=MainThread file=base_modinput.py:log_info:295 | End Period: 2023-06-06 21:14:27
2023-06-06 21:14:27,436 INFO pid=3722 tid=MainThread file=base_modinput.py:log_info:295 | Single instance mode
2023-06-06 21:14:27,436 INFO pid=3722 tid=MainThread file=base_modinput.py:log_info:295 | Inputs: chungusEvents
2023-06-06 21:14:27,436 INFO pid=3722 tid=MainThread file=base_modinput.py:log_info:295 | Inputs Num: 13
2023-06-06 21:14:27,436 INFO pid=3722 tid=MainThread file=base_modinput.py:log_info:295 | single_name: chungusEvents
2023-06-06 21:14:27,436 INFO pid=3722 tid=MainThread file=base_modinput.py:log_info:295 | Pulling results from Events API
2023-06-06 21:14:27,436 INFO pid=3722 tid=MainThread file=base_modinput.py:log_info:295 | Processing page 1
2023-06-06 21:14:27,436 INFO pid=3722 tid=MainThread file=setup_util.py:log_info:117 | Proxy is not enabled!
2023-06-06 21:14:28,279 INFO pid=3722 tid=MainThread file=base_modinput.py:log_info:295 | Number of Events for Page: 1
2023-06-06 21:14:28,279 INFO pid=3722 tid=MainThread file=base_modinput.py:log_info:295 | Data: {'id': '647fa1aedb8096f291edb39a', 'eventType': 'loggingModeChanged', 'msgData': {'mode': 'block', 'oldMode': 'log'}, 'message': 'jeremy (xxx@fastly.com) changed agent mode from "log" to "block"', 'attachments': [], 'created': '2023-06-06T21:14:22Z'}
2023-06-06 21:14:28,279 INFO pid=3722 tid=MainThread file=base_modinput.py:log_info:295 | Total Events Pulled: 1
2023-06-06 21:14:28,279 INFO pid=3722 tid=MainThread file=base_modinput.py:log_info:295 | Concurrent Mode
2023-06-06 21:14:28,279 INFO pid=3722 tid=MainThread file=base_modinput.py:log_info:295 | source_type: sigsci-event
2023-06-06 21:14:28,279 INFO pid=3722 tid=MainThread file=base_modinput.py:log_info:295 | source_type_info: <class 'str'>
2023-06-06 21:14:28,279 INFO pid=3722 tid=MainThread file=base_modinput.py:log_info:295 | index: default
2023-06-06 21:14:28,280 INFO pid=3722 tid=MainThread file=base_modinput.py:log_info:295 | index_info: <class 'str'>
2023-06-06 21:14:28,280 INFO pid=3722 tid=MainThread file=base_modinput.py:log_info:295 | single_name: chungusEvents
2023-06-06 21:14:28,280 INFO pid=3722 tid=MainThread file=base_modinput.py:log_info:295 | single_name_info: <class 'str'>
2023-06-06 21:14:28,280 INFO pid=3722 tid=MainThread file=base_modinput.py:log_info:295 | current_event: "{\"id\": \"647fa1aedb8096f291edb39a\", \"eventType\": \"loggingModeChanged\", \"msgData\": {\"mode\": \"block\", \"oldMode\": \"log\"}, \"message\": \"jeremy (xxx@fastly.com) changed agent mode from \\\"log\\\" to \\\"block\\\"\", \"attachments\": [], \"created\": \"2023-06-06T21:14:22Z\", \"headersOut\": [], \"headersIn\": []}"
2023-06-06 21:14:28,280 INFO pid=3722 tid=MainThread file=base_modinput.py:log_info:295 | current_event_info: <class 'str'>
2023-06-06 21:14:28,280 INFO pid=3722 tid=MainThread file=base_modinput.py:log_info:295 | Total Event Output Time: 0.0 seconds
2023-06-06 21:14:28,281 INFO pid=3722 tid=MainThread file=base_modinput.py:log_info:295 | Finished Pulling Events for chungus
2023-06-06 21:14:28,281 INFO pid=3722 tid=MainThread file=base_modinput.py:log_info:295 | Total Script Time: 0.89 seconds

Will raise PR, however it is more of a quick win. I am assuming those keys are most likely not going to be present in the event data.

brectanus-sigsci commented 1 year ago

@dacoburn Can you take a look at this? Will need a new release if this looks good.