Unexpected error: Expecting value: line 2 column 1 (char 1).

[awx-vsyr]

hello Julian,

I'm getting this every...100 or so alerts. (latest official version on splunkbase. I think v1.1.1 )

03:33:12.692 | 2023-02-20 03:33:12,692 ERROR pid=1 tid=MainThread file=cim_actions.py:message:431 \| sendmodaction - worker="<snip>.splunkcloud.com" signature="Unexpected error: Expecting value: line 2 column 1 (char 1)." action_name="forward_alert_to_splunk_hec" search_name="<snip>" sid="scheduler_c3ZjLWluZm9zZWMtc3BsdW5rLWFkc0BhaXJ3YWxsZXguY29t_SU5GT1NFQy1hcHA__RMD5158ebb72ae744ae4_at_1676863800_30210" rid="1" app="<snip>-app" user="<snip>" digest_mode="0" action_mode="saved" action_status="failure"host = <snip>.splunkcloud.com
source = /opt/splunk/var/log/splunk/forward_alert_to_splunk_hec_modalert.log
sourcetype = taalertforwarder:log

any ideas? cim_actions.py:message:431 :D that's not even a proper code line ? https://github.com/Stjubit/TA-alert_forwarder/blob/1.1.1/TA-alert_forwarder/bin/ta_alert_forwarder/cim_actions.py#L431

~/Downloads/TA-alert_forwarder_1.1.1.                                                                                                                                                                                              15:19:15
❯ egrep -iR "unexpected error" ./*/*/*/*/*.py
./bin/ta_alert_forwarder/aob_py3/splunktaucclib/alert_actions_base.py:            msg = "Unexpected error: {}."
~/Downloads/TA-alert_forwarder_1.1.1.                                                                                                                                                                                              15:19:18
❯ egrep -iR "unexpected error" ./*/*/*/*.py
~/Downloads/TA-alert_forwarder_1.1.1.                                                                                                                                                                                              15:19:21
❯ egrep -iR "unexpected error" ./*/*/*.py
./bin/ta_alert_forwarder/alert_actions_base.py:            msg = "Unexpected error: {}."
~/Downloads/TA-alert_forwarder_1.1.1.                                                                                                                                                                                              15:19:25
❯ egrep -iR "unexpected error" ./*/*.py
./bin/forward_alert_to_splunk_hec.py:            msg = "Unexpected error: {}."

i suppose - which file is hitting and why aren't we getting a proper traceback.

all of those have code along the lines of

./bin/forward_alert_to_splunk_hec.py:            msg = "Unexpected error: {}."
./bin/forward_alert_to_splunk_hec.py-            if e:
./bin/forward_alert_to_splunk_hec.py-                self.log_error(msg.format(str(e)))
./bin/forward_alert_to_splunk_hec.py-            else:
./bin/forward_alert_to_splunk_hec.py-                import traceback
./bin/forward_alert_to_splunk_hec.py-                self.log_error(msg.format(traceback.format_exc()))

Is it possible to get a 'fixed' debug build to self service install, or perhaps on appbase with the 'definitely proper traceback' enabled so we can troubleshoot this further

[Stjubit]

Hey, I have the same version running in 9 deployments and never saw this error in the past 90 days 🤔. The exception is caught in Add-on Builder generated code when calling the modalert:

def process_event(self, *args, **kwargs):
    status = 0
    try:
        if not self.validate_params():
            return 3
        status = modalert_forward_alert_to_splunk_hec_helper.process_event(self, *args, **kwargs)
    except (AttributeError, TypeError) as ae:
        self.log_error("Error: {}. Please double check spelling and also verify that a compatible version of Splunk_SA_CIM is installed.".format(str(ae)))
        return 4
    except Exception as e:
        msg = "Unexpected error: {}."
        if e:
            self.log_error(msg.format(str(e)))
        else:
            import traceback
            self.log_error(msg.format(traceback.format_exc()))
        return 5
    return status

That's the traceback message: Unexpected error: Expecting value: line 2 column 1 (char 1). So I'm pretty sure this is a JSONDecodeError.

Most probably, this is the line that throws the exception:

payload = {
    "event": json.dumps(field_list),
}

I assume that the search that triggers the alert does not return any events. Could you please check the search results of the search job that triggers the alert action?

[awx-vsyr]

hmmm doesn't look like it's 0 results.

this still happens for us roughly every 1 month

the latest one i caught before the sid expires and it looks normal. (as in normal sid expanding to 1 row of results)

I tried to pipe the alert into the index via sendalert but this doesn't seem possible? (or is this not supported/im doing it wrong)

| sendalert forward_alert_to_splunk_hec param.splunk_hec_target="infosec_splunk_hec" param.index="ads_alert_track"

signature="Unexpected error: Found multiple entities named ''; please specify a namespace.." action_name="forward_alert_to_splunk_hec" sid="1716371710.3623897 that's absolutely bizzare

any chance of a debug build at all?

[awx-vsyr]

@Stjubit Julian?