[BUG]

[mattypete]

Request Type

Bug

Work Environment

Question	Answer
OS version (server)	Splunk Cloud Version: 9.1.2308.203
TheHive version / git hash	TheHive Cloud Platform v5.2.10-1
TheHive-Cortex Splunk App version	3.3

Problem Description

I’m attempting to use the TheHive-Cortex Splunk app to create alerts with the TheHive - Create a New Alert adaptive response action. I’m trying to troubleshoot why only a portion of the alerts are getting created. Unfortunately, I only see half the puzzle from the Splunk side’s audit logs (attached). The error message I’m seeing just states “TheHive alert creation failed.” I've also opened a support ticket with StrangeBee to get their input. However, I’m suspicious that it has to do with the “SID” field as it is used for the Reference ID in the alert in TheHive. I’m speculating that the reuse of the same “sid” value when the adaptive response action is configured to use ES_MODE across multiple alerts is causing the alert creation failure, but I’d like your team to review and give your input as well before I reach out to the developer of the TheHive-Cortex app for Splunk.

Steps to Reproduce

1. Run a stats search that returns more than one row and save it as a Splunk correlation search
2. Configure the "TheHive - Create a New Alert" adaptive response action and configure the "ES notable mode" alert mode
3. Once the correlation search finds a match with multiple results, it will attempt to send the alert to TheHive with the same SID (TheHive Reference ID) and only create one alert instead of one per row of the search results
4. Observe the audit log for the corresponding SID and you'll notice all but one of the processes to create an alert fail with error "TheHive alert creation has failed"
5. Alternatively, if you use the "sendalert" command for "thehive_create_a_new_alert" parameter (and all the other required parameters for it to successfully send the alert to TheHive) and the param.alert_mode="es_mode" at the end of your search in step 1 above, the process will only send a single alert to TheHive and you'll observe the error mentioned in the description above for every other row in the search output from step 1 above as the process attempts to send an alert to TheHive for each result in the search using the same SID for the Reference ID in TheHive.

Possible Solutions

• Configure the "thehive_create_a_new_alert" action and the "TheHive - Create a new alert" adaptive response action to use a unique SID when sending an alert for each result in the scheduled or manually executed search
• Configure the "thehive_create_a_new_alert" action and the "TheHive - Create a new alert" adaptive response action to generate a unique reference ID when sending an alert for each result in the scheduled or manually executed search

Logs (issued from the search.log with logging mode set to DEBUG under Settings/Configuration)

• I can make this available, privately, upon request. The details in the debug logs are a potential security risk when posted here, publicly. It is also a violation of my company's policies to share these details without an NDA. However, the steps above should explain the issue well enough and are easy enough to reproduce that logs shouldn't be required.

[mattypete]

Here's an example of the error StrangeBee is seeing on their side of the failure/error events in the audit log:

Example of how to understand the following line: org.thp.scalligraph.CreateError: Alert alert:splunk:1708032930.251131 already exists in organisation ORG1 The alert with the following characteristics already exist in TheHive: type: alert source: splunk sourceRef: 1708032930.251131

I misspoke in my original post. When I mentioned "reference ID" I was referring to the "sourceRef" field in TheHive. That field cannot be duplicated across alerts, but the current "es_mode" for the alert mode of TheHive-Cortex app is attempting to reuse that field for every row of the search results when creating new alerts in TheHive using the Splunk app.

[mattypete]

I was reviewing the code in your modalert_thehive_common.py script and realized that if I calculated a unique value for field "event_id" and listed that in my stats command so it would show in my statistics table in Splunk before using the sendalert command as mentioned in step 5 of my original post, then I was able to send all search results/rows as individual alerts successfully (i.e., if my search returned 8 results/rows, I was finally able to send 8 individual alerts to TheHive) when using the param.alert_mode=es_mode in my sendalert command.

My eval statement is as follows: | eval event_id=$_cd$

_cd is a unique event-specific field assigned to each event by Splunk. Granted, that detail is based on 10 minutes of research, but it appears to be true.

[LetMeR00t]

Hello @mattypete , Thank you for raising this issue. I wasn't able to reproduce your error as on my side I was able to set up a correlation search with an adaptative response to create an alert (or a case as it worked too).

Just to make sure on my side:

• Did you checked the logs provided in the Audit logs dashboard?
• Can you provide the details about your search settings and the SPL content (with fake data using a makeresults command but keeping the logic if possible)?

For the data, I fully understand the issue and I would advise you to send me any log only by email (letmer00t@gmail.com) and removed from any internal data. You can simply start by sending all logs related to a WARNING or ERROR with sensitive information removed and we can discuss further by email.

Thank you

[LetMeR00t]

I was reviewing the code in your modalert_thehive_common.py script and realized that if I calculated a unique value for field "event_id" and listed that in my stats command so it would show in my statistics table in Splunk before using the sendalert command as mentioned in step 5 of my original post, then I was able to send all search results/rows as individual alerts successfully (i.e., if my search returned 8 results/rows, I was finally able to send 8 individual alerts to TheHive) when using the param.alert_mode=es_mode in my sendalert command.

My eval statement is as follows: | eval event_id=$_cd$

_cd is a unique event-specific field assigned to each event by Splunk. Granted, that detail is based on 10 minutes of research, but it appears to be true.

By the way, did you tried with the "Alert mode" set to "Alert action mode"?

[LetMeR00t]

Hi @mattypete

Any update on this please?

Thank you

[mattypete]

Sorry for the delay. Life's been busy and I haven't had much time to work on this during work hours.

I tried using "Alert action mode" for my adaptive response action, but it then created a single alert with all the results from my search when I am wanting a single alert for every result.

I understand you weren't able to recreate the issue. However, I started having the same issue even with my custom eval statement to set the event_id field value ( | eval event_id=$_cd$). The issue started happening again as I presume Splunk recycles the _cd value applied to events. To get around that, I was able to change my eval statement to produce a nearly impossible repeatable result as such (note: pID is just a process ID for the returned result): | eval event_id=(random() * _time) / pID

As far as recreating the issue, you should be able to use the following to recreate the issue. Example one will recreate the issue, while Example 2 is an example of what I did to workaround the issue.

Example 1

In the following example a single alert will be created in TheHive containing the values of all the fields in the results even though the param.alert_mode value is set to "es_mode": | makeresults count=4 | streamstats count | eval age = case(count=1, 25, count=2, 39, count=3, 31, count=4, null()) | eval city = case(count=1 OR count=3, "San Francisco", count=2 OR count=4, "New York") | eval pID = round((random() % 9000) + 1000) | fillnull value="none" | eval pID:ignore_similarity=1, city:ignore_similarity=1, age:ignore_similarity=1 | stats values(city) as city values(age) as age values(user:ignore_similarity) as user:ignore_similarity values(pID:ignore_similarity) as pID:ignore_similarity values(city:ignore_similarity) as city:ignore_similarity values(age:ignore_similarity) as age:ignore_similarity count by pID | eval thehive_title="This is a generic alert for testing" | eval thehive_description="This is a generic value for an alert description within TheHive" | sendalert thehive_create_a_new_alert param.thehive_instance_id=12345678 param.alert_mode="es_mode" param.unique_id_field="" param.case_template="Template 1" param.type="alert" param.source="splunk" param.timestamp_field="" param.title=thehive_title param.description=thehive_description param.tags="Testing" param.scope=1 param.severity=2 param.tlp=4 param.pap=3

Example 2

In the following example, multiple alerts will be created in TheHive with each alert only containing the value of it's specific row (the way es_mode is intended to function, if my understanding is correct): *| makeresults count=4 | streamstats count | eval age = case(count=1, 25, count=2, 39, count=3, 31, count=4, null()) | eval city = case(count=1 OR count=3, "San Francisco", count=2 OR count=4, "New York") | eval pID = round((random() % 9000) + 1000) | eval event_id=(random() _time) / pID | fillnull value="none" | eval pID:ignore_similarity=1, city:ignore_similarity=1, age:ignore_similarity=1 | stats sum(event_id) as event_id values(city) as city values(age) as age values(user:ignore_similarity) as user:ignore_similarity values(pID:ignore_similarity) as pID:ignore_similarity values(city:ignore_similarity) as city:ignore_similarity values(age:ignore_similarity) as age:ignore_similarity count by pID | eval thehive_title="This is a generic alert for testing" | eval thehive_description="This is a generic value for an alert description within TheHive" | sendalert thehive_create_a_new_alert param.thehive_instance_id=12345678 param.alert_mode="es_mode" param.unique_id_field="" param.case_template="Template 1" param.type="alert" param.source="splunk" param.timestamp_field="" param.title=thehive_title param.description=thehive_description param.tags="Testing" param.scope=1 param.severity=2 param.tlp=4 param.pap=3**

[mattypete]

Hi @LetMeR00t,

Just checking in to see if the information I provided is what you were looking for. Anything else you need from me at this time?

We're looking to make some changes to our Splunk searches to make sure we are accounting for the event_id issue and are at a point where we need to either wait for a fix or implement our workaround.

Any update you might have would be appreciated. Thanks for your work on this Splunk app thus far. It's been super helpful for us.

[LetMeR00t]

Hi Thank you for the feedback and the precious details I need to take some time to work on it and I’ll try to do so as soon as possible I’ll keep you posted

[LetMeR00t]

Hi @mattypete , I just went through your two examples. For what I understand, you want natively to create one alert per row in your first example. As it's only creating one alert with all the rows, you did the example 2 workaround to have what you expected.

If so, please note that yes, example 2 is working because you are doing the expected behavior when notables from Splunk ES are processed. Nevertheless, you can easily do the same without that. You shall have noticed that in the parameters of the custom alert action, there was a "unique_id_field" parameter which is used to indicates which field shall be used to consider the uniqueness of the alert.

Please take this example as the expected solution:

| makeresults count=4
| streamstats count
| eval age = case(count=1, 25, count=2, 39, count=3, 31, count=4, null())
| eval city = case(count=1 OR count=3, "San Francisco", count=2 OR count=4, "New York")
| eval pID = round((random() % 9000) + 1000)
| eval mycustomuniquefield=(random() * _time)
| fillnull value="none"
| eval pID:ignore_similarity=1, city:ignore_similarity=1, age:ignore_similarity=1
| stats values(mycustomuniquefield) as mycustomuniquefield, values(city) as city values(age) as age values(user:ignore_similarity) as user:ignore_similarity values(pID:ignore_similarity) as pID:ignore_similarity values(city:ignore_similarity) as city:ignore_similarity values(age:ignore_similarity) as age:ignore_similarity count by pID
| eval thehive_title="This is a generic alert for testing (test #1)"
| eval thehive_description="This is a generic value for an alert description within TheHive"
| sendalert thehive_create_a_new_alert param.thehive_instance_id=12345678 param.alert_mode="es_mode" param.unique_id_field="mycustomuniquefield" param.case_template="Template 1" param.type="alert" param.source="splunk" param.timestamp_field="" param.title=thehive_title param.description=thehive_description param.tags="Testing" param.scope=1 param.severity=2 param.tlp=4 param.pap=3

Please note the param.unique_id_field="mycustomuniquefield"

Does it solves your issue? If not, please provide me your feedback

Thank you

[LetMeR00t]

Hello @mattypete Any update on your side ? Was it helpful ? Thank you

[LetMeR00t]

Considered as closed If needed, please reopen an issue