[Support] External search command 'thehivecases' returned error code 1.

[mekhaleraj]

Request Type

Support

Work Environment

Question	Answer
OS version (server)
TheHive version / git hash	5

Problem Description

Not able to pull/push data from Splunk to Thehive and Cortex

Steps to Reproduce

1. Install App
2. Configure Accounts
3. Add Instances
4. Try to fetch data

Possible Solutions

-

Logs (issued from the search.log with logging mode set to DEBUG under Settings/Configuration)

• No Audit logs generated

[LetMeR00t]

Hi Are you using a HTTPS instance ? Could you define the architecture on which you are working on ? Did you enabled the DEBUG logging mode ? Thank you

[mekhaleraj]

Hi,

I'm using HTTPS instance. Splunk version 9. Both Splunk and TheHive instance running in different subnets. Traffic is allowed between them as I can run Cortex analysers. DEBUG mode is enabled. I can see only Instance created logs under Audit Log section.

I don't see any request hitting thehive or cortex instance from splunk/ splunk api user.

[LetMeR00t]

Okay so you’re on the LAN then. Once you’ve configured the instances and try to make a call using the dashboard or the custom alert action you should have logs coming in, overall with DEBUG What was your test for your connection ? how did you do it and with which data ? did you try a tcpdump on the Splunk side to see if the request is going out Splunk at least ? Thank you

[mekhaleraj]

We have self signed certificates installed as these are internal applications.

I tried alerts/cases pull request. Cortex pull request. Hive Alert creation with run function and create new alert. I did tests from dashboard and from alerts section with API key user configured.

I can see "[S5] New instance detected, getting {data}" in audit log but no other logs. Strange behaviour observed in logs: Even if I send request to TheHive, audit log shows cortex entry.

[marcnil815]

I have the same error, and I am trying to narrow it down.

@mekhaleraj Missing logs: you need to make sure that you have at least once saved the logging configuration. That will move the ta_thehive_cortex_settings.conf file from default to local (the app fails if it is not in local).

However, my problem is right now that no matter what I do with the account that is added, the app complains in common.py

This is from an instance where we upgraded (and rebooted the SH)

05-02-2023 13:01:03.016 ERROR ScriptRunner [193491 phase_1] - stderr from '/pack/splunk/bin/python3.7 /pack/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py c18618bf':  Traceback (most recent call last):
05-02-2023 13:01:03.016 ERROR ScriptRunner [193491 phase_1] - stderr from '/pack/splunk/bin/python3.7 /pack/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py c18618bf':    File "/pack/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py", line 29, in <module>
05-02-2023 13:01:03.016 ERROR ScriptRunner [193491 phase_1] - stderr from '/pack/splunk/bin/python3.7 /pack/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py c18618bf':      (thehive, configuration, defaults, logger) = initialize_thehive_instance(keywords, settings ,logger_name="thehive_search_cases")
05-02-2023 13:01:03.016 ERROR ScriptRunner [193491 phase_1] - stderr from '/pack/splunk/bin/python3.7 /pack/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py c18618bf':    File "/pack/splunk/etc/apps/TA-thehive-cortex/bin/thehive.py", line 22, in initialize_thehive_instance
05-02-2023 13:01:03.016 ERROR ScriptRunner [193491 phase_1] - stderr from '/pack/splunk/bin/python3.7 /pack/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py c18618bf':      return create_thehive_instance(instance_id, settings, logger)
05-02-2023 13:01:03.016 ERROR ScriptRunner [193491 phase_1] - stderr from '/pack/splunk/bin/python3.7 /pack/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py c18618bf':    File "/pack/splunk/etc/apps/TA-thehive-cortex/bin/thehive.py", line 30, in create_thehive_instance
05-02-2023 13:01:03.016 ERROR ScriptRunner [193491 phase_1] - stderr from '/pack/splunk/bin/python3.7 /pack/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py c18618bf':      configuration = Settings(spl, settings, logger)
05-02-2023 13:01:03.016 ERROR ScriptRunner [193491 phase_1] - stderr from '/pack/splunk/bin/python3.7 /pack/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py c18618bf':    File "/pack/splunk/etc/apps/TA-thehive-cortex/bin/common.py", line 49, in __init__
05-02-2023 13:01:03.016 ERROR ScriptRunner [193491 phase_1] - stderr from '/pack/splunk/bin/python3.7 /pack/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py c18618bf':      if "password" in credential['clear_password']:
05-02-2023 13:01:03.016 ERROR ScriptRunner [193491 phase_1] - stderr from '/pack/splunk/bin/python3.7 /pack/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py c18618bf':  TypeError: argument of type 'NoneType' is not iterable

I installed another instance, complete clean (i.e. TA-thehive-cortex has never been installed before). So this is from an instance that is fresh

05-02-2023 12:45:43.697 ERROR ScriptRunner [84736 phase_1] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py aa16d12d':  Traceback (most recent call last):
05-02-2023 12:45:43.698 ERROR ScriptRunner [84736 phase_1] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py aa16d12d':    File "/opt/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py", line 29, in <module>
05-02-2023 12:45:43.698 ERROR ScriptRunner [84736 phase_1] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py aa16d12d':      (thehive, configuration, defaults, logger) = initialize_thehive_instance(keywords, settings ,logger_name="thehive_search_cases")
05-02-2023 12:45:43.698 ERROR ScriptRunner [84736 phase_1] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py aa16d12d':    File "/opt/splunk/etc/apps/TA-thehive-cortex/bin/thehive.py", line 22, in initialize_thehive_instance
05-02-2023 12:45:43.698 ERROR ScriptRunner [84736 phase_1] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py aa16d12d':      return create_thehive_instance(instance_id, settings, logger)
05-02-2023 12:45:43.698 ERROR ScriptRunner [84736 phase_1] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py aa16d12d':    File "/opt/splunk/etc/apps/TA-thehive-cortex/bin/thehive.py", line 30, in create_thehive_instance
05-02-2023 12:45:43.698 ERROR ScriptRunner [84736 phase_1] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py aa16d12d':      configuration = Settings(spl, settings, logger)
05-02-2023 12:45:43.698 ERROR ScriptRunner [84736 phase_1] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py aa16d12d':    File "/opt/splunk/etc/apps/TA-thehive-cortex/bin/common.py", line 51, in __init__
05-02-2023 12:45:43.698 ERROR ScriptRunner [84736 phase_1] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py aa16d12d':      self._passwords[username] = clear_password["password"]
05-02-2023 12:45:43.698 ERROR ScriptRunner [84736 phase_1] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py aa16d12d':  KeyError: 'password'

I tried multiple times creating the account to make sure that a user is present. Everything seems to be there...

4 -rw-------  1 splunk splunk  621 May  2 12:15 passwords.conf
4 -rw-------  1 splunk splunk   72 May  2 12:15 ta_thehive_cortex_account.conf
4 -rw-------  1 splunk splunk   27 May  2 12:37 ta_thehive_cortex_settings.conf

[marcnil815]

The app seems to iterate through all usernames, outside the TA-thehive-cortex namespace, and that's what borks it up....(best guess by stepping through).

[marcnil815]

Ah...then it hits a user without a password and cannot handle it.

So:

1. The app should keep in it's namespace (if that's a thing).
2. It cannot handle "empty" passwords

[marcnil815]

common.py lines 55, wrapped in a try/except works:

 55                         try:
 56                             self._passwords[username] = clear_password["password"]
 57                         except: pass

[LetMeR00t]

Hello @mekhaleraj , @marcnil815 , I provided a recent fix for both of you on the related issues you are triggering at the moment and it's provided in the 'develop' branch. If you want, you can directly replace those lines within your code to see the effect (and not wait for the Splunk Cloud vet)

Let me know if it helped you to solve your issue

See https://github.com/LetMeR00t/TA-thehive-cortex/compare/main...develop for the complete changes to be pushed as soon as possible on Splunkbase

[marcnil815]

@LetMeR00t implemented your changes regarding the namespace (which tbh was a long shot). I still get these errors:

05-02-2023 19:20:02.134 ERROR ScriptRunner [852561 phase_1] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py aa16d12d':  Traceback (most recent call last):
05-02-2023 19:20:02.134 ERROR ScriptRunner [852561 phase_1] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py aa16d12d':    File "/opt/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py", line 29, in <module>
05-02-2023 19:20:02.134 ERROR ScriptRunner [852561 phase_1] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py aa16d12d':      (thehive, configuration, defaults, logger) = initialize_thehive_instance(keywords, settings ,logger_name="thehive_search_cases")
05-02-2023 19:20:02.134 ERROR ScriptRunner [852561 phase_1] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py aa16d12d':    File "/opt/splunk/etc/apps/TA-thehive-cortex/bin/thehive.py", line 22, in initialize_thehive_instance
05-02-2023 19:20:02.134 ERROR ScriptRunner [852561 phase_1] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py aa16d12d':      return create_thehive_instance(instance_id, settings, logger)
05-02-2023 19:20:02.134 ERROR ScriptRunner [852561 phase_1] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py aa16d12d':    File "/opt/splunk/etc/apps/TA-thehive-cortex/bin/thehive.py", line 31, in create_thehive_instance
05-02-2023 19:20:02.134 ERROR ScriptRunner [852561 phase_1] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py aa16d12d':      configuration = Settings(spl, settings, logger)
05-02-2023 19:20:02.134 ERROR ScriptRunner [852561 phase_1] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py aa16d12d':    File "/opt/splunk/etc/apps/TA-thehive-cortex/bin/common.py", line 56, in __init__
05-02-2023 19:20:02.134 ERROR ScriptRunner [852561 phase_1] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py aa16d12d':      self._passwords[username] = clear_password["password"]
05-02-2023 19:20:02.134 ERROR ScriptRunner [852561 phase_1] - stderr from '/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-thehive-cortex/bin/thehive_search_cases.py aa16d12d':  KeyError: 'password'

So, what seems to happen (probably because I have elevated rights), is the code is hitting the following password in the json blobs that are returned (I inserted several more logging statements to make this visible to me):

2023-05-02 19:20:02,132 DEBUG   common:54 - [N4] json blob: {'search_head_password': 'REDACTED'}

And the call to the key "password" fails.

[LetMeR00t]

Hi @marcnil815 , Thanks for the quick answer. Is the password REDACTED in your example the one you've set up for your account ? Are you running on Splunk Cloud ?

[marcnil815]

It definitely looks like something built-in. I have no knowledge of this (might be from another app?). Never seen this before. It's not the password of the user I am logging in with.

I cannot rule out that during the setup of Splunk there was some setting somewhere where I set up a password that is reflected like this....

[LetMeR00t]

@marcnil815 ,

Difficult to fix an issue like that honestly. However, sometimes we can face issues with the storage password but as your code is reaching the "clear_password["password"] line, it means that everything was great before.

Could you debug in the logger this variable ? https://github.com/LetMeR00t/TA-thehive-cortex/blob/a6121f7ec3f1b0e9e014ddb5a900f918017c700f/TA-thehive-cortex/bin/common.py#L36 Don't send me the content, but if you can just find if your password setup for the TheHive instance is shown as cleartext in this variable or not at all ? Thank you

[marcnil815]

Already did all that. So the username associated with that search_head_password is the hostname of the entire server.

The sp variable is not clear text, it's an obbject of a type splunklib.client.StoragePasswords Once it moves into the for loop in line 37 (no idea how you post the actual code lines that you do....apologies), everything is fine until that "host name user" is hit.

This user has nothing to do with TheHive, and once i wrap it in a try/except and just move-along-nothing-to-see-here, everything works fine.

[marcnil815]

if it wasn't clear: the app loops through all possible password from all apps. I am logged in as an admin (full admin), so I am thinking: since my admin user is triggering the "retrieve password action" I get more than a normal user would.

That's why I though a namespace/app lockdown would work....

Just spitballing here. Not a splunk dev (or for that matter a dev at all :) )

[LetMeR00t]

@marcnil815 , no problem, any idea is always interesting to be investigated ! Thank you for your sharing at least.

Did you tried to debug the "username" field too ? It should not be a complex object structure but rather a classic dict... https://github.com/LetMeR00t/TA-thehive-cortex/blob/a6121f7ec3f1b0e9e014ddb5a900f918017c700f/TA-thehive-cortex/bin/common.py#L38

If you replace this line: https://github.com/LetMeR00t/TA-thehive-cortex/blob/a6121f7ec3f1b0e9e014ddb5a900f918017c700f/TA-thehive-cortex/bin/thehive.py#L28 You should have a kind of namespace/lockdown as the script is connecting with the TA-thehive-cortex context only (at least, this is my guess) so it's weird to get all the passwords also from other apps ...

FYI, I'm using the "permalink", you can get it by using the three dots next to a given line in Github:

[marcnil815]

Let me show you the relevant debug lines I am using.

 33         if client is not None:
 34             self._passwords = {}
 35             proxy_clear_password = None
 36             sp = self.client.storage_passwords
 37             self.logger.debug("[N1] sp: "+ str(sp))
 38             for credential in sp:
 39                 self.logger.debug("[N2] credential: "+ str(credential))
 40                 username = credential['username'].split("``")[0]
 41                 self.logger.debug("[N3] username: "+ str(username))
 42                 # Process proxy credentials settings
 43                 if 'proxy' in username:
 44                     clear_credentials = credential['clear_password']
 45                     if 'proxy_password' in clear_credentials:
 46                         proxy_creds = json.loads(clear_credentials)
 47                         proxy_clear_password = str(proxy_creds['proxy_password'])
 48                         #TODO: Review this part as it's seems to not be used
 49                 # Otherwise, keep it as a standard user
 50                 else:
 51                     # Only keep the value if it's a clear dictionnary
 52                     if "password" in credential['clear_password']:
 53                         clear_password = json.loads(credential['clear_password'])
 54                         self.logger.debug("[N4] json blob: "+ str(clear_password))
 55                         #try:
 56                         self._passwords[username] = clear_password["password"]
 57                         #except: pass

The app loops through N1-N3 a couple of times (we are talking username for office 365 apps, and other apps, not local users), until it hits N4 a single time, and fails (because it hits the aforementioned user/pw combo)

You can see my try/except as well to just get out of that. Once I implement that hack, it moves past and iterates over the remaining username/password combos.

Other username/passwords where this might fail are called "alert_action_password" (once the try/except is in place it iterates over these as well).

The password for the hive also reaches N4. No problems here.

So basically the app is assuming that the clear_password = json.loads(credential['clear_password'])' always has the key "password".

The keys I can find , at least on my instance, are: search_head_password alert_action_password

And yes: I implemented the 3.0.2 fix with the "app" context.

[LetMeR00t]

Okay Your information are interesting because it means that my test isn’t sufficient , considering that I was only with the TA-thehive-cortex but in fact not really Do you know the name of the TheHive username field that is having your password ? At least the key that match the password condition ?

I’ll try to see if using a list(credential) I can find something else relevant (additionally to the username that I can use to limit the scope for this…

thank you, I’ll keep you posted

[marcnil815]

They key for the TheHive username field is what the current app assumes it is: password. Here are the relevant outputs from the debugging. This is a copy and paste from splunk. Read it bottom up. (notice how it hit's the same user twice....)

2023-05-02 20:39:32,172 DEBUG   common:41 - [N3] username: the_hive_BAR
2023-05-02 20:39:32,172 DEBUG   common:39 - [N2] credential: <splunklib.client.StoragePassword object at FOO>
2023-05-02 20:39:32,172 DEBUG   common:54 - [N4] json blob: {'password': 'REDACTED'}
2023-05-02 20:39:32,172 DEBUG   common:41 - [N3] username: the_hive_BAR
2023-05-02 20:39:32,172 DEBUG   common:39 - [N2] credential: <splunklib.client.StoragePassword object at FOO>

the FOO and BAR illustrate identical strings. So it loops twice though the same object. The second time it does not move into N4

[LetMeR00t]

Hi @marcnil815, I've pushed a new commit this morning : https://github.com/LetMeR00t/TA-thehive-cortex/commit/063ebcc17a8348626eab78f8340ca78e82425c75 It should fix two issues, the one with the storage password namespace respected and also the issue that the local folder is failing the script if it's not exisiting.

The more important line is this one: https://github.com/LetMeR00t/TA-thehive-cortex/blob/063ebcc17a8348626eab78f8340ca78e82425c75/TA-thehive-cortex/bin/common.py#L38

Let me know if it solve your issue please

[marcnil815]

Hi

Sorry for the late reply. The latest change https://github.com/LetMeR00t/TA-thehive-cortex/commit/063ebcc17a8348626eab78f8340ca78e82425c75 seemed to do the trick. No more errors on my side as fas as I can see.

[LetMeR00t]

Hello v3.0.2 was released on the Splunkbase but still waiting for the Cloud Vet. If after installing this version, you keep having issues, please reopen this issue. For now, I'm considering it done. Thank you for your comprehension

[mekhaleraj]

Update on the issue: I'm still facing the same issue.

Steps taken: Upgraded the installed app version with fresh installation. Installed self signed certificate. In Audit log I only get instance created logs. The ta_thehive_cortex_thehive_data.log file on server is empty. Ran Tcpdump, no outgoing request found.

I tried on two different Splunk instances. Same error observed.

@marcnil815 Did it work for you?

@LetMeR00t Are there any specific requirements that needs to be satisfied?

[LetMeR00t]

Hello @mekhaleraj Did you installed the v3.0.2 from scratch ? Which version of Splunk/TheHive are you using please?

If you are facing issues with certificates, it should be shown into the logs. Did you try to get the cases from the dashboard ? (not only creating the instance)

[mekhaleraj]

Hi @LetMeR00t,

I tried both the options. Upgraded current installed version and installed latest version from scratch on new Splunk instance.

Splunk version : 9 Build: 419ad9369127 ( Used for both the instances) TheHive Version : 5.1.5-1

I tried all possible options to connect to TheHive and Cortex like pull and push from dashboard, alerts action etc.

I don't see any certificate related issue in the log but I also don't see outgoing request from Splunk instance.

[LetMeR00t]

Which is weird actually Did you tried to set the DEBUG logging mode and check the lots ? To be sure , do you tested your with a curl request on the server ?

[mekhaleraj]

Yes, DEBUG mode is set for the the instances.

I tested connection with cURL and with custom python script for TheHive API. Both worked very well.

[LetMeR00t]

Hello @mekhaleraj Could it be possible to continue this discussion by mail in order to provide me more details without putting this public ? you can reach me on letmer00t@gmail.com I need to have the full picture of your configuration to help you

[mekhaleraj]

Issue resolved with updated version.