Error running new Upload API Indicator

[PiusO]

Cudeso, any help on this would be appreciated

I ran python3 script.py

script.py:1: DeprecationWarning: As our baseline system is the latest Ubuntu LTS, and Ubuntu LTS 22.04 has Python 3.10 available, we will officially deprecate python versions below 3.10 on January 1st 2024. Please update your codebase. from pymisp import ExpandedPyMISP Unknown error: the response is not in JSON. Something is broken server-side, please send us everything that follows (careful with the auth key): Request headers: {'User-Agent': 'PyMISP 2.4.172 - Python 3.8', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Cookie': 'CAKEPHP=e2r4gou0h5q5ra55g8tfo66nq2edmoc8', 'Content-Length': '381', 'content-type': 'application/json'} Request body: {"returnFormat": "stix2", "page": 2, "limit": 50, "tags": ["workflow:state=\"complete\""], "withAttachments": 0, "metadata": 0, "published": 1, "enforceWarninglist": 0, "to_ids": 1, "includeEventUuid": 0, "includeEventTags": 0, "sgReferenceOnly": 0, "includeContext": 0, "headerless": 0, "includeSightings": 0, "includeDecayScore": 0, "includeCorrelations": 0, "excludeDecayed": 0} Response (if any): {"name":"An Internal Error Has Occurred.","message":"An Internal Error Has Occurred.","url":"\/events\/restSearch"} ----------------CLEAR existing_indicators_hash---------------------------

[cudeso]

This is "expected", in the sense that for now it's not possible to get the number of pages (result of paginated search) that are left in the MISP search. Hence the only way to get all the results is querying MISP until it errors-out. When querying for JSON or others it doesn't throw an error, but the MISP-STIX is passed on to another Python script and when requesting for an in-existing page this Python script returns an error.

See: https://github.com/cudeso/misp2sentinel/tree/upload_indicators_api#upload-indicators-api-misp-to-stix-and-errors-an-internal-error-has-occurred and https://github.com/MISP/misp-stix/issues/44

[PiusO]

Thank you for responding quickly.

Those links seem more like explanations for the problem rather than solutions?

the first link makes reference to settings relevant to Upload indicators API which aren't actually in the config.py script for example, ms_api_version = "2022-07-01" I've added that to my script, but also wondering what else isn't in the script in the repo?

Should I assume that it isn't possible to get MISP data into Sentinel using this script for Upload Indicator API currently? Is there an alternative

[cudeso]

Looking more closely at your first input, the line of code "CLEAR existing_indicators_hash" is no longer in the Upload Indicators branch > https://github.com/cudeso/misp2sentinel/blob/upload_indicators_api/RequestManager.py It does exist in the main branch though. I think you're branch is not set to "upload_indicators_api".

Check with git branch. Switch branches with git checkout upload_indicators_api

The uploading of indicators to Sentinel with the Upload Indicator API definitely works (setups with between 1k and 45k indicators syncing).

[PiusO]

Yes my branch was set to main branch, now switched to upload_indicator_api by first doing git stash I noticed in the upload indicators branch there's no config.py only config.py.default?

So I got rid of everything cloned the upload_indicator_api branch, and i keep getting the config.py version from the main, I then update it's contents with that of config.py.default. So every time I clone, I get the main branch, not sure if others have the same issue.

---

Ran python3 script.py and the error is what I think you were expecting above, but It's not clear what the solution is:

python3 script.py script.py:1: DeprecationWarning: As our baseline system is the latest Ubuntu LTS, and Ubuntu LTS 22.04 has Python 3.10 available, we will officially deprecate python versions below 3.10 on January 1st 2024. Please update your codebase. from pymisp import * Unknown error: the response is not in JSON. Something is broken server-side, please send us everything that follows (careful with the auth key): Request headers: {'User-Agent': 'PyMISP 2.4.172 - Python 3.8', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Cookie': 'CAKEPHP=m2ogf6p7c079bm25vo98a97tivpic4sn', 'Content-Length': '381', 'content-type': 'application/json'} Request body: {"returnFormat": "stix2", "page": 2, "limit": 50, "tags": ["workflow:state=\"complete\""], "withAttachments": 0, "metadata": 0, "published": 1, "enforceWarninglist": 1, "to_ids": 1, "includeEventUuid": 0, "includeEventTags": 1, "sgReferenceOnly": 0, "includeContext": 0, "headerless": 0, "includeSightings": 0, "includeDecayScore": 0, "includeCorrelations": 0, "excludeDecayed": 0} Response (if any): {"name":"An Internal Error Has Occurred.","message":"An Internal Error Has Occurred.","url":"\/events\/restSearch"}

You can ignore 'An Internal Error Has Occurred' message thrown by MISP. The MISP STIX export does not return the number of page results. We query for results until we get an error. Also see: https://github.com/MISP/misp-stix/issues/44

[PiusO]

How do you use misp-stix directly instead of restsearch? Which one of your script need to be configured for that to happen Or do I need to develop something

Are there any other steps required not described under README.md?

To get indicators into Sentinel all I'm doing is python3 script.py

There's something wrong with cloning this repo, files not in upload_indicator_api get pulled, I've had this issue multiple times, can you try replicating the intrusions in your document I see others are running into the same issue.

---

This is my most recent error - my RequestObject.py is the right one, although that wasn't clear at first

python3 script.py script.py:1: DeprecationWarning: As our baseline system is the latest Ubuntu LTS, and Ubuntu LTS 22.04 has Python 3.10 available, we will officially deprecate python versions below 3.10 on January 1st 2024. Please update your codebase. from pymisp import * Traceback (most recent call last): File "script.py", line 6, in from RequestObject import RequestObject, RequestObject_Event, RequestObject_Indicator File "/home/SOM-Admin/sentinel/misp2sentinel/RequestObject.py", line 5, in from stix2.base import STIXJSONEncoder ModuleNotFoundError: No module named 'stix2'

[cudeso]

Hello. Can you check with the latest commit to the branch upload_indicators_api? This should fix the Exception error. There's an update for the documentation and for the script. The integration now requires misp-stix (it's part of the requirements file). You can also manually add it to your Python virtual environment with pip install misp-stix.

[PiusO]

downloaded misp-stix, which removed the error message now I'm left with:

python3 script.py script.py:1: DeprecationWarning: As our baseline system is the latest Ubuntu LTS, and Ubuntu LTS 22.04 has Python 3.10 available, we will officially deprecate python versions below 3.10 on January 1st 2024. Please update your codebase. from pymisp import *

---

Logs :

2023-07-19 14:13:55,121 - misp2sentinel - INFO - Start MISP2Sentinel 2023-07-19 14:13:55,121 - misp2sentinel - INFO - Fetching and parsing data from MISP ... 2023-07-19 14:13:55,121 - misp2sentinel - INFO - Using Microsoft Upload Indicator API 2023-07-19 14:13:55,658 - misp2sentinel - INFO - Received MISP events page 1 with 5 events 2023-07-19 14:13:56,005 - misp2sentinel - INFO - Received MISP events page 2 with 5 events 2023-07-19 14:13:56,186 - misp2sentinel - ERROR - Ignoring non STIX pattern type yara 2023-07-19 14:13:56,186 - misp2sentinel - ERROR - Unable to process indicator 2023-07-19 14:13:56,187 - misp2sentinel - ERROR - Ignoring non STIX pattern type yara 2023-07-19 14:13:56,187 - misp2sentinel - ERROR - Unable to process indicator 2023-07-19 14:13:56,187 - misp2sentinel - ERROR - Ignoring non STIX pattern type yara 2023-07-19 14:13:56,187 - misp2sentinel - ERROR - Unable to process indicator 2023-07-19 14:13:56,570 - misp2sentinel - INFO - Received MISP events page 3 with 5 events 2023-07-19 14:13:56,981 - misp2sentinel - INFO - Received MISP events page 4 with 5 events 2023-07-19 14:13:57,002 - misp2sentinel - ERROR - Ignoring non STIX pattern type yara 2023-07-19 14:13:57,002 - misp2sentinel - ERROR - Unable to process indicator 2023-07-19 14:13:57,002 - misp2sentinel - ERROR - Ignoring non STIX pattern type yara 2023-07-19 14:13:57,002 - misp2sentinel - ERROR - Unable to process indicator 2023-07-19 14:13:57,336 - misp2sentinel - INFO - Received MISP events page 5 with 1 events 2023-07-19 14:13:57,503 - misp2sentinel - INFO - Received 392 indicators in MISP 2023-07-19 14:13:58,668 - misp2sentinel - INFO - Start uploading indicators 2023-07-19 14:14:01,280 - misp2sentinel - ERROR - Error when submitting indicators. {"error":{"code":"UnauthorizedAccess","message":"The Object ID [] does not have required permission to perform this action on the workspace [].","details":[],"target":null,"additionalInfo":null}} 2023-07-19 14:14:01,280 - misp2sentinel - INFO - Finished uploading indicators 2023-07-19 14:14:01,281 - misp2sentinel - INFO - End MISP2Sentinel

[cudeso]

{"code":"UnauthorizedAccess","message":"The Object ID does not have required permission to perform this action on the workspace .","details":[],"target":null,"additionalInfo":null}}: your Azure App does not have the necessary permissions to access the workspace.

You can check the required permissions here https://github.com/cudeso/misp2sentinel/tree/upload_indicators_api#installation and here https://learn.microsoft.com/en-us/azure/sentinel/connect-threat-intelligence-upload-api#register-an-azure-ad-application

[PiusO]

Thanks for pointing that out can you remove the azure details from your comment, thank you, I'll try that now

[lnfernux]

Thanks for pointing that out can you remove the azure details from your comment, thank you, I'll try that now

I will also add that sometimes even if the permissions are given, Azure can "take some time" to apply them. So allow for "Azure-time" to take hold.

[PiusO]

I provided the new config script the same details as what i used for Graph API.

With the new Upload API it also needs Sentinel Contributor role given to it which i didn't provide and now have and finally logs have arrived.

Thank you very much for your hard work and taking time to respond!

---

It doesn't change screen and start to tally indicators sent to the API like it used to, so at first it doesn't seem as though its sent, had to check the logs, but confirmation is there

---

Errors still show in the logs not sure if this will cause an issue later or if they are expected:

2023-07-19 16:32:10,679 - misp2sentinel - INFO - Start MISP2Sentinel 2023-07-19 16:32:10,679 - misp2sentinel - INFO - Fetching and parsing data from MISP ... 2023-07-19 16:32:10,679 - misp2sentinel - INFO - Using Microsoft Upload Indicator API 2023-07-19 16:32:11,219 - misp2sentinel - INFO - Received MISP events page 1 with 5 events 2023-07-19 16:32:11,564 - misp2sentinel - INFO - Received MISP events page 2 with 5 events 2023-07-19 16:32:11,744 - misp2sentinel - ERROR - Ignoring non STIX pattern type yara 2023-07-19 16:32:11,744 - misp2sentinel - ERROR - Unable to process indicator 2023-07-19 16:32:11,745 - misp2sentinel - ERROR - Ignoring non STIX pattern type yara 2023-07-19 16:32:11,745 - misp2sentinel - ERROR - Unable to process indicator 2023-07-19 16:32:11,745 - misp2sentinel - ERROR - Ignoring non STIX pattern type yara 2023-07-19 16:32:11,745 - misp2sentinel - ERROR - Unable to process indicator 2023-07-19 16:32:12,034 - misp2sentinel - INFO - Received MISP events page 3 with 5 events 2023-07-19 16:32:12,527 - misp2sentinel - INFO - Received MISP events page 4 with 5 events 2023-07-19 16:32:12,548 - misp2sentinel - ERROR - Ignoring non STIX pattern type yara 2023-07-19 16:32:12,548 - misp2sentinel - ERROR - Unable to process indicator 2023-07-19 16:32:12,548 - misp2sentinel - ERROR - Ignoring non STIX pattern type yara 2023-07-19 16:32:12,548 - misp2sentinel - ERROR - Unable to process indicator 2023-07-19 16:32:12,793 - misp2sentinel - INFO - Received MISP events page 5 with 1 events 2023-07-19 16:32:12,959 - misp2sentinel - INFO - Received 392 indicators in MISP 2023-07-19 16:32:14,087 - misp2sentinel - INFO - Start uploading indicators 2023-07-19 16:32:18,820 - misp2sentinel - INFO - Indicators sent - request number: 0 / indicators: 100 / remaining: 292 2023-07-19 16:32:21,761 - misp2sentinel - INFO - Indicators sent - request number: 1 / indicators: 100 / remaining: 192 2023-07-19 16:32:22,283 - misp2sentinel - INFO - Indicators sent - request number: 2 / indicators: 100 / remaining: 92 2023-07-19 16:32:26,936 - misp2sentinel - INFO - Indicators sent - request number: 3 / indicators: 92 / remaining: 0 2023-07-19 16:32:26,937 - misp2sentinel - INFO - Finished uploading indicators 2023-07-19 16:32:26,937 - misp2sentinel - INFO - End MISP2Sentinel

[cudeso]

Hello. Thank you, and good to hear everything worked out.

The errors are due to the fact that only those indicators that can be translated to STIX are synchronised. YARA (and Sigma) indicators cannot be imported in Sentinel, mostly because Sentinel does not accept them -yet-. > https://github.com/cudeso/misp2sentinel/tree/upload_indicators_api#ignored-types