Issues uploading Threat Indicators to Sentinel/Defender ATP

blockanz commented 2 months ago

Description

Currently trying to get threat indicators to load into Sentinel/Defender ATP. My .yml config is correct and I can see the connection from OpenCTI to my tenant and enterprise app.

When the connector tries to upload data from a Live Stream getting the following errors:

{"log":"{\"timestamp\": \"2024-04-21T20:45:47.662010Z\", \"level\": \"ERROR\", \"name\": \"sentinel\", \"message\": \"[ERROR] Message data {{\\"data\\":{\\"id\\":\\"ipv4-addr--672b6f92-df0e-5985-85ff-020f608157b2\\",\\"spec_version\\":\\"2.1\\",\\"type\\":\\"ipv4-addr\\",\\"extensions\\":{\\"extension-definition--ea279b3e-5c71-4632-ac08-831c66a786ba\\":{\\"extension_type\\":\\"property-extension\\",\\"id\\":\\"11b682c0-b0d2-4146-ba8f-fec8ea07f071\\",\\"type\\":\\"IPv4-Addr\\",\\"created_at\\":\\"2024-04-17T00:27:38.018Z\\",\\"updated_at\\":\\"2024-04-17T00:27:38.226Z\\",\\"is_inferred\\":false,\\"creator_ids\\":[\\"88ec0c6a-13ce-5e39-b486-354fe4a7084f\\"],\\"labels_ids\\":[\\"055fcb09-3c1c-4237-99ac-45736dc3147b\\"],\\"created_by_ref_id\\":\\"a6585c81-45ed-44b8-b402-5552e6e71d12\\"},\\"extension-definition--f93e2c80-4231-4f9a-af8b-95c9bd566a82\\":{\\"extension_type\\":\\"property-extension\\",\\"labels\\":[\\"osint:source-type=\\\\"block-or-filter-list\\\\"\\"],\\"score\\":50,\\"created_by_ref\\":\\"identity--acc88828-68cf-514f-a9b4-1be7f4c514ae\\"}},\\"object_marking_refs\\":[\\"marking-definition--613f2e26-407d-48c7-9eca-b8e91df99dc9\\"],\\"value\\":\\"202.189.172.168\\"},\\"message\\":\\"creates a IPv4-Addr 202.189.172.168\\",\\"origin\\":{\\"referer\\":\\"init-create\\"},\\"version\\":\\"4\\"}}\", \"exc_info\": \"Traceback (most recent call last):\n File \\"/opt/opencti-connector-sentinel/sentinel.py\\", line 458, in _process_message\n self._create_observable(data)\n File \\"/opt/opencti-connector-sentinel/sentinel.py\\", line 180, in _create_observable\n days = int(self.expire_time)\n ^^^^^^^^^^^^^^^^^^^^^\nTypeError: int() argument must be a string, a bytes-like object or a real number, not 'NoneType'\"}\n","stream":"stderr","time":"2024-04-21T20:45:47.66224639Z"}

This seems to be happening to all data coming from the stream.

Environment

OS - Ubuntu 22.04
OpenCTI version: 6.0.10
OpenCTI client: frontend
Other environment details:

Reproducible Steps

Steps to create the smallest reproducible scenario:

Create a new live stream
stop all containers
update sentinel connector with stream ID
Start all docker containers including sentinel connector

Expected Output

Threat Indicators are uploaded correctly to sentinel/defender

Actual Output

Error received as above, and no Threat Indicators are uploaded or visible in MS environment

Additional information

Screenshots (optional)

nino-filigran commented 2 months ago

@The-Stuke I saw that you created the connector. Do you know what's happening? Otherwise @Megafredo or @helene-nguyen could you have a look when you have time? This is a connector being under the community supervision FYI

Megafredo commented 2 months ago

Hi @blockanz, this error occurs when the environment variable "EXPIRE_TIME" is either missing or empty, can you check this variable in your .yml ? By default in the README: EXPIRE_TIME=30

blockanz commented 2 months ago

@Megafredo

I made the change and added EXPIRE_TIME=30.

Now I get the following errors:

{"log":"{\"timestamp\": \"2024-04-25T01:21:59.424348Z\", \"level\": \"ERROR\", \"name\": \"sentinel\", \"message\": \"[ERROR] Failed processing data {can only concatenate str (not \\"NoneType\\") to str}\", \"exc_info\": \"Traceback (most recent call last):\n File \\"/opt/opencti-connector-sentinel/sentinel.py\\", line 458, in _process_message\n self._create_observable(data)\n File \\"/opt/opencti-connector-sentinel/sentinel.py\\", line 266, in _create_observable\n self.resource_url + self.request_url,\n ~~~~^~~~\nTypeError: can only concatenate str (not \\"NoneType\\") to str\"}\n","stream":"stderr","time":"2024-04-25T01:21:59.42478946Z"} {"log":"{\"timestamp\": \"2024-04-25T01:21:59.425073Z\", \"level\": \"ERROR\", \"name\": \"sentinel\", \"message\": \"[ERROR] Message data {{\\"version\\":\\"4\\",\\"type\\":\\"create\\",\\"scope\\":\\"external\\",\\"message\\":\\"creates a IPv4-Addr 123.14.18.239\\",\\"origin\\":{\\"socket\\":\\"query\\",\\"ip\\":\\"::ffff:192.168.48.1\\",\\"user_id\\":\\"88ec0c6a-13ce-5e39-b486-354fe4a7084f\\",\\"group_ids\\":[\\"576aa993-0257-46cf-844d-8d5a44128257\\"],\\"organization_ids\\":[],\\"user_metadata\\":{},\\"applicant_id\\":\\"88ec0c6a-13ce-5e39-b486-354fe4a7084f\\",\\"call_retry_number\\":\\"1\\"},\\"data\\":{\\"id\\":\\"ipv4-addr--21075343-2f26-5461-9993-263f210858ff\\",\\"spec_version\\":\\"2.1\\",\\"type\\":\\"ipv4-addr\\",\\"extensions\\":{\\"extension-definition--ea279b3e-5c71-4632-ac08-831c66a786ba\\":{\\"extension_type\\":\\"property-extension\\",\\"id\\":\\"94ce7581-6907-41e3-a065-0c9a27bfba74\\",\\"type\\":\\"IPv4-Addr\\",\\"created_at\\":\\"2024-04-25T01:21:58.817Z\\",\\"updated_at\\":\\"2024-04-25T01:21:58.817Z\\",\\"is_inferred\\":false,\\"creator_ids\\":[\\"88ec0c6a-13ce-5e39-b486-354fe4a7084f\\"],\\"labels_ids\\":[\\"c13f46fe-addf-4d20-9907-dbc599753220\\",\\"8de95e15-5aeb-4d81-af06-ff7f102fc32b\\"],\\"created_by_ref_id\\":\\"9faf421d-5355-41d9-8731-7f63dc0509ca\\"},\\"extension-definition--f93e2c80-4231-4f9a-af8b-95c9bd566a82\\":{\\"extension_type\\":\\"property-extension\\",\\"labels\\":[\\"elf\\",\\"mozi\\"],\\"description\\":\\"Malware payload delivery host\\",\\"score\\":60,\\"created_by_ref\\":\\"identity--0303206b-ec74-5e9e-81df-e6532e9c1e91\\"}},\\"object_marking_refs\\":[\\"marking-definition--613f2e26-407d-48c7-9eca-b8e91df99dc9\\"],\\"value\\":\\"123.14.18.239\\"}}}\", \"exc_info\": \"Traceback (most recent call last):\n File \\"/opt/opencti-connector-sentinel/sentinel.py\\", line 458, in _process_message\n self._create_observable(data)\n File \\"/opt/opencti-connector-sentinel/sentinel.py\\", line 266, in _create_observable\n self.resource_url + self.request_url,\n ~~~~^~~~\nTypeError: can only concatenate str (not \\"NoneType\\") to str\"}\n","stream":"stderr","time":"2024-04-25T01:21:59.42533194Z"} {"log":"{\"timestamp\": \"2024-04-25T01:21:59.879506Z\", \"level\": \"ERROR\", \"name\": \"sentinel\", \"message\": \"[ERROR] Failed processing data {can only concatenate str (not \\"NoneType\\") to str}\", \"exc_info\": \"Traceback (most recent call last):\n File \\"/opt/opencti-connector-sentinel/sentinel.py\\", line 458, in _process_message\n self._create_observable(data)\n File \\"/opt/opencti-connector-sentinel/sentinel.py\\", line 266, in _create_observable\n self.resource_url + self.request_url,\n ~~~~^~~~\nTypeError: can only concatenate str (not \\"NoneType\\") to str\"}\n","stream":"stderr","time":"2024-04-25T01:21:59.880151227Z"} {"log":"{\"timestamp\": \"2024-04-25T01:21:59.881081Z\", \"level\": \"ERROR\", \"name\": \"sentinel\", \"message\": \"[ERROR] Message data {{\\"version\\":\\"4\\",\\"type\\":\\"create\\",\\"scope\\":\\"external\\",\\"message\\":\\"creates a IPv4-Addr 123.14.251.202\\",\\"origin\\":{\\"socket\\":\\"query\\",\\"ip\\":\\"::ffff:192.168.48.1\\",\\"user_id\\":\\"88ec0c6a-13ce-5e39-b486-354fe4a7084f\\",\\"group_ids\\":[\\"576aa993-0257-46cf-844d-8d5a44128257\\"],\\"organization_ids\\":[],\\"user_metadata\\":{},\\"applicant_id\\":\\"88ec0c6a-13ce-5e39-b486-354fe4a7084f\\",\\"call_retry_number\\":\\"1\\"},\\"data\\":{\\"id\\":\\"ipv4-addr--305c4cae-d829-5ee5-a850-c8fe145146a1\\",\\"spec_version\\":\\"2.1\\",\\"type\\":\\"ipv4-addr\\",\\"extensions\\":{\\"extension-definition--ea279b3e-5c71-4632-ac08-831c66a786ba\\":{\\"extension_type\\":\\"property-extension\\",\\"id\\":\\"cf824fcc-d364-499c-8311-f5e9e3e84126\\",\\"type\\":\\"IPv4-Addr\\",\\"created_at\\":\\"2024-04-25T01:21:59.351Z\\",\\"updated_at\\":\\"2024-04-25T01:21:59.351Z\\",\\"is_inferred\\":false,\\"creator_ids\\":[\\"88ec0c6a-13ce-5e39-b486-354fe4a7084f\\"],\\"labels_ids\\":[\\"c13f46fe-addf-4d20-9907-dbc599753220\\",\\"8de95e15-5aeb-4d81-af06-ff7f102fc32b\\"],\\"created_by_ref_id\\":\\"9faf421d-5355-41d9-8731-7f63dc0509ca\\"},\\"extension-definition--f93e2c80-4231-4f9a-af8b-95c9bd566a82\\":{\\"extension_type\\":\\"property-extension\\",\\"labels\\":[\\"elf\\",\\"mozi\\"],\\"description\\":\\"Malware payload delivery host\\",\\"score\\":60,\\"created_by_ref\\":\\"identity--0303206b-ec74-5e9e-81df-e6532e9c1e91\\"}},\\"object_marking_refs\\":[\\"marking-definition--613f2e26-407d-48c7-9eca-b8e91df99dc9\\"],\\"value\\":\\"123.14.251.202\\"}}}\", \"exc_info\": \"Traceback (most recent call last):\n File \\"/opt/opencti-connector-sentinel/sentinel.py\\", line 458, in _process_message\n self._create_observable(data)\n File \\"/opt/opencti-connector-sentinel/sentinel.py\\", line 266, in _create_observable\n self.resource_url + self.request_url,\n ~~~~^~~~\nTypeError: can only concatenate str (not \\"NoneType\\") to str\"}\n","stream":"stderr","time":"2024-04-25T01:21:59.881292975Z"} {"log":"{\"timestamp\": \"2024-04-25T01:22:00.162560Z\", \"level\": \"ERROR\", \"name\": \"sentinel\", \"message\": \"[ERROR] Failed processing data {can only concatenate str (not \\"NoneType\\") to str}\", \"exc_info\": \"Traceback (most recent call last):\n File \\"/opt/opencti-connector-sentinel/sentinel.py\\", line 458, in _process_message\n self._create_observable(data)\n File \\"/opt/opencti-connector-sentinel/sentinel.py\\", line 266, in _create_observable\n self.resource_url + self.request_url,\n ~~~~^~~~\nTypeError: can only concatenate str (not \\"NoneType\\") to str\"}\n","stream":"stderr","time":"2024-04-25T01:22:00.162989455Z"} {"log":"{\"timestamp\": \"2024-04-25T01:22:00.163477Z\", \"level\": \"ERROR\", \"name\": \"sentinel\", \"message\": \"[ERROR] Message data {{\\"version\\":\\"4\\",\\"type\\":\\"create\\",\\"scope\\":\\"external\\",\\"message\\":\\"creates a IPv4-Addr 123.14.252.72\\",\\"origin\\":{\\"socket\\":\\"query\\",\\"ip\\":\\"::ffff:192.168.48.1\\",\\"user_id\\":\\"88ec0c6a-13ce-5e39-b486-354fe4a7084f\\",\\"group_ids\\":[\\"576aa993-0257-46cf-844d-8d5a44128257\\"],\\"organization_ids\\":[],\\"user_metadata\\":{},\\"applicant_id\\":\\"88ec0c6a-13ce-5e39-b486-354fe4a7084f\\",\\"call_retry_number\\":\\"1\\"},\\"data\\":{\\"id\\":\\"ipv4-addr--d11fbddd-56a6-5f3a-ac93-0456a333fcd6\\",\\"spec_version\\":\\"2.1\\",\\"type\\":\\"ipv4-addr\\",\\"extensions\\":{\\"extension-definition--ea279b3e-5c71-4632-ac08-831c66a786ba\\":{\\"extension_type\\":\\"property-extension\\",\\"id\\":\\"04dd40dc-6b01-46e4-9c37-bc511669cd10\\",\\"type\\":\\"IPv4-Addr\\",\\"created_at\\":\\"2024-04-25T01:21:59.435Z\\",\\"updated_at\\":\\"2024-04-25T01:21:59.435Z\\",\\"is_inferred\\":false,\\"creator_ids\\":[\\"88ec0c6a-13ce-5e39-b486-354fe4a7084f\\"],\\"labels_ids\\":[\\"4491d7c7-5744-408e-aa4b-837dd2dd172d\\",\\"c13f46fe-addf-4d20-9907-dbc599753220\\",\\"42c9846a-d05b-4bf4-9956-236dfdae90e6\\",\\"8de95e15-5aeb-4d81-af06-ff7f102fc32b\\"],\\"created_by_ref_id\\":\\"9faf421d-5355-41d9-8731-7f63dc0509ca\\"},\\"extension-definition--f93e2c80-4231-4f9a-af8b-95c9bd566a82\\":{\\"extension_type\\":\\"property-extension\\",\\"labels\\":[\\"32-bit\\",\\"elf\\",\\"mips\\",\\"mozi\\"],\\"description\\":\\"Malware payload delivery host\\",\\"score\\":60,\\"created_by_ref\\":\\"identity--0303206b-ec74-5e9e-81df-e6532e9c1e91\\"}},\\"object_marking_refs\\":[\\"marking-definition--613f2e26-407d-48c7-9eca-b8e91df99dc9\\"],\\"value\\":\\"123.14.252.72\\"}}}\", \"exc_info\": \"Traceback (most recent call last):\n File \\"/opt/opencti-connector-sentinel/sentinel.py\\", line 458, in _process_message\n self._create_observable(data)\n File \\"/opt/opencti-connector-sentinel/sentinel.py\\", line 266, in _create_observable\n self.resource_url + self.request_url,\n ~~~~^~~~\nTypeError: can only concatenate str (not \\"NoneType\\") to str\"}\n","stream":"stderr","time":"2024-04-25T01:22:00.163711235Z"}

Any ideas?

Megafredo commented 2 months ago

Hi @blockanz, it seems that there is another environment variable missing in your yml, given the error I would say :

      - RESOURCE_URL=https://graph.microsoft.com
      - REQUEST_URL=/beta/security/tiIndicators

Here is the link to docker-compose with all the environment variables for sentinel, can you compare it with the one you have ? I hope this solves your problem.

blockanz commented 2 months ago

Grrr. I somehow mistyped things and had the INCIDENT_URL with the REQUEST_URL value. Thanks you. I'll test this with the proper values and advise.

blockanz commented 2 months ago

I'm no longer getting the errors I had previously, however I am not seeing any data loaded into my tiIndicators in the Defender portal. Do these take a while to get logged in?

blockanz commented 2 months ago

And do you know if there are any logs in Defender for Endpoint/Entra that can show me if the upload is successful or not, and if not the issue? I'm seeing no errors in my Sentinel connector at all now, and no indicators uploaded. There is definitely connection as I can see all the successful connection attempts in my sign-in logs.

Megafredo commented 2 months ago

Hi @blockanz, then I know what you put in the variable ?

CONNECTOR_LIVE_STREAM_ID=ChangeMe

The two valid cases are : // General stream

CONNECTOR_LIVE_STREAM_ID=live

// Stream with filters applied

CONNECTOR_LIVE_STREAM_ID=(UUID generated by OpenCTI)

If you already have one of these cases, you would need more information on the log side at the connector level, you can replace "error" in "info" for this variable:

CONNECTOR_LOG_LEVEL=info

blockanz commented 2 months ago

I have changed log level and can now see the following:

INFO [CREATE] Processing data {3d4a8c43-87e2-48fc-9134-b975a5e1cecd} | timestamp=2024-04-29T20:55:33.443926Z name=sentinel INFO [CREATE] ID {3d4a8c43-87e2-48fc-9134-b975a5e1cecd Failed and got }<Response [400]> status code. | timestamp=2024-04-29T20:55:34.083821Z name=sentinel

Any ideas why I am getting a Failed with response [400]? I can see the connection to the API successful when I review the sign-in logs in Entra, application should have the appropriate rights to read/write to DefenderATP graph.

Response 400 suggests bad or malformed request so not sure where that is occurring.

Any help would be greatly appreciated @Megafredo

blockanz commented 2 months ago

I made some changes to the application permissions which seems to have resolved some things. Now I am seeing below in the logs:

INFO Starting to listen stream events | timestamp=2024-04-29T22:30:55.949460Z name=sentinel attributes={"live_stream_url":"http://192.168.16.80:8080/stream/1ac36339-a9fd-4a44-b4ad-0bab4a165f08?recover=2024-04-26T01:49:55Z","listen_delete":"false","no_dependencies":"true","with_inferences":"false"} INFO Initiate work | timestamp=2024-04-29T22:38:08.105670Z name=api attributes={"connector_id":"aaa73d9b-c481-e5e9-d6a7-7acd72df2abb"} INFO Update action expectations | timestamp=2024-04-29T22:38:08.210204Z name=api attributes={"work_id":"work_aaa73d9b-c481-e5e9-d6a7-7acd72df2abb_2024-04-29T22:38:08.132Z","expectations":13} INFO sentinel sending bundle to queue | timestamp=2024-04-29T22:38:08.334139Z name=sentinel INFO Reporting work update_processed | timestamp=2024-04-29T22:38:08.416719Z name=api attributes={"work_id":"work_aaa73d9b-c481-e5e9-d6a7-7acd72df2abb_2024-04-29T22:38:08.132Z"} INFO Initiate work | timestamp=2024-04-29T22:39:09.729092Z name=api attributes={"connector_id":"aaa73d9b-c481-e5e9-d6a7-7acd72df2abb"} INFO Update action expectations | timestamp=2024-04-29T22:39:09.857278Z name=api attributes={"work_id":"work_aaa73d9b-c481-e5e9-d6a7-7acd72df2abb_2024-04-29T22:39:09.753Z","expectations":13} INFO sentinel sending bundle to queue | timestamp=2024-04-29T22:39:09.967434Z name=sentinel INFO Reporting work update_processed | timestamp=2024-04-29T22:39:10.024125Z name=api attributes={"work_id":"work_aaa73d9b-c481-e5e9-d6a7-7acd72df2abb_2024-04-29T22:39:09.753Z"}

Unfortunately I am still not seeing indicators reaching Defender, so not sure they are working. Documentation does state that these can take several hours, so I will wait and see if things change.

Megafredo commented 2 months ago

@blockanz, "when I review the sign-in logs in Entra, application should have the appropriate rights to read/write to DefenderATP graph."

Have you set up the necessary permissions on Sentinel ?

You must have in portal Azure: Home > Application Registration > OpenCTI (your name) > API Permissions And prioritize the permissions for "ThreatIndicators.ReadWrite.OwnedBy".