TAXII Ingestion: Undefined taxii objects - When running as non SYSTEM

[MaxwellDPS]

Description

In the event a TAXII server is requires a trailing / the taxii ingestion fails with Undefined taxii objects

In this instance for the TAXII server that is failing

• This works potato.taxi/taxii2/collections/abcd/objects/
• This does not potato.taxi/taxii2/collections/abcd/objects and will return {}

Environment

1. OS (where OpenCTI server runs): CentOS Stream
2. OpenCTI version: 6.2.6
3. OpenCTI client: frontend
4. Other environment details: Clusered

Reproducible Steps

Steps to create the smallest reproducible scenario:

1. Setup taxii server that requires a trailing / or returns {} w/o a trailing slash ex potato.taxi/taxii2/collections/abcd/objects
2. Add as a taxii source
3. Nothing imports

Expected Output

N/A

Actual Output

N/A

Additional information

{
    "category": "APP",
    "context": "Taxii 2.1 transform",
    "errors": [
        {
            "attributes": {
                "genre": "TECHNICAL",
                "http_status": 500
            },
            "message": "Undefined taxii objects",
            "name": "UNKNOWN_ERROR",
            "stack": "UNKNOWN_ERROR: Undefined taxii objects\n    at error (/opt/opencti/build/src/config/errors.js:8:10)\n    at UnknownError (/opt/opencti/build/src/config/errors.js:82:47)\n    at taxiiV21DataHandler (/opt/opencti/build/src/manager/ingestionManager.ts:256:19)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async Promise.all (index 0)\n    at async Promise.all (index 1)\n    at ingestionHandler (/opt/opencti/build/src/manager/ingestionManager.ts:369:5)\n    at /opt/opencti/build/src/manager/ingestionManager.ts:388:9\n    at eSt.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13)\n    at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)"
        }
    ],
    "level": "error",
    "message": "Undefined taxii objects",
    "name": "THAT-FEED",
    "source": "backend",
    "timestamp": "2024-07-18T23:50:49.175Z",
    "version": "6.2.6"
}

[MaxwellDPS]

Linking #7403 as this was previously not an issue

[MaxwellDPS]

Feed seems to have started running, still seeing intermittent errors and hangin on a single data however

[aHenryJard]

Hello, you should not need to add "/collections/abcd/objects" it's automatically added in the code. So potato.taxi/taxii2 and potato.taxi/taxii2/ should both work.

[MaxwellDPS]

Yeah I am not adding that it is set to potato.taxi/taxii2/ - Was previously working in the exact same config, just upgraded to fix this https://github.com/OpenCTI-Platform/opencti/issues/7403

[aHenryJard]

Yeah I am not adding that it is set to potato.taxi/taxii2/ - Was previously working in the exact same config, just upgraded to fix this #7403

This fix changed nothing on trailing /;, it's just allowing "next" cursor to be undefined from Taxii response in which case the date is used in 'X-TAXII-Date-Added-Last' for requests. Does your taxii server answer with a empty "next" field but still requires a "next" in the follow up query ?

Where do you see that there is a missing '/' if you are using "potato.taxi/taxii2/" in configuration because it's still there in code so I don't understand what is happening.

https://github.com/OpenCTI-Platform/opencti/blob/892e878d4433fab7f3748b88bf30bed2d565e786/opencti-platform/opencti-graphql/src/manager/ingestionManager.ts#L229

[MaxwellDPS]

So looking into this more, there seems to be little to no logging on this, / was an assumption due to this. But whenever the user is set it wont consume the taxii feed (even if set to an admin user). Only time the feed runs on 6.2.6 is if you dont fill in a user and let SYSTEM run the import.

[aHenryJard]

So looking into this more, there seems to be little to no logging on this, / was an assumption due to this. But whenever the user is set it wont consume the taxii feed (even if set to an admin user). Only time the feed runs on 6.2.6 is if you dont fill in a user and let SYSTEM run the import.

Ok thanks I should be able to reproduce that

[MaxwellDPS]

Awesome, thanks! If you reach out over slack I can provide some more detail

[aHenryJard]

I'm not reproducing any issue with a user different from SYSTEM, I am getting data from taxii feed with bith a user that has bypass, and another user that has this role:

I am adding some logs at info level to better understand. Does you taxii feed requires both "next" and "added_after" parameter or is using only one of them, or it's not a paginated feed ?

[aHenryJard]

Do you have maybe an organisation setup at platform level and is the taxii feed user in the platform organization too ? The user is not used to query data to the taxii server, so you should have some error at ingestion time in workers and/opencti logs.

[nino-filigran]

@aHenryJard did you get more details?

[MaxwellDPS]

Do you have maybe an organisation setup at platform level and is the taxii feed user in the platform organization too ? The user is not used to query data to the taxii server, so you should have some error at ingestion time in workers and/opencti logs.

Hey @aHenryJard Thanks! But nope no org platform level 🙁