OpenCTI TAXII Feed - 413 Content Too Large

[agrawald]

Description

We are trying to connect OpenTAXII collection, which is very large, to OpenCTI, using TAXII feeds. While processing the collection, we are getting following error

{"category":"APP","context":"Taxii ingestion execution","errors":[{"attributes":{"genre":"TECHNICAL","http_status":500},"message":"Request failed with status code 413","name":"UNKNOWN_ERROR","stack":"GraphQLError: Request failed with status code 413 at error (/opt/opencti/build/src/config/errors.js:7:10) at UnknownError (/opt/opencti/build/src/config/errors.js:81:47) at Object._logWithError (/opt/opencti/build/src/config/conf.js:238:17) at Object.error (/opt/opencti/build/src/config/conf.js:247:48) at /opt/opencti/build/src/manager/ingestionManager.ts:402:18 at processTicksAndRejections (node:internal/process/task_queues:95:5) at async Promise.all (index 0) at async Promise.all (index 1) at ingestionHandler (/opt/opencti/build/src/manager/ingestionManager.ts:508:5) at /opt/opencti/build/src/manager/ingestionManager.ts:529:9 at iit.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13) at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)"},{"message":"Request failed with status code 413","name":"AxiosError","stack":"AxiosError: Request failed with status code 413 at settle (/opt/opencti/build/node_modules/axios/lib/core/settle.js:19:12) at IncomingMessage.handleStreamEnd (/opt/opencti/build/node_modules/axios/lib/adapters/http.js:599:11) at IncomingMessage.emit (node:events:531:35) at endReadableNT (node:internal/streams/readable:1696:12) at processTicksAndRejections (node:internal/process/task_queues:82:21) at yKt.request (/opt/opencti/build/node_modules/axios/lib/core/Axios.js:45:41) at processTicksAndRejections (node:internal/process/task_queues:95:5) at taxiiHttpGet (/opt/opencti/build/src/manager/ingestionManager.ts:314:29) at taxiiV21DataHandler (/opt/opencti/build/src/manager/ingestionManager.ts:375:24) at async Promise.all (index 0) at async Promise.all (index 1) at ingestionHandler (/opt/opencti/build/src/manager/ingestionManager.ts:508:5) at /opt/opencti/build/src/manager/ingestionManager.ts:529:9 at iit.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13) at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)"}],"level":"error","message":"Request failed with status code 413","name":"BLOCK_FEED_CONTEXT","source":"backend","timestamp":"2024-10-18T03:41:31.368Z","version":"6.3.6"}

Environment

1. OS (where OpenCTI server runs): AWS RHEL EC2
2. OpenCTI version: OpenCTI 6.3.6
3. OpenCTI client: NA
4. Other environment details: NA

Reproducible Steps

Steps to create the smallest reproducible scenario:

1. Configure a TAXII Feeds for OpenTAXII which has a large collection

Expected Output

Large Collection from OpenTAXII should make use of paginations to fetch thereby, importing all the objects.

Actual Output

Errors out as OpenCTI is trying to fetch everything in one REST API call.

Additional information

NA

Screenshots (optional)

NA

[nino-filigran]

@agrawald do you maybe a feed that we can use to be able to reproduce?

[agrawald]

Apologies @nino-filigran I am not allowed to share the feed as part of my corporate agreement. However, I can tell you that the feed has more than 45000 STIX records. I will still check with my team.

[agrawald]

Apologies @nino-filigran I am not allowed to share the feed as part of my corporate agreement. However, I can tell you that the feed has more than 45000 STIX records. I will still check with my team.

Apologies, will not be able to help you with the test data. However, if you do decide to implement pagination for TAXII feeds, I can download and test it out for you on a branch if you would prefer.

[nino-filigran]

Thanks @agrawald, I'm still trying to figure out a Taxii with this amount of data for now!