Open agrawald opened 1 week ago
@agrawald do you maybe a feed that we can use to be able to reproduce?
Apologies @nino-filigran I am not allowed to share the feed as part of my corporate agreement. However, I can tell you that the feed has more than 45000 STIX records. I will still check with my team.
Apologies @nino-filigran I am not allowed to share the feed as part of my corporate agreement. However, I can tell you that the feed has more than 45000 STIX records. I will still check with my team.
Apologies, will not be able to help you with the test data. However, if you do decide to implement pagination for TAXII feeds, I can download and test it out for you on a branch if you would prefer.
Thanks @agrawald, I'm still trying to figure out a Taxii with this amount of data for now!
Description
We are trying to connect OpenTAXII collection, which is very large, to OpenCTI, using TAXII feeds. While processing the collection, we are getting following error
{"category":"APP","context":"Taxii ingestion execution","errors":[{"attributes":{"genre":"TECHNICAL","http_status":500},"message":"Request failed with status code 413","name":"UNKNOWN_ERROR","stack":"GraphQLError: Request failed with status code 413 at error (/opt/opencti/build/src/config/errors.js:7:10) at UnknownError (/opt/opencti/build/src/config/errors.js:81:47) at Object._logWithError (/opt/opencti/build/src/config/conf.js:238:17) at Object.error (/opt/opencti/build/src/config/conf.js:247:48) at /opt/opencti/build/src/manager/ingestionManager.ts:402:18 at processTicksAndRejections (node:internal/process/task_queues:95:5) at async Promise.all (index 0) at async Promise.all (index 1) at ingestionHandler (/opt/opencti/build/src/manager/ingestionManager.ts:508:5) at /opt/opencti/build/src/manager/ingestionManager.ts:529:9 at iit.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13) at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)"},{"message":"Request failed with status code 413","name":"AxiosError","stack":"AxiosError: Request failed with status code 413 at settle (/opt/opencti/build/node_modules/axios/lib/core/settle.js:19:12) at IncomingMessage.handleStreamEnd (/opt/opencti/build/node_modules/axios/lib/adapters/http.js:599:11) at IncomingMessage.emit (node:events:531:35) at endReadableNT (node:internal/streams/readable:1696:12) at processTicksAndRejections (node:internal/process/task_queues:82:21) at yKt.request (/opt/opencti/build/node_modules/axios/lib/core/Axios.js:45:41) at processTicksAndRejections (node:internal/process/task_queues:95:5) at taxiiHttpGet (/opt/opencti/build/src/manager/ingestionManager.ts:314:29) at taxiiV21DataHandler (/opt/opencti/build/src/manager/ingestionManager.ts:375:24) at async Promise.all (index 0) at async Promise.all (index 1) at ingestionHandler (/opt/opencti/build/src/manager/ingestionManager.ts:508:5) at /opt/opencti/build/src/manager/ingestionManager.ts:529:9 at iit.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13) at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)"}],"level":"error","message":"Request failed with status code 413","name":"BLOCK_FEED_CONTEXT","source":"backend","timestamp":"2024-10-18T03:41:31.368Z","version":"6.3.6"}
Environment
Reproducible Steps
Steps to create the smallest reproducible scenario:
Expected Output
Large Collection from OpenTAXII should make use of paginations to fetch thereby, importing all the objects.
Actual Output
Errors out as OpenCTI is trying to fetch everything in one REST API call.
Additional information
NA
Screenshots (optional)
NA