search

OpenCTI-Platform / opencti

Open Cyber Threat Intelligence Platform
https://opencti.io
Other
6.27k stars 929 forks source link

[rss] rss feed issues #7659

Closed dominictory closed 2 months ago

dominictory commented 3 months ago

Description

I have a number of RSS feeds configured, however reports are either not creating or are creating extremely slowly, and I have validated that the configured URLs are correct. One RSS feed I deleted, yet reports from it are still being created. Our ImportExternalReference connector is filtered by the [F] RSS user that is used by each configuration to create reports. Example configuration, where each one was initialised with Import from date as 01/01/2024:

image

Observed errors in logs below:

ERR connect ETIMEDOUT 192.0.66.233:443 | category=APP context=Rss execution errors=[{"attributes":{"genre":"TECHNICAL","http_status":500},"message":"connect ETIMEDOUT 192.0.66.233:443","name":"UNKNOWN_ERROR","stack":"UNKNOWN_ERROR: connect ETIMEDOUT 192.0.66.233:443\n    at error (/opt/opencti/build/src/config/errors.js:8:10)\n    at UnknownError (/opt/opencti/build/src/config/errors.js:82:47)\n    at Object._logWithError (/opt/opencti/build/src/config/conf.js:235:17)\n    at Object.error (/opt/opencti/build/src/config/conf.js:244:48)\n    at /opt/opencti/build/src/manager/ingestionManager.ts:198:16\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async Promise.all (index 16)\n    at async Promise.all (index 0)\n    at ingestionHandler (/opt/opencti/build/src/manager/ingestionManager.ts:369:5)\n    at /opt/opencti/build/src/manager/ingestionManager.ts:388:9\n    at Wbt.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13)\n    at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)"},{"message":"connect ETIMEDOUT 192.0.66.233:443","name":"Error","stack":"Error: connect ETIMEDOUT 192.0.66.233:443\n    at Function.Gce.from (/opt/opencti/build/node_modules/axios/lib/core/AxiosError.js:89:14)\n    at bx.handleRequestError (/opt/opencti/build/node_modules/axios/lib/adapters/http.js:610:25)\n    at bx.emit (node:events:519:28)\n    at ClientRequest.Ayn.<computed> (/opt/opencti/build/node_modules/follow-redirects/index.js:38:24)\n    at ClientRequest.emit (node:events:519:28)\n    at TLSSocket.socketErrorListener (node:_http_client:500:9)\n    at TLSSocket.emit (node:events:519:28)\n    at emitErrorNT (node:internal/streams/destroy:169:8)\n    at emitErrorCloseNT (node:internal/streams/destroy:128:3)\n    at processTicksAndRejections (node:internal/process/task_queues:82:21)\n    at Hyn.request (/opt/opencti/build/node_modules/axios/lib/core/Axios.js:45:41)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at /opt/opencti/build/src/manager/ingestionManager.ts:117:22\n    at rssDataHandler (/opt/opencti/build/src/manager/ingestionManager.ts:147:16)\n    at async Promise.all (index 16)\n    at async Promise.all (index 0)\n    at ingestionHandler (/opt/opencti/build/src/manager/ingestionManager.ts:369:5)\n    at /opt/opencti/build/src/manager/ingestionManager.ts:388:9\n    at Wbt.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13)\n    at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)"}] name=Malwarebytes Labs source=backend timestamp=2024-07-10T00:41:23.680Z version=6.2.0

ory":"APP","context":"Rss execution","errors":[{"attributes":{"genre":"TECHNICAL","http_status":500},"message":"Request failed with status code 404","name":"UNKNOWN_ERROR","stack":"UNKNOWN_ERROR: Request failed with status code 404\n    at error (/opt/opencti/build/src/config/errors.js:8:10)\n    at UnknownError (/opt/opencti/build/src/config/errors.js:82:47)\n    at Object._logWithError (/opt/opencti/build/src/config/conf.js:235:17)\n    at Object.error (/opt/opencti/build/src/config/conf.js:244:48)\n    at /opt/opencti/build/src/manager/ingestionManager.ts:198:16\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async Promise.all (index 9)\n    at async Promise.all (index 0)\n    at ingestionHandler (/opt/opencti/build/src/manager/ingestionManager.ts:369:5)\n    at /opt/opencti/build/src/manager/ingestionManager.ts:388:9\n    at Wbt.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13)\n    at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)"},{"message":"Request failed with status code 404","name":"AxiosError","stack":"AxiosError: Request failed with status code 404\n    at settle (/opt/opencti/build/node_modules/axios/lib/core/settle.js:19:12)\n    at Unzip.handleStreamEnd (/opt/opencti/build/node_modules/axios/lib/adapters/http.js:589:11)\n    at Unzip.emit (node:events:531:35)\n    at endReadableNT (node:internal/streams/readable:1696:12)\n    at processTicksAndRejections (node:internal/process/task_queues:82:21)\n    at Hyn.request (/opt/opencti/build/node_modules/axios/lib/core/Axios.js:45:41)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at /opt/opencti/build/src/manager/ingestionManager.ts:117:22\n    at rssDataHandler (/opt/opencti/build/src/manager/ingestionManager.ts:147:16)\n    at async Promise.all (index 9)\n    at async Promise.all (index 0)\n    at ingestionHandler (/opt/opencti/build/src/manager/ingestionManager.ts:369:5)\n    at /opt/opencti/build/src/manager/ingestionManager.ts:388:9\n    at Wbt.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13)\n    at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)"}],"level":"error","message":"Request failed with status code 404","name":"Dark Reading","source":"backend","timestamp":"2024-07-10T01:59:42.875Z","version":"6.2.0"}

ory":"APP","context":"Rss execution","errors":[{"attributes":{"genre":"TECHNICAL","http_status":500},"message":"Request failed with status code 504","name":"UNKNOWN_ERROR","stack":"UNKNOWN_ERROR: Request failed with status code 504\n    at error (/opt/opencti/build/src/config/errors.js:8:10)\n    at UnknownError (/opt/opencti/build/src/config/errors.js:82:47)\n    at Object._logWithError (/opt/opencti/build/src/config/conf.js:235:17)\n    at Object.error (/opt/opencti/build/src/config/conf.js:244:48)\n    at /opt/opencti/build/src/manager/ingestionManager.ts:198:16\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async Promise.all (index 12)\n    at async Promise.all (index 0)\n    at ingestionHandler (/opt/opencti/build/src/manager/ingestionManager.ts:369:5)\n    at /opt/opencti/build/src/manager/ingestionManager.ts:388:9\n    at Wbt.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13)\n    at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)"},{"message":"Request failed with status code 504","name":"AxiosError","stack":"AxiosError: Request failed with status code 504\n    at settle (/opt/opencti/build/node_modules/axios/lib/core/settle.js:19:12)\n    at IncomingMessage.handleStreamEnd (/opt/opencti/build/node_modules/axios/lib/adapters/http.js:589:11)\n    at IncomingMessage.emit (node:events:531:35)\n    at endReadableNT (node:internal/streams/readable:1696:12)\n    at processTicksAndRejections (node:internal/process/task_queues:82:21)\n    at Hyn.request (/opt/opencti/build/node_modules/axios/lib/core/Axios.js:45:41)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at /opt/opencti/build/src/manager/ingestionManager.ts:117:22\n    at rssDataHandler (/opt/opencti/build/src/manager/ingestionManager.ts:147:16)\n    at async Promise.all (index 12)\n    at async Promise.all (index 0)\n    at ingestionHandler (/opt/opencti/build/src/manager/ingestionManager.ts:369:5)\n    at /opt/opencti/build/src/manager/ingestionManager.ts:388:9\n    at Wbt.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13)\n    at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)"}],"level":"error","message":"Request failed with status code 504","name":"SOC Prime","source":"backend","timestamp":"2024-07-10T07:40:22.945Z","version":"6.2.0"}

Environment

6.2.0

Reproducible Steps

Configure an RSS feed as above and start

Expected Output

RSS feed accesses feed URL and creates reports

Actual Output

Reports are created either extremely slowly or not at all. One RSS feed that I deleted still creates reports.

Additional information

I wanted to ask as well, when a feed URL downloads a feed file rather than links directly to the XML feed page, will this still work? Example is https://socprime.com/blog/feed/

romain-filigran commented 2 months ago

Hello @dominictory, we're aware of some problems with the RSS ingestor that we're looking into. To answer your second question, yes, it works. I've just tried it with the source you mention and OpenCTI retrieves the articles just fine. Capture d'écran 2024-07-16 130440

romain-filigran commented 2 months ago

Are RSS reports now imported into your instance?

For your information, RSS ingestion is triggered every 5 minutes, so there may be an import delay.

If you notice that some reports are still not being imported, please try to connect to the RabbitMQ connector and let me know the result of the command:

rabbitmqctl list_queues

dominictory commented 2 months ago

Are RSS reports now imported into your instance?

For your information, RSS ingestion is triggered every 5 minutes, so there may be an import delay.

If you notice that some reports are still not being imported, please try to connect to the RabbitMQ connector and let me know the result of the command:

rabbitmqctl list_queues

RSS feeds appear to be working fine now, happy to close.