Open dominictory opened 3 weeks ago
I @dominictory. So your logs show a lot of "TimeoutError: Request timed out" related to elastic. You also have some CONNRESET. For me it could be a sign of:
I @dominictory. So your logs show a lot of "TimeoutError: Request timed out" related to elastic. You also have some CONNRESET. For me it could be a sign of:
- undersized elasticsearch
- Stability issue on redis or elastic
- network issues Difficult to help more
Feels like network issues, but intermittently as bundles do eventually get processed (albeit slowly), then this stops. In RabbitMQ, I see the below showing that workers keep disconnecting from it. Why might this be happening when the server is not under heavy load? For the health checks, might increasing the interval/timeout/retries values help?
2024-11-06T10:24:56.546011051Z 2024-11-06 10:24:56.545169+00:00 [warning] <0.125861.0> closing AMQP connection <0.125861.0> (172.25.0.48:48816 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:56.546052965Z 2024-11-06 10:24:56.545169+00:00 [warning] <0.125861.0> client unexpectedly closed TCP connection
2024-11-06T10:24:56.547771429Z 2024-11-06 10:24:56.545506+00:00 [warning] <0.125915.0> closing AMQP connection <0.125915.0> (172.25.0.48:48824 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:56.547831218Z 2024-11-06 10:24:56.545506+00:00 [warning] <0.125915.0> client unexpectedly closed TCP connection
2024-11-06T10:24:56.547851409Z 2024-11-06 10:24:56.545698+00:00 [warning] <0.125972.0> closing AMQP connection <0.125972.0> (172.25.0.48:48828 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:56.547868721Z 2024-11-06 10:24:56.545698+00:00 [warning] <0.125972.0> client unexpectedly closed TCP connection
2024-11-06T10:24:56.547987644Z 2024-11-06 10:24:56.545764+00:00 [warning] <0.126032.0> closing AMQP connection <0.126032.0> (172.25.0.48:48832 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:56.548028644Z 2024-11-06 10:24:56.545764+00:00 [warning] <0.126032.0> client unexpectedly closed TCP connection
2024-11-06T10:24:56.549025465Z 2024-11-06 10:24:56.546536+00:00 [warning] <0.126245.0> closing AMQP connection <0.126245.0> (172.25.0.48:48862 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:56.549044703Z 2024-11-06 10:24:56.546536+00:00 [warning] <0.126245.0> client unexpectedly closed TCP connection
2024-11-06T10:24:56.549094285Z 2024-11-06 10:24:56.546632+00:00 [warning] <0.126142.0> closing AMQP connection <0.126142.0> (172.25.0.48:48846 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:56.549116221Z 2024-11-06 10:24:56.546632+00:00 [warning] <0.126142.0> client unexpectedly closed TCP connection
2024-11-06T10:24:56.550258484Z 2024-11-06 10:24:56.546747+00:00 [warning] <0.126353.0> closing AMQP connection <0.126353.0> (172.25.0.48:47202 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:56.550284374Z 2024-11-06 10:24:56.546747+00:00 [warning] <0.126353.0> client unexpectedly closed TCP connection
2024-11-06T10:24:56.550574367Z 2024-11-06 10:24:56.546984+00:00 [warning] <0.126498.0> closing AMQP connection <0.126498.0> (172.25.0.48:47214 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:56.550599060Z 2024-11-06 10:24:56.546984+00:00 [warning] <0.126498.0> client unexpectedly closed TCP connection
2024-11-06T10:24:56.550614094Z 2024-11-06 10:24:56.547603+00:00 [warning] <0.126580.0> closing AMQP connection <0.126580.0> (172.25.0.48:47220 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:56.550627583Z 2024-11-06 10:24:56.547603+00:00 [warning] <0.126580.0> client unexpectedly closed TCP connection
2024-11-06T10:24:56.550875141Z 2024-11-06 10:24:56.547523+00:00 [warning] <0.130067.0> closing AMQP connection <0.130067.0> (172.25.0.48:51788 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:56.550900211Z 2024-11-06 10:24:56.547523+00:00 [warning] <0.130067.0> client unexpectedly closed TCP connection
2024-11-06T10:24:56.550915287Z 2024-11-06 10:24:56.547762+00:00 [warning] <0.129946.0> closing AMQP connection <0.129946.0> (172.25.0.48:51758 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:56.550928862Z 2024-11-06 10:24:56.547762+00:00 [warning] <0.129946.0> client unexpectedly closed TCP connection
2024-11-06T10:24:56.550942628Z 2024-11-06 10:24:56.548075+00:00 [warning] <0.130000.0> closing AMQP connection <0.130000.0> (172.25.0.48:51768 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:56.550982067Z 2024-11-06 10:24:56.548075+00:00 [warning] <0.130000.0> client unexpectedly closed TCP connection
2024-11-06T10:24:56.551203591Z 2024-11-06 10:24:56.547898+00:00 [warning] <0.129982.0> closing AMQP connection <0.129982.0> (172.25.0.48:51764 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:56.551225365Z 2024-11-06 10:24:56.547898+00:00 [warning] <0.129982.0> client unexpectedly closed TCP connection
2024-11-06T10:24:56.551239063Z 2024-11-06 10:24:56.548370+00:00 [warning] <0.130025.0> closing AMQP connection <0.130025.0> (172.25.0.48:51770 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:56.551252206Z 2024-11-06 10:24:56.548370+00:00 [warning] <0.130025.0> client unexpectedly closed TCP connection
2024-11-06T10:24:56.551268711Z 2024-11-06 10:24:56.548369+00:00 [warning] <0.130042.0> closing AMQP connection <0.130042.0> (172.25.0.48:51776 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:56.551283733Z 2024-11-06 10:24:56.548369+00:00 [warning] <0.130042.0> client unexpectedly closed TCP connection
2024-11-06T10:24:56.551298110Z 2024-11-06 10:24:56.549036+00:00 [warning] <0.130085.0> closing AMQP connection <0.130085.0> (172.25.0.48:51796 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:56.551311371Z 2024-11-06 10:24:56.549036+00:00 [warning] <0.130085.0> client unexpectedly closed TCP connection
2024-11-06T10:24:56.551750907Z 2024-11-06 10:24:56.549278+00:00 [warning] <0.130102.0> closing AMQP connection <0.130102.0> (172.25.0.48:51804 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:56.551819033Z 2024-11-06 10:24:56.549278+00:00 [warning] <0.130102.0> client unexpectedly closed TCP connection
2024-11-06T10:24:56.551870926Z 2024-11-06 10:24:56.549286+00:00 [warning] <0.130119.0> closing AMQP connection <0.130119.0> (172.25.0.48:51812 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:56.551887024Z 2024-11-06 10:24:56.549286+00:00 [warning] <0.130119.0> client unexpectedly closed TCP connection
2024-11-06T10:24:57.602306974Z 2024-11-06 10:24:57.601362+00:00 [warning] <0.125937.0> closing AMQP connection <0.125937.0> (172.25.0.29:47568 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:57.602349697Z 2024-11-06 10:24:57.601362+00:00 [warning] <0.125937.0> client unexpectedly closed TCP connection
2024-11-06T10:24:57.602359197Z 2024-11-06 10:24:57.601628+00:00 [warning] <0.125898.0> closing AMQP connection <0.125898.0> (172.25.0.29:47566 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:57.602367122Z 2024-11-06 10:24:57.601628+00:00 [warning] <0.125898.0> client unexpectedly closed TCP connection
2024-11-06T10:24:57.602473680Z 2024-11-06 10:24:57.601545+00:00 [warning] <0.125842.0> closing AMQP connection <0.125842.0> (172.25.0.29:47564 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:57.602506058Z 2024-11-06 10:24:57.601545+00:00 [warning] <0.125842.0> client unexpectedly closed TCP connection
2024-11-06T10:24:57.602913590Z 2024-11-06 10:24:57.602551+00:00 [warning] <0.126458.0> closing AMQP connection <0.126458.0> (172.25.0.29:33450 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:57.602929170Z 2024-11-06 10:24:57.602551+00:00 [warning] <0.126458.0> client unexpectedly closed TCP connection
2024-11-06T10:24:57.603039455Z 2024-11-06 10:24:57.602743+00:00 [warning] <0.126010.0> closing AMQP connection <0.126010.0> (172.25.0.29:47578 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:57.603062545Z 2024-11-06 10:24:57.602743+00:00 [warning] <0.126010.0> client unexpectedly closed TCP connection
2024-11-06T10:24:57.603439729Z 2024-11-06 10:24:57.602953+00:00 [warning] <0.126228.0> closing AMQP connection <0.126228.0> (172.25.0.29:47596 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:57.603462200Z 2024-11-06 10:24:57.602953+00:00 [warning] <0.126228.0> client unexpectedly closed TCP connection
2024-11-06T10:24:57.603611859Z 2024-11-06 10:24:57.602910+00:00 [warning] <0.126127.0> closing AMQP connection <0.126127.0> (172.25.0.29:47592 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:57.603627466Z 2024-11-06 10:24:57.602910+00:00 [warning] <0.126127.0> client unexpectedly closed TCP connection
2024-11-06T10:24:57.603849466Z 2024-11-06 10:24:57.603163+00:00 [warning] <0.126328.0> closing AMQP connection <0.126328.0> (172.25.0.29:47612 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:57.603866117Z 2024-11-06 10:24:57.603163+00:00 [warning] <0.126328.0> client unexpectedly closed TCP connection
2024-11-06T10:24:57.604468578Z 2024-11-06 10:24:57.603835+00:00 [warning] <0.126527.0> closing AMQP connection <0.126527.0> (172.25.0.29:33462 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:57.604486050Z 2024-11-06 10:24:57.603835+00:00 [warning] <0.126527.0> client unexpectedly closed TCP connection
2024-11-06T10:24:57.605163023Z 2024-11-06 10:24:57.603902+00:00 [warning] <0.129793.0> closing AMQP connection <0.129793.0> (172.25.0.29:41496 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:57.605198165Z 2024-11-06 10:24:57.603902+00:00 [warning] <0.129793.0> client unexpectedly closed TCP connection
2024-11-06T10:24:57.605215790Z 2024-11-06 10:24:57.603949+00:00 [warning] <0.129810.0> closing AMQP connection <0.129810.0> (172.25.0.29:41510 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:57.605246026Z 2024-11-06 10:24:57.603949+00:00 [warning] <0.129810.0> client unexpectedly closed TCP connection
2024-11-06T10:24:57.605312953Z 2024-11-06 10:24:57.604216+00:00 [warning] <0.129844.0> closing AMQP connection <0.129844.0> (172.25.0.29:41532 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:57.605338413Z 2024-11-06 10:24:57.604216+00:00 [warning] <0.129844.0> client unexpectedly closed TCP connection
2024-11-06T10:24:57.605463009Z 2024-11-06 10:24:57.604228+00:00 [warning] <0.129827.0> closing AMQP connection <0.129827.0> (172.25.0.29:41524 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:57.605475769Z 2024-11-06 10:24:57.604228+00:00 [warning] <0.129827.0> client unexpectedly closed TCP connection
2024-11-06T10:24:57.605535648Z 2024-11-06 10:24:57.604413+00:00 [warning] <0.129861.0> closing AMQP connection <0.129861.0> (172.25.0.29:41534 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:57.605547398Z 2024-11-06 10:24:57.604413+00:00 [warning] <0.129861.0> client unexpectedly closed TCP connection
2024-11-06T10:24:57.605721591Z 2024-11-06 10:24:57.604569+00:00 [warning] <0.129878.0> closing AMQP connection <0.129878.0> (172.25.0.29:41540 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:57.605734739Z 2024-11-06 10:24:57.604569+00:00 [warning] <0.129878.0> client unexpectedly closed TCP connection
2024-11-06T10:24:57.605755636Z 2024-11-06 10:24:57.605338+00:00 [warning] <0.129895.0> closing AMQP connection <0.129895.0> (172.25.0.29:41556 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:57.605764098Z 2024-11-06 10:24:57.605338+00:00 [warning] <0.129895.0> client unexpectedly closed TCP connection
2024-11-06T10:24:57.605890008Z 2024-11-06 10:24:57.605349+00:00 [warning] <0.129913.0> closing AMQP connection <0.129913.0> (172.25.0.29:41560 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:57.605903232Z 2024-11-06 10:24:57.605349+00:00 [warning] <0.129913.0> client unexpectedly closed TCP connection
2024-11-06T10:24:57.605962844Z 2024-11-06 10:24:57.605473+00:00 [warning] <0.129931.0> closing AMQP connection <0.129931.0> (172.25.0.29:41574 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):
2024-11-06T10:24:57.605988721Z 2024-11-06 10:24:57.605473+00:00 [warning] <0.129931.0> client unexpectedly closed TCP connection
I also see the below RabbitMQ errors:
2024-11-06T10:24:04.761345202Z 2024-11-06 10:24:04.760892+00:00 [error] <0.133857.0> Channel error on connection <0.133847.0> (172.25.0.29:53380 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'), channel 1:
2024-11-06T10:24:04.761378515Z 2024-11-06 10:24:04.760892+00:00 [error] <0.133857.0> operation basic.consume caused a channel exception not_found: no queue 'push_26387a1c-edd1-444b-baa6-a2de3cc3f9a6' in vhost '/'
Workers directly consume the rabbitmq to process the messages. If you have connection errors here it could be a problem on rabbitmq or a general problem in your network, really hard to know
Workers directly consume the rabbitmq to process the messages. If you have connection errors here it could be a problem on rabbitmq or a general problem in your network, really hard to know
Hi, I notice when ingestion goes to 0, I see Redis errors. Not sure why as there is plenty resource available. Interestingly, I see this:
2024-11-19T13:51:38.890688789Z ERR Http call interceptor fail | category=APP errors=[{"attributes":{"genre":"TECHNICAL","http_status":500},"message":"Http call interceptor fail","name":"UNKNOWN_ERROR","stack":"GraphQLError: Http call interceptor fail\n at error (/opt/opencti/build/src/config/errors.js:7:10)\n at UnknownError (/opt/opencti/build/src/config/errors.js:81:47)\n at fn (/opt/opencti/build/src/http/httpPlatform.js:510:19)\n at Qge.handle_error (/opt/opencti/build/node_modules/express/lib/router/layer.js:71:5)\n at trim_prefix (/opt/opencti/build/node_modules/express/lib/router/index.js:326:13)\n at done (/opt/opencti/build/node_modules/express/lib/router/index.js:286:9)\n at Function.process_params (/opt/opencti/build/node_modules/express/lib/router/index.js:346:12)\n at next (/opt/opencti/build/node_modules/express/lib/router/index.js:280:10)\n at Qge.handle_error (/opt/opencti/build/node_modules/express/lib/router/layer.js:67:12)\n at trim_prefix (/opt/opencti/build/node_modules/express/lib/router/index.js:326:13)\n at done (/opt/opencti/build/node_modules/express/lib/router/index.js:286:9)\n at Function.process_params (/opt/opencti/build/node_modules/express/lib/router/index.js:346:12)\n at next (/opt/opencti/build/node_modules/express/lib/router/index.js:280:10)\n at Qge.handle_error (/opt/opencti/build/node_modules/express/lib/router/layer.js:67:12)\n at trim_prefix (/opt/opencti/build/node_modules/express/lib/router/index.js:326:13)\n at done (/opt/opencti/build/node_modules/express/lib/router/index.js:286:9)\n at Function.process_params (/opt/opencti/build/node_modules/express/lib/router/index.js:346:12)\n at next (/opt/opencti/build/node_modules/express/lib/router/index.js:280:10)\n at Qge.handle_error (/opt/opencti/build/node_modules/express/lib/router/layer.js:67:12)\n at trim_prefix (/opt/opencti/build/node_modules/express/lib/router/index.js:326:13)\n at done (/opt/opencti/build/node_modules/express/lib/router/index.js:286:9)\n at Function.process_params (/opt/opencti/build/node_modules/express/lib/router/index.js:346:12)"},{"message":"stream is not readable","name":"InternalServerError","stack":"InternalServerError: stream is not readable\n at readStream (/opt/opencti/build/node_modules/raw-body/index.js:185:17)\n at getBody (/opt/opencti/build/node_modules/raw-body/index.js:116:12)\n at read (/opt/opencti/build/node_modules/body-parser/lib/read.js:79:3)\n at fn (/opt/opencti/build/node_modules/body-parser/lib/types/json.js:138:5)\n at Qge.handle [as handle_request] (/opt/opencti/build/node_modules/express/lib/router/layer.js:95:5)\n at trim_prefix (/opt/opencti/build/node_modules/express/lib/router/index.js:328:13)\n at done (/opt/opencti/build/node_modules/express/lib/router/index.js:286:9)\n at Function.process_params (/opt/opencti/build/node_modules/express/lib/router/index.js:346:12)\n at next (/opt/opencti/build/node_modules/express/lib/router/index.js:280:10)\n at fn (/opt/opencti/build/node_modules/passport/lib/middleware/initialize.js:98:5)\n at Qge.handle [as handle_request] (/opt/opencti/build/node_modules/express/lib/router/layer.js:95:5)\n at trim_prefix (/opt/opencti/build/node_modules/express/lib/router/index.js:328:13)\n at done (/opt/opencti/build/node_modules/express/lib/router/index.js:286:9)\n at Function.process_params (/opt/opencti/build/node_modules/express/lib/router/index.js:346:12)\n at next (/opt/opencti/build/node_modules/express/lib/router/index.js:280:10)\n at cb (/opt/opencti/build/node_modules/express-session/index.js:514:7)\n at cb (/opt/opencti/build/src/database/sessionStore-redis.js:38:14)\n at done (/opt/opencti/build/node_modules/async-lock/lib/index.js:104:6)\n at done (/opt/opencti/build/node_modules/async-lock/lib/index.js:157:7)\n at /opt/opencti/build/src/database/sessionStore-redis.js:34:16\n at processTicksAndRejections (node:internal/process/task_queues:95:5)"}] source=backend timestamp=2024-11-19T13:51:38.890Z version=6.3.12
Followed by:
2024-11-19T13:52:13.204347378Z ERR [REDIS] Failed to extend resource | category=APP manager=PLAYBOOK_MANAGER source=backend timestamp=2024-11-19T13:52:13.203Z version=6.3.12
2024-11-19T13:52:13.205227759Z ERR [REDIS] Failed to extend resource | category=APP manager=NOTIFICATION_MANAGER source=backend timestamp=2024-11-19T13:52:13.205Z version=6.3.12
2024-11-19T14:04:28.561901304Z ERR [REDIS] Failed to extend resource | category=APP manager=PLAYBOOK_MANAGER source=backend timestamp=2024-11-19T14:04:28.561Z version=6.3.12
2024-11-19T14:04:28.562351859Z ERR [REDIS] Failed to extend resource | category=APP manager=PUBLISHER_MANAGER source=backend timestamp=2024-11-19T14:04:28.562Z version=6.3.12
2024-11-19T14:04:28.704856463Z ERR [REDIS] Failed to extend resource | category=APP manager=SYNC_MANAGER source=backend timestamp=2024-11-19T14:04:28.704Z version=6.3.12
2024-11-19T14:04:56.050114555Z ERR [REDIS] Failed to extend resource | category=APP manager=NOTIFICATION_MANAGER source=backend timestamp=2024-11-19T14:04:56.049Z version=6.3.12
2024-11-19T14:04:56.050749282Z ERR [REDIS] Failed to extend resource | category=APP manager=FILE_INDEX_MANAGER source=backend timestamp=2024-11-19T14:04:56.050Z version=6.3.12
2024-11-19T14:04:56.051124843Z ERR [REDIS] Failed to extend resource | category=APP manager=HISTORY_MANAGER source=backend timestamp=2024-11-19T14:04:56.050Z version=6.3.12
2024-11-19T14:05:28.668389192Z ERR [REDIS] Failed to extend resource | category=APP manager=PLAYBOOK_MANAGER source=backend timestamp=2024-11-19T14:05:28.668Z version=6.3.12
2024-11-19T14:05:28.669282576Z ERR [REDIS] Failed to extend resource | category=APP manager=NOTIFICATION_MANAGER source=backend timestamp=2024-11-19T14:05:28.669Z version=6.3.12
2024-11-19T14:06:28.227151449Z (node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Socket]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit
2024-11-19T14:06:28.227201678Z (node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Socket]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit
2024-11-19T14:06:38.232223781Z (node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Socket]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit
2024-11-19T14:06:41.433610954Z (node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Socket]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit
2024-11-19T14:06:43.783062168Z (node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Socket]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit
2024-11-19T14:31:32.974197302Z ERR [REDIS] Failed to extend resource | category=APP manager=SYNC_MANAGER source=backend timestamp=2024-11-19T14:31:32.973Z version=6.3.12
2024-11-19T14:31:51.087401987Z ERR [REDIS] Failed to extend resource | category=APP manager=PLAYBOOK_MANAGER source=backend timestamp=2024-11-19T14:31:51.086Z version=6.3.12
2024-11-19T14:31:51.088535129Z ERR [REDIS] Failed to extend resource | category=APP manager=PUBLISHER_MANAGER source=backend timestamp=2024-11-19T14:31:51.088Z version=6.3.12
2024-11-19T14:31:51.088981302Z ERR [REDIS] Failed to extend resource | category=APP manager=HISTORY_MANAGER source=backend timestamp=2024-11-19T14:31:51.088Z version=6.3.12
2024-11-19T14:31:51.089486082Z ERR [REDIS] Failed to extend resource | category=APP manager=FILE_INDEX_MANAGER source=backend timestamp=2024-11-19T14:31:51.089Z version=6.3.12
2024-11-19T14:31:51.089820070Z ERR [REDIS] Failed to extend resource | category=APP manager=NOTIFICATION_MANAGER source=backend timestamp=2024-11-19T14:31:51.089Z version=6.3.12
2024-11-19T14:32:32.820099024Z ERR [REDIS] Failed to extend resource | category=APP manager=PLAYBOOK_MANAGER source=backend timestamp=2024-11-19T14:32:32.819Z version=6.3.12
2024-11-19T14:32:32.821175144Z ERR [REDIS] Failed to extend resource | category=APP manager=NOTIFICATION_MANAGER source=backend timestamp=2024-11-19T14:32:32.820Z version=6.3.12
2024-11-19T14:33:32.900546556Z (node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Socket]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit
2024-11-19T14:33:32.900588266Z (node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Socket]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit
2024-11-19T14:33:34.183766645Z (node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Socket]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit
2024-11-19T14:33:34.183827267Z (node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Socket]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit
2024-11-19T14:33:38.152781775Z (node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Socket]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit
2024-11-19T14:52:09.116799274Z ERR [REDIS] Failed to extend resource | category=APP manager=RULE_ENGINE source=backend timestamp=2024-11-19T14:52:09.116Z version=6.3.12
2024-11-19T14:52:09.706540422Z ERR [REDIS] Failed to extend resource | category=APP manager=SYNC_MANAGER source=backend timestamp=2024-11-19T14:52:09.706Z version=6.3.12
2024-11-19T14:52:11.360711244Z ERR [REDIS] Failed to extend resource | category=APP manager=PUBLISHER_MANAGER source=backend timestamp=2024-11-19T14:52:11.360Z version=6.3.12
2024-11-19T14:52:11.361108232Z ERR [REDIS] Failed to extend resource | category=APP manager=PLAYBOOK_MANAGER source=backend timestamp=2024-11-19T14:52:11.360Z version=6.3.12
2024-11-19T14:52:11.361651575Z ERR [REDIS] Failed to extend resource | category=APP manager=NOTIFICATION_MANAGER source=backend timestamp=2024-11-19T14:52:11.361Z version=6.3.12
2024-11-19T14:52:53.205200384Z ERR [REDIS] Failed to extend resource | category=APP manager=PLAYBOOK_MANAGER source=backend timestamp=2024-11-19T14:52:53.204Z version=6.3.12
2024-11-19T14:52:53.206582203Z ERR [REDIS] Failed to extend resource | category=APP manager=NOTIFICATION_MANAGER source=backend timestamp=2024-11-19T14:52:53.206Z version=6.3.12
2024-11-19T14:53:55.608742352Z (node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Socket]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit
2024-11-19T14:53:55.608833311Z (node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Socket]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit
2024-11-19T14:53:57.854654459Z (node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Socket]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit
Are the Redis errors and MaxListenersExceededWarning logs potentially linked? Is the Redis error somehow due to resource, even though there is plenty of headroom available on the server? The case is always the same, I restart the platform, everything comes up as normal, ingestion is fast (20-30 bundles/sec), then after some time passes, it slows down, eventually going to 0.
I'm no expert, but I came across the below that talks about MaxListenersExceededWarning in the context of Redis client:
I say this as since upgrading to 6.3.12, I again removed all connectors in UI and reset RabbitMQ. I see no more RabbitMQ errors/warnings since doing so. The below is something I haven't seen before with Redis, is it concering? It appears multiple times.
1:M 19 Nov 2024 14:50:54.124 # Client id=10620 addr=172.27.0.10:39176 laddr=172.27.0.39:6379 fd=13 name= age=34 idle=1 flags=P db=0 sub=0 psub=2 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=3 omem=34078792 tot-mem=34101272 events=rw cmd=psubscribe user=default redir=-1 resp=2 lib-name= lib-ver= scheduled to be closed ASAP for overcoming of output buffer limits.
Similar message on https://github.com/OpenCTI-Platform/opencti/issues/4936
There's these HUGE logs in worker logs too. This one in particular was due to AlienVault connector which currently has ~2k bundles queued and not getting processed:
This was followed by the above Redis errors.
Description
At present, I have 68k queued bundles, of which <1 are getting processed/sec, as below, which is constant at the moment. In worker and connector logs, whilst some bundles do get processed, there are sporadic connection reset errors, which point to RabbitMQ which is showing as healthy in the stack, as below, with no errors other than the occasional errors below every 5 mins or so. The backend platform does show as unhealthy, with numerous errors, as below at the bottom. Connectors seem to go between active and inactive regularly. I would appreciate some help troubleshooting further please, and especially with Redis/platform if I can try any different configurations. Thanks :)
My docker-compose setup is as follows:
1x frontend platform (users, internal export/import connectors only, no managers) 1x backend platform (3x workers, connectors, all managers)
VM: Ubuntu 22.04 LTS; 16 cores; 64GB RAM
Current load average: 7.22, 5.52, 4.45 Memory usage: total used free shared buff/cache available Mem: 62Gi 48Gi 4.1Gi 43Mi 10Gi 13Gi Swap: 8.0Gi 5.4Gi 2.6Gi
docker-compose.yml sample:
.env:
RabbitMQ log sample:
Ingestion platform logs: