SouadHadjiat
commented
6 months ago Hello @ckane, Could you please send your complete opencti log ? This would help us identify more precisely at which step the start up is failing.
Closed ckane closed 6 months ago
SouadHadjiat
commented
6 months ago Hello @ckane, Could you please send your complete opencti log ? This would help us identify more precisely at which step the start up is failing.
ckane
commented
6 months ago @SouadHadjiat here are the logs from opencti when running with debug logging level:
opencti-1 | {"category":"APP","environment":"production","level":"info","message":"[OPENCTI] Starting platform","source":"backend","timestamp":"2024-05-14T15:04:29.906Z","version":"6.1.0"}
opencti-1 | {"category":"APP","level":"info","message":"[OPENCTI] Checking dependencies statuses","source":"backend","timestamp":"2024-05-14T15:04:29.909Z","version":"6.1.0"}
opencti-1 | {"category":"APP","level":"info","message":"[SEARCH] Engine client not specified, trying to discover it with opensearch client","source":"backend","timestamp":"2024-05-14T15:04:29.920Z","version":"6.1.0"}
opencti-1 | {"category":"APP","level":"info","message":"[SEARCH] Engine detected to elk","source":"backend","timestamp":"2024-05-14T15:04:29.952Z","version":"6.1.0"}
opencti-1 | {"category":"APP","level":"info","message":"[SEARCH] elk (8.13.4) client selected / runtime sorting enabled / attachment processor enabled","source":"backend","timestamp":"2024-05-14T15:04:29.979Z","version":"6.1.0"}
opencti-1 | {"category":"APP","level":"info","message":"[CHECK] Search engine is alive","source":"backend","timestamp":"2024-05-14T15:04:29.980Z","version":"6.1.0"}
opencti-1 | {"category":"APP","cause":{"$metadata":{"attempts":1,"totalRetryDelay":0}},"level":"error","message":"[OPENCTI] System dependencies check failed","source":"backend","timestamp":"2024-05-14T15:04:29.998Z","version":"6.1.0"}
opencti-1 | {"category":"APP","errors":[{"attributes":{"genre":"TECHNICAL","http_status":500},"message":"Resolved credential object is not valid","name":"UNKNOWN_ERROR","stack":"UNKNOWN_ERROR: Resolved credential object is not valid\n at error (/opt/opencti/build/src/config/errors.js:8:10)\n at UnknownError (/opt/opencti/build/src/config/errors.js:82:47)\n at Object._logWithError (/opt/opencti/build/src/config/conf.js:218:15)\n at Object.error (/opt/opencti/build/src/config/conf.js:227:48)\n at platformStart (/opt/opencti/build/src/boot.js:41:12)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)"},{"message":"Resolved credential object is not valid","name":"Error","stack":"Error: Resolved credential object is not valid\n at KMe.validateResolvedCredentials (/opt/opencti/build/node_modules/@smithy/signature-v4/dist-cjs/index.js:562:13)\n at KMe.signRequest (/opt/opencti/build/node_modules/@smithy/signature-v4/dist-cjs/index.js:487:10)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)"}],"level":"error","message":"Platform unmanaged direct error","source":"backend","timestamp":"2024-05-14T15:04:30.001Z","version":"6.1.0"}
opencti-1 | {"category":"APP","environment":"production","level":"info","message":"[OPENCTI] Starting platform","source":"backend","timestamp":"2024-05-14T15:04:37.858Z","version":"6.1.0"}
opencti-1 | {"category":"APP","level":"info","message":"[OPENCTI] Checking dependencies statuses","source":"backend","timestamp":"2024-05-14T15:04:37.861Z","version":"6.1.0"}
opencti-1 | {"category":"APP","level":"info","message":"[SEARCH] Engine client not specified, trying to discover it with opensearch client","source":"backend","timestamp":"2024-05-14T15:04:37.872Z","version":"6.1.0"}
opencti-1 | {"category":"APP","level":"info","message":"[SEARCH] Engine detected to elk","source":"backend","timestamp":"2024-05-14T15:04:37.905Z","version":"6.1.0"}
opencti-1 | {"category":"APP","level":"info","message":"[SEARCH] elk (8.13.4) client selected / runtime sorting enabled / attachment processor enabled","source":"backend","timestamp":"2024-05-14T15:04:37.930Z","version":"6.1.0"}
opencti-1 | {"category":"APP","level":"info","message":"[CHECK] Search engine is alive","source":"backend","timestamp":"2024-05-14T15:04:37.931Z","version":"6.1.0"}
opencti-1 | {"category":"APP","cause":{"$metadata":{"attempts":1,"totalRetryDelay":0}},"level":"error","message":"[OPENCTI] System dependencies check failed","source":"backend","timestamp":"2024-05-14T15:04:37.947Z","version":"6.1.0"}
opencti-1 | {"category":"APP","errors":[{"attributes":{"genre":"TECHNICAL","http_status":500},"message":"Resolved credential object is not valid","name":"UNKNOWN_ERROR","stack":"UNKNOWN_ERROR: Resolved credential object is not valid\n at error (/opt/opencti/build/src/config/errors.js:8:10)\n at UnknownError (/opt/opencti/build/src/config/errors.js:82:47)\n at Object._logWithError (/opt/opencti/build/src/config/conf.js:218:15)\n at Object.error (/opt/opencti/build/src/config/conf.js:227:48)\n at platformStart (/opt/opencti/build/src/boot.js:41:12)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)"},{"message":"Resolved credential object is not valid","name":"Error","stack":"Error: Resolved credential object is not valid\n at KMe.validateResolvedCredentials (/opt/opencti/build/node_modules/@smithy/signature-v4/dist-cjs/index.js:562:13)\n at KMe.signRequest (/opt/opencti/build/node_modules/@smithy/signature-v4/dist-cjs/index.js:487:10)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)"}],"level":"error","message":"Platform unmanaged direct error","source":"backend","timestamp":"2024-05-14T15:04:37.950Z","version":"6.1.0"}The only containers running for this test are redis, elasticsearch, rabbitmq, and opencti - to eliminate any potential interference by other containers. This is the second ARM64 (AWS Graviton) server that I am able to repeat the behavior on. I had opencti replicated on this server with 2 replicas, but even setting that down to 1 replica, the failure remains, so it isn't related to clustering either.
Similar to reported before, if I change the version back to 6.0.10 then everything works fine again (albeit on the old version) and there's no indication of data corruption in elasticsearch.
ckane
commented
6 months ago My docker-config.yml: https://github.com/ckane/opencti-docker/blob/tf-main/docker-compose.yml
SouadHadjiat
commented
6 months ago Startup is failing during Minio/S3 storage initialisation, did something change in your configuration regarding your storage connection ? @ckane
ckane
commented
6 months ago No, it's the exact same configuration for both: using AWS S3 and it's running on an ec2 instance with an IAM execution role for access. Same configuration works using 6.0.10 and stops working when version is changed to 6.1.0, then works again when version changed back to 6.0.10.
richard-julien
commented
6 months ago I found the problem @ckane , misunderstanding on my side about the lib and his capability to handle async function. Problem will be solved in 6.1.2
ckane
commented
6 months ago TY! Glad you were able to track it down so quickly
Description
I tried upgrading from 6.0.10 to 6.1.0 today, and
openctiwon't start up and keeps sending the following errors to the log:I was able to successfully downgrade the same system back to 6.0.10 and it continued to work fine on that version. I tried again to upgrade to 6.1.0, consistently failed, but was still able to revert to 6.0.10 in place (so it doesn't appear to be due to a data corruption in ElasticSearch or something like that). The platform is connecting to ElasticSearch, and this error appears to occur shortly afterward.
Environment
Reproducible Steps
Steps to create the smallest reproducible scenario:
docker-compose.ymlopenctiI am deploying on an arm64 AWS Graviton instance ... maybe something wrong/missing with the arm64 container image?
Expected Output
Expected seamless upgrade
Actual Output
openctiwould never start. Had to revert system to 6.0.10 in order for it to work