search

OpenCTI-Platform / opencti

Open Cyber Threat Intelligence Platform
https://opencti.io
Other
6.16k stars 911 forks source link

Manual deployment problem #2113

Open quark95 opened 2 years ago

quark95 commented 2 years ago

Prerequisites

Description

OpenCTI test fails with "The Access Key Id you provided does not exist in our records."

After 9 attempts that involved starting again and reading all of the manuals I can find, I cannot solve this problem.

All dependencies were successfully installed and configuration files updated.

Environment

  1. OS: Ubuntu Server 22.04 LTS in Azure
  2. OpenCTI version: opencti-release-5.3.0
  3. OpenCTI client: I don't know, python3 i guess.
  4. Other environment details: ElasticSearch

Reproducible Steps

Steps to create the smallest reproducible scenario:

Build OpenCTI manually.

Additional information

sudo nano /etc/default/minio

MINIO_ACCESS_KEY="DobfhPf9Ek4zaV" MINIO_VOLUMES="/miniodata/" MINIO_OPTS="-C /etc/minio --address 0.0.0.0:9000 --console-address :9090" MINIO_SECRET_KEY="TyMygr7287LkPR3TXbWvtQMhySo2cgd3" MINIO_ROOT_USER="miniouser" MINIO_ROOT_PASSWORD="DobfhPf9Ek4zaV"

sudo nano /usr/local/opencti/config/production.json

"admin": {
  "email": "admin@opencti.io",
  "password": "zGCpp2ZZD57Yjd",
  "token": "1401bf5f-0289-48ca-9781-a386943cc680"
}

"minio": { "bucket_name": "opencti-bucket", "endpoint": "localhost", "port": 9000, "use_ssl": false, "access_key": "DobfhPf9Ek4zaV", "secret_key": "TyMygr7287LkPR3TXbWvtQMhySo2cgd3" },

npm test

{"category":"APP","level":"info","message":"[REDIS] Redis 'Pubsub subscriber' client ready","timestamp":"2022-05-25T09:02:06.619Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Cache manager initialized","timestamp":"2022-05-25T09:02:06.620Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[REDIS] Redis 'Client context' client ready","timestamp":"2022-05-25T09:02:06.624Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[INIT] Starting platform initialization","timestamp":"2022-05-25T09:02:06.625Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[INIT] New platform detected, initialization...","timestamp":"2022-05-25T09:02:06.629Z","version":"5.3.0"} {"category":"APP","error":{"data":{"_stack":"S3Error: The Access Key Id you provided does not exist in our records.\n at Object.parseError (/usr/local/opencti/node_modules/minio/dist/main/xml-parsers.js:71:11)\n at /usr/local/opencti/node_modules/minio/dist/main/transformers.js:166:22\n at DestroyableTransform._flush (/usr/local/opencti/node_modules/minio/dist/main/transformers.js:90:10)\n at DestroyableTransform.prefinish (/usr/local/opencti/node_modules/readable-stream/lib/_stream_transform.js:129:10)\n at DestroyableTransform.emit (node:events:527:28)\n at prefinish (/usr/local/opencti/node_modules/readable-stream/lib/_stream_writable.js:611:14)\n at finishMaybe (/usr/local/opencti/node_modules/readable-stream/lib/_stream_writable.js:620:5)\n at endWritable (/usr/local/opencti/node_modules/readable-stream/lib/_stream_writable.js:643:3)\n at DestroyableTransform.Object..Writable.end (/usr/local/opencti/node_modules/readable-stream/lib/_stream_writable.js:571:22)\n at IncomingMessage.onend (node:internal/streams/readable:705:10)\n at Object.onceWrapper (node:events:641:28)\n at IncomingMessage.emit (node:events:539:35)\n at endReadableNT (node:internal/streams/readable:1344:12)\n at processTicksAndRejections (node:internal/process/task_queues:82:21)","message":"The Access Key Id you provided does not exist in our records."},"name":"UnknownError"},"level":"error","message":"[OPENCTI] Platform initialization fail","timestamp":"2022-05-25T09:02:06.636Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[INIT] Platform initialization done","timestamp":"2022-05-25T09:02:06.637Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI] API ready on port 4000","timestamp":"2022-05-25T09:02:07.626Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Running Expiration manager","timestamp":"2022-05-25T09:02:07.626Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Running retention manager","timestamp":"2022-05-25T09:02:07.626Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Running task manager","timestamp":"2022-05-25T09:02:07.627Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Subscription manager not started (disabled by configuration)","timestamp":"2022-05-25T09:02:07.640Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[BRIDGE] ERROR:root:A TOKEN must be set","timestamp":"2022-05-25T09:02:07.900Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[BRIDGE] Traceback (most recent call last):","timestamp":"2022-05-25T09:02:07.901Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[BRIDGE] File \"/usr/local/opencti/src/python/local_importer.py\", line 31, in ","timestamp":"2022-05-25T09:02:07.901Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[BRIDGE] ).inject()","timestamp":"2022-05-25T09:02:07.902Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[BRIDGE] File \"/usr/local/opencti/src/python/local_importer.py\", line 15, in inject","timestamp":"2022-05-25T09:02:07.902Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[BRIDGE] opencti_api_client = OpenCTIApiClient(self.api_url, self.api_token)","timestamp":"2022-05-25T09:02:07.902Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[BRIDGE] File \"/home/azureuser/.local/lib/python3.10/site-packages/pycti/api/opencti_api_client.py\", line 114, in init","timestamp":"2022-05-25T09:02:07.902Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[BRIDGE] raise ValueError(\"A TOKEN must be set\")","timestamp":"2022-05-25T09:02:07.902Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[BRIDGE] ValueError: A TOKEN must be set","timestamp":"2022-05-25T09:02:07.902Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI] API ready on port 4000","timestamp":"2022-05-25T09:02:08.769Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Running Expiration manager","timestamp":"2022-05-25T09:02:08.769Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Running retention manager","timestamp":"2022-05-25T09:02:08.769Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Running task manager","timestamp":"2022-05-25T09:02:08.769Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Subscription manager not started (disabled by configuration)","timestamp":"2022-05-25T09:02:08.777Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[BRIDGE] ERROR:root:A TOKEN must be set","timestamp":"2022-05-25T09:02:09.016Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[BRIDGE] Traceback (most recent call last):","timestamp":"2022-05-25T09:02:09.017Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[BRIDGE] File \"/usr/local/opencti/src/python/local_importer.py\", line 31, in ","timestamp":"2022-05-25T09:02:09.017Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[BRIDGE] ).inject()","timestamp":"2022-05-25T09:02:09.017Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[BRIDGE] File \"/usr/local/opencti/src/python/local_importer.py\", line 15, in inject","timestamp":"2022-05-25T09:02:09.017Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[BRIDGE] opencti_api_client = OpenCTIApiClient(self.api_url, self.api_token)","timestamp":"2022-05-25T09:02:09.017Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[BRIDGE] File \"/home/azureuser/.local/lib/python3.10/site-packages/pycti/api/opencti_api_client.py\", line 114, in init","timestamp":"2022-05-25T09:02:09.017Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[BRIDGE] raise ValueError(\"A TOKEN must be set\")","timestamp":"2022-05-25T09:02:09.018Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[BRIDGE] ValueError: A TOKEN must be set","timestamp":"2022-05-25T09:02:09.018Z","version":"5.3.0"} FAIL tests/02-integration/00-inject/loader-test.js Database provision ✕ should platform init (17 ms) ✕ Should import creation succeed (1264 ms) ✕ Should import update succeed (1114 ms)

● Database provision › should platform init

expect(received).resolves.toBe()

Received promise rejected instead of resolved
Rejected to value: [Error: [OPENCTI] Platform initialization fail]

  21 |       expect(execution.status).toEqual('success');
  22 |       await shutdownModules();
> 23 |     },
     |            ^
  24 |     FIVE_MINUTES
  25 |   );
  26 |   // Python lib is fixed but we need to wait for a new release

  at expect (node_modules/expect/build/index.js:178:15)
  at Object.<anonymous> (tests/02-integration/00-inject/loader-test.js:23:12)

● Database provision › Should import creation succeed

ERROR:root:A TOKEN must be set

● Database provision › Should import update succeed

listen EADDRINUSE: address already in use :::4000

  at src/http/httpServer.js:2862:47

● Database provision › Should import update succeed

ERROR:root:A TOKEN must be set
richard-julien commented 2 years ago

Hi @quark95 . First remark, be careful to user YARN and not NPM to manage the platform. Now for your problem OpenCTI fail to communicate with minio. I think its because you dont use the same credential.

In minio

MINIO_ROOT_USER="miniouser"
MINIO_ROOT_PASSWORD="DobfhPf9Ek4zaV"

In OCTI

"access_key": "DobfhPf9Ek4zaV",
"secret_key": "TyMygr7287LkPR3TXbWvtQMhySo2cgd3"

Should be in OCTI

"access_key": "miniouser",
"secret_key": "DobfhPf9Ek4zaV"

Variables names are not perfectly aligned because minio change the naming recently.

quark95 commented 2 years ago

Thanks for your suggestion @richard-julien :-) I updated OCTI but still get the same error with yarn test.

sudo nano /usr/local/opencti/config/production.json

"minio": { "bucket_name": "opencti-bucket", "endpoint": "localhost", "port": 9000, "use_ssl": false, "access_key": "miniouser", "secret_key": "DobfhPf9Ek4zaV" },

richard-julien commented 2 years ago

Its because yarn test doesn't use the production profile. For testing (yarn test) test.json will be use. To start in production, you need to "yarn build" to build the production javascript, then "yarn serv" to start the platform with the production profile.

quark95 commented 2 years ago

Thanks again @richard-julien :-)

Unfortunately, I am still getting the same error.

yarn build Defaulting to user installation because normal site-packages is not writeable

sudo yarn build Collecting git+https://github.com/opencti-platform/client-python@master#pycti (from -r src/python/requirements.txt (line 3))

WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv ✔ Parse configuration ✔ Generate outputs

build/index.js 9.1mb ⚠️ build/script-clean-relations.js 2.4mb ⚠️ build/index.js.map 7.6mb build/script-clean-relations.js.map 2.3mb

yarn serv {"category":"APP","level":"info","message":"[OPENCTI] Starting platform","timestamp":"2022-05-25T22:34:12.588Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI] Checking dependencies statuses","timestamp":"2022-05-25T22:34:12.589Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[SEARCH ENGINE] Elasticsearch (7.17.4) client selected / runtime sorting enabled","timestamp":"2022-05-25T22:34:12.657Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[CHECK] Search engine is alive","timestamp":"2022-05-25T22:34:12.657Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[CHECK] Minio is alive","timestamp":"2022-05-25T22:34:12.664Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[CHECK] RabbitMQ is alive","timestamp":"2022-05-25T22:34:12.679Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[CHECK] Redis is alive","timestamp":"2022-05-25T22:34:12.684Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[CHECK] Python3 is available","timestamp":"2022-05-25T22:34:12.881Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[REDIS] Redis 'Pubsub subscriber' client ready","timestamp":"2022-05-25T22:34:12.936Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Cache manager initialized","timestamp":"2022-05-25T22:34:12.937Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[REDIS] Redis 'Client context' client ready","timestamp":"2022-05-25T22:34:12.939Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[INIT] Starting platform initialization","timestamp":"2022-05-25T22:34:12.940Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[INIT] New platform detected, initialization...","timestamp":"2022-05-25T22:34:12.945Z","version":"5.3.0"} {"category":"APP","error":{"_error":{},"_showLocations":false,"_showPath":false,"data":{"category":"technical","http_status":500,"reason":"[INIT] Fail initialize schema, index already exists, previous initialization fail because you kill the platform before the end of the initialization. Please remove your elastic/opensearch data and restart."},"internalData":{},"name":"ConfigurationError","time_thrown":"2022-05-25T22:34:12.951Z"},"level":"error","message":"[OPENCTI] Platform initialization fail","timestamp":"2022-05-25T22:34:12.951Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[INIT] Platform initialization done","timestamp":"2022-05-25T22:34:12.952Z","version":"5.3.0"} {"_error":{},"_showLocations":false,"_showPath":false,"category":"APP","data":{"category":"technical","http_status":500,"reason":"[INIT] Fail initialize schema, index already exists, previous initialization fail because you kill the platform before the end of the initialization. Please remove your elastic/opensearch data and restart."},"internalData":{},"level":"error","message":"A configuration error has occurred","name":"ConfigurationError","time_thrown":"2022-05-25T22:34:12.951Z","timestamp":"2022-05-25T22:34:12.952Z","version":"5.3.0"}

yarn test

{"category":"APP","error":{"data":{"_stack":"S3Error: The Access Key Id you provided does not exist in our records.\n at Object.parseError (/usr/local/opencti/node_modules/minio/dist/main/xml-parsers.js:71:11)\n at /usr/local/opencti/node_modules/minio/dist/main/transformers.js:166:22\n at DestroyableTransform._flush (/usr/local/opencti/node_modules/minio/dist/main/transformers.js:90:10)\n at DestroyableTransform.prefinish (/usr/local/opencti/node_modules/readable-stream/lib/_stream_transform.js:129:10)\n at DestroyableTransform.emit (node:events:527:28)\n at prefinish (/usr/local/opencti/node_modules/readable-stream/lib/_stream_writable.js:611:14)\n at finishMaybe (/usr/local/opencti/node_modules/readable-stream/lib/_stream_writable.js:620:5)\n at endWritable (/usr/local/opencti/node_modules/readable-stream/lib/_stream_writable.js:643:3)\n at DestroyableTransform.Object..Writable.end (/usr/local/opencti/node_modules/readable-stream/lib/_stream_writable.js:571:22)\n at IncomingMessage.onend (node:internal/streams/readable:705:10)\n at Object.onceWrapper (node:events:641:28)\n at IncomingMessage.emit (node:events:539:35)\n at endReadableNT (node:internal/streams/readable:1344:12)\n at processTicksAndRejections (node:internal/process/task_queues:82:21)","message":"The Access Key Id you provided does not exist in our records."},"name":"UnknownError"},"level":"error","message":"[OPENCTI] Platform initialization fail","timestamp":"2022-05-25T22:38:20.939Z","version":"5.3.0"}

FAIL tests/02-integration/00-inject/loader-test.js (9.257 s) Database provision ✕ should platform init (16 ms) ✕ Should import creation succeed (1241 ms) ✕ Should import update succeed (1103 ms)

● Database provision › should platform init

expect(received).resolves.toBe()

Received promise rejected instead of resolved
Rejected to value: [Error: [OPENCTI] Platform initialization fail]

  21 |       expect(execution.status).toEqual('success');
  22 |       await shutdownModules();
> 23 |     },
     |            ^
  24 |     FIVE_MINUTES
  25 |   );
  26 |   // Python lib is fixed but we need to wait for a new release

  at expect (node_modules/expect/build/index.js:178:15)
  at Object.<anonymous> (tests/02-integration/00-inject/loader-test.js:23:12)

● Database provision › Should import creation succeed

ERROR:root:A TOKEN must be set

● Database provision › Should import update succeed

listen EADDRINUSE: address already in use :::4000

  at src/http/httpServer.js:2862:47

● Database provision › Should import update succeed

ERROR:root:A TOKEN must be set
quark95 commented 2 years ago

This document describes the process I used to install OCTI.

OpenCTI Manual deployment instructions web.docx

richard-julien commented 2 years ago

Sometimes you have some good information in the log.

[INIT] Fail initialize schema, index already exists, previous initialization fail because you kill the platform before the end of the initialization. Please remove your elastic/opensearch data and restart.

Not sure what i can add to this :)

quark95 commented 2 years ago

Thanks heaps @richard-julien :-)

I am sorry to be such a pain, I have probably done something really stupid. I have been trying for weeks to get this running without success. When I am finished I will publish an installation manual to contribute to the community.

I deleted the ElasticSearch index and now it hangs on sudo yarn server.

I have also rebooted and restarted the ElasticSearch server.

If I run sudo yarn test it produces a different error now but it does not make sense. I have updated the environment vars.

{"category":"APP","error":{"_error":{},"_showLocations":false,"_showPath":false,"data":{"category":"technical","http_status":500,"reason":"You need to configure the environment vars"},"internalData":{},"name":"ConfigurationError","time_thrown":"2022-05-26T00:06:39.582Z"},"level":"error","message":"[OPENCTI] Platform initialization fail","timestamp":"2022-05-26T00:06:39.582Z","version":"5.3.0"}

The output is in the attached txt file.

shell_output_1.txt

richard-julien commented 2 years ago

Hi @quark95 , A bit difficult to help you as you mix different way of starting an different problems. For now forget about the yarn test command. Please build and use yarn serv along with a correctly configured production.json file. You must at least provide admin credentinal.

  "app": {
    "admin": {
      "email": "admin@opencti.io",
      "password": "Password you want",
      "token": "VALID UUID V4"
    }
  },
quark95 commented 2 years ago

Thanks @richard-julien, much appreciated.

sudo nano /usr/local/opencti/config/production.json

"admin": {
  "email": "admin@opencti.io",
  "password": "vtCpp2ZeD57Yj",
  "token": "1401bf5f-0289-48ca-9781-a386943cc680"
}

The token came from here https://www.uuidgenerator.net/ and is version 4.

Running yarn serv does not complete. It hangs for 10 minutes and I have to ctrl-c to end the process.

yarn serv {"category":"APP","level":"info","message":"[OPENCTI] Starting platform","timestamp":"2022-05-28T10:06:40.613Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI] Starting platform","timestamp":"2022-05-28T10:06:40.613Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI] Checking dependencies statuses","timestamp":"2022-05-28T10:06:40.615Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[SEARCH ENGINE] Elasticsearch (7.17.4) client selected / runtime sorting enabled","timestamp":"2022-05-28T10:06:40.633Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[CHECK] Search engine is alive","timestamp":"2022-05-28T10:06:40.633Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[CHECK] Minio is alive","timestamp":"2022-05-28T10:06:40.641Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[CHECK] RabbitMQ is alive","timestamp":"2022-05-28T10:06:40.657Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[CHECK] Redis is alive","timestamp":"2022-05-28T10:06:40.663Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[CHECK] Python3 is available","timestamp":"2022-05-28T10:06:40.847Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[REDIS] Redis 'Pubsub subscriber' client ready","timestamp":"2022-05-28T10:06:40.873Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Cache manager initialized","timestamp":"2022-05-28T10:06:40.873Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[REDIS] Redis 'Client context' client ready","timestamp":"2022-05-28T10:06:40.876Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[INIT] Starting platform initialization","timestamp":"2022-05-28T10:06:40.877Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[INIT] Existing platform detected, initialization...","timestamp":"2022-05-28T10:06:40.882Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[INIT] admin user initialized","timestamp":"2022-05-28T10:06:41.012Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[MIGRATION] Read 0 migrations from the database","timestamp":"2022-05-28T10:06:41.033Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[MIGRATION] Platform already up to date, nothing to migrate","timestamp":"2022-05-28T10:06:41.033Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[MIGRATION] Migration process completed","timestamp":"2022-05-28T10:06:41.034Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[MIGRATION] Platform version updated to 5.3.0","timestamp":"2022-05-28T10:06:41.047Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[INIT] Platform initialization done","timestamp":"2022-05-28T10:06:41.048Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI] API ready on port 4000","timestamp":"2022-05-28T10:06:41.577Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Running Expiration manager","timestamp":"2022-05-28T10:06:41.578Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Running retention manager","timestamp":"2022-05-28T10:06:41.578Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Running task manager","timestamp":"2022-05-28T10:06:41.578Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Subscription manager not started (disabled by configuration)","timestamp":"2022-05-28T10:06:41.585Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Running rule manager","timestamp":"2022-05-28T10:06:51.586Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[STREAM] Starting stream processor for Rule manager","timestamp":"2022-05-28T10:06:51.587Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[REDIS] Redis 'Rule manager' client ready","timestamp":"2022-05-28T10:06:51.588Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Running history manager","timestamp":"2022-05-28T10:06:51.592Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[STREAM] Starting stream processor for History manager","timestamp":"2022-05-28T10:06:51.592Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[REDIS] Redis 'History manager' client ready","timestamp":"2022-05-28T10:06:51.594Z","version":"5.3.0"} ^C {"category":"APP","level":"info","message":"[OPENCTI] SIGINT signal received, stopping OpenCTI","timestamp":"2022-05-28T10:08:13.839Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI] Shutdown SIGINT signal received, stopping OpenCTI","timestamp":"2022-05-28T10:08:13.840Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI] API stopped in 0 ms","timestamp":"2022-05-28T10:08:13.841Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Expiration manager stopped in 0 ms","timestamp":"2022-05-28T10:08:13.841Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Retention manager stopped in 0 ms","timestamp":"2022-05-28T10:08:13.842Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Task manager stopped in 0 ms","timestamp":"2022-05-28T10:08:13.842Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[STREAM] Shutdown stream processor for Rule manager","timestamp":"2022-05-28T10:08:15.620Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[STREAM] Stream processor current promise terminated","timestamp":"2022-05-28T10:08:15.867Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Rule engine stopped in 2025 ms","timestamp":"2022-05-28T10:08:15.867Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[REDIS] Redis 'Rule manager' client closed","timestamp":"2022-05-28T10:08:15.868Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Sync manager stopped in 1917 ms","timestamp":"2022-05-28T10:08:17.784Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[STREAM] Shutdown stream processor for History manager","timestamp":"2022-05-28T10:08:19.622Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[STREAM] Stream processor current promise terminated","timestamp":"2022-05-28T10:08:20.074Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] History manager stopped in 2290 ms","timestamp":"2022-05-28T10:08:20.075Z","version":"5.3.0"} {"category":"APP","level":"info","message":"[OPENCTI] All modules have been stopped, exiting process","timestamp":"2022-05-28T10:08:20.075Z","version":"5.3.0"}

richard-julien commented 2 years ago

'yarn serv' is not a background command so its normal that you are blocked into the execution.

quark95 commented 2 years ago

Thanks, @richard-julien :-) you have been enormously helpful.

quark95 commented 2 years ago

Still not working :-( Probably user error...

Summary of all failing tests FAIL tests/02-integration/00-inject/loader-test.js ● Database provision › should platform init

expect(received).resolves.toBe()

Received promise rejected instead of resolved
Rejected to value: [Error: [OPENCTI] Platform initialization fail]

  21 |       expect(execution.status).toEqual('success');
  22 |       await shutdownModules();
> 23 |     },
     |            ^
  24 |     FIVE_MINUTES
  25 |   );
  26 |   // Python lib is fixed but we need to wait for a new release

  at expect (node_modules/expect/build/index.js:178:15)
  at Object.<anonymous> (tests/02-integration/00-inject/loader-test.js:23:12)

● Database provision › Should import creation succeed

ERROR:root:A TOKEN must be set

● Database provision › Should import update succeed

listen EADDRINUSE: address already in use :::4000

  at src/http/httpServer.js:2862:47

● Database provision › Should import update succeed

ERROR:root:A TOKEN must be set

Test Suites: 1 failed, 6 passed, 7 of 56 total Tests: 3 failed, 20 passed, 23 total Snapshots: 0 total Time: 10.223 s Ran all test suites matching /test/i.

quark95 commented 2 years ago

azureuser@vm-opencti:/usr/local/opencti/worker$ python3 worker.py & [1] 1821 azureuser@vm-opencti:/usr/local/opencti/worker$ INFO:root:Listing Threat-Actors with filters null. Traceback (most recent call last): File "/usr/local/opencti/worker/worker.py", line 399, in worker = Worker() File "", line 6, in init File "/usr/local/opencti/worker/worker.py", line 299, in __post_init self.api = OpenCTIApiClient( File "/home/azureuser/.local/lib/python3.10/site-packages/pycti/api/opencti_api_client.py", line 179, in init__ raise ValueError( ValueError: OpenCTI API is not reachable. Waiting for OpenCTI API to start or check your configuration...

[1]+ Killed python3 worker.py

faustus25 commented 1 year ago

Can you share the production json section for accessing OpenCTI: app": { "port": 443, "base_path": "", "base_url": "https://opencti.domain.com/", "enabled": true, "https_cert": { "ca": [], "key": "/etc/ssl/private/xxx..key", "crt": "/etc/ssl/private/xxx..crt", "reject_unauthorized": true, "cookie_secure": false },

Also did you allow firewall rules on the local server to access OpenCTI:

sudo ufw status Status: active 80/tcp ALLOW Anywhere 443/tcp ALLOW Anywhere

spicy-bear commented 1 year ago

Fresh install on 2 different systems I'm getting an error on yarn install from the opencti folder: opencti$ yarn install 00h00m00s 0/0: : ERROR: [Errno 2] No Such File or directory: 'install'

and getting the same A TOKEN must be set on running python3 worker.py