search

OpenCTI-Platform / opencti

Open Cyber Threat Intelligence Platform
https://opencti.io
Other
6.31k stars 932 forks source link

ModuleNotFoundError: No module named 'stix2' #2187

Closed faustus25 closed 2 years ago

faustus25 commented 2 years ago

Description

Manual installation of OpenCTI fails when running 'yarn serv'

Environment

  1. OS , Ubuntu 20.04
  2. OpenCTI version 5.3.6
  3. OpenCTI client: python
  4. Other environment details:

Reproducible Steps

Steps to create the smallest reproducible scenario:

  1. Run: yarn serv
  2. Error: yarn run v1.22.19 $ node build/index.js {"category":"APP","level":"info","message":"[OPENCTI] Starting platform","timestamp":"2022-06-24T19:47:38.485Z","version":"5.3.6"} {"category":"APP","level":"info","message":"[OPENCTI] Checking dependencies statuses","timestamp":"2022-06-24T19:47:38.487Z","version":"5.3.6"} {"category":"APP","level":"info","message":"[SEARCH ENGINE] Elasticsearch (8.2.3) client selected / runtime sorting enabled","timestamp":"2022-06-24T19:47:38.513Z","version":"5.3.6"} {"category":"APP","level":"info","message":"[CHECK] Search engine is alive","timestamp":"2022-06-24T19:47:38.514Z","version":"5.3.6"} {"category":"APP","level":"info","message":"[CHECK] Minio is alive","timestamp":"2022-06-24T19:47:38.529Z","version":"5.3.6"} {"category":"APP","level":"info","message":"[CHECK] RabbitMQ is alive","timestamp":"2022-06-24T19:47:38.551Z","version":"5.3.6"} {"category":"APP","level":"info","message":"[CHECK] Redis is alive","timestamp":"2022-06-24T19:47:38.558Z","version":"5.3.6"} {"category":"APP","level":"info","message":"[BRIDGE] Traceback (most recent call last):","timestamp":"2022-06-24T19:47:38.782Z","version":"5.3.6"} {"category":"APP","level":"info","message":"[BRIDGE] File \"src/python/stix2_create_pattern.py\", line 3, in ","timestamp":"2022-06-24T19:47:38.783Z","version":"5.3.6"} {"category":"APP","level":"info","message":"[BRIDGE] from stix2 import (","timestamp":"2022-06-24T19:47:38.783Z","version":"5.3.6"} {"category":"APP","level":"info","message":"[BRIDGE] ModuleNotFoundError: No module named 'stix2'","timestamp":"2022-06-24T19:47:38.784Z","version":"5.3.6"} {"category":"APP","error":{"context":{},"message":"ModuleNotFoundError: No module named 'stix2'","name":"Error","stack":"Error: ModuleNotFoundError: No module named 'stix2'\n at d2.parseError (/opt/opencti/build/node_modules/python-shell/index.ts:366:21)\n at terminateIfNeeded (/opt/opencti/build/node_modules/python-shell/index.ts:235:32)\n at ChildProcess. (/opt/opencti/build/node_modules/python-shell/index.ts:226:13)\n at ChildProcess.emit (node:events:527:28)\n at Process.ChildProcess._handle.onexit (node:internal/child_process:291:12)\n ----- Python Traceback -----\n File \"src/python/stix2_create_pattern.py\", line 3, in \n from stix2 import ("},"level":"error","message":"[OPENCTI] Platform start fail","timestamp":"2022-06-24T19:47:38.815Z","version":"5.3.6"} error Command failed with exit code 1.

Expected Output

{"category":"APP","level":"info","message":"[OPENCTI] Starting platform","timestamp":"2022-06-22T11:19:39.160Z","version":"5.3.3"} {"category":"APP","level":"info","message":"[OPENCTI] Checking dependencies statuses","timestamp":"2022-06-22T11:19:39.164Z","version":"5.3.3"} {"category":"APP","level":"info","message":"[SEARCH ENGINE] Elasticsearch (7.16.2) client selected / runtime sorting enabled","timestamp":"2022-06-22T11:19:39.293Z","version":"5.3.3"} {"category":"APP","level":"info","message":"[CHECK] Search engine is alive","timestamp":"2022-06-22T11:19:39.294Z","version":"5.3.3"} {"category":"APP","level":"info","message":"[CHECK] Minio is alive","timestamp":"2022-06-22T11:19:40.717Z","version":"5.3.3"} {"category":"APP","level":"info","message":"[CHECK] RabbitMQ is alive","timestamp":"2022-06-22T11:19:40.775Z","version":"5.3.3"} {"category":"APP","level":"info","message":"[CHECK] Redis is alive","timestamp":"2022-06-22T11:19:40.810Z","version":"5.3.3"} {"category":"APP","level":"info","message":"[CHECK] SMTP is alive","timestamp":"2022-06-22T11:19:41.076Z","version":"5.3.3"} {"category":"APP","level":"info","message":"[CHECK] Python3 is available","timestamp":"2022-06-22T11:19:43.239Z","version":"5.3.3"} {"category":"APP","level":"info","message":"[REDIS] Redis 'Pubsub subscriber' client ready","timestamp":"2022-06-22T11:19:46.882Z","version":"5.3.3"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Cache manager initialized","timestamp":"2022-06-22T11:19:46.917Z","version":"5.3.3"} {"category":"APP","level":"info","message":"[REDIS] Redis 'Client context' client ready","timestamp":"2022-06-22T11:19:46.960Z","version":"5.3.3"} {"category":"APP","level":"info","message":"[INIT] Starting platform initialization","timestamp":"2022-06-22T11:19:46.979Z","version":"5.3.3"} {"category":"APP","level":"info","message":"[INIT] Existing platform detected, initialization...","timestamp":"2022-06-22T11:19:47.040Z","version":"5.3.3"} {"category":"APP","level":"info","message":"[INIT] admin user initialized","timestamp":"2022-06-22T11:19:51.884Z","version":"5.3.3"} {"category":"APP","level":"info","message":"[MIGRATION] Read 11 migrations from the database","timestamp":"2022-06-22T11:19:54.258Z","version":"5.3.3"} {"category":"APP","level":"info","message":"[MIGRATION] Platform already up to date, nothing to migrate","timestamp":"2022-06-22T11:19:54.260Z","version":"5.3.3"} {"category":"APP","level":"info","message":"[MIGRATION] Migration process completed","timestamp":"2022-06-22T11:19:54.261Z","version":"5.3.3"} {"category":"APP","level":"info","message":"[MIGRATION] Platform version updated to 5.3.3","timestamp":"2022-06-22T11:19:54.754Z","version":"5.3.3"} {"category":"APP","level":"info","message":"[INIT] Platform initialization done","timestamp":"2022-06-22T11:19:54.755Z","version":"5.3.3"} {"category":"APP","level":"info","message":"Adding prometheus middleware (for metrics) on path: /prometheus/metrics","timestamp":"2022-06-22T11:19:55.733Z","version":"5.3.3"} {"category":"APP","level":"info","message":"[OPENCTI] API ready on port 443","timestamp":"2022-06-22T11:19:55.788Z","version":"5.3.3"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Running Expiration manager","timestamp":"2022-06-22T11:19:55.788Z","version":"5.3.3"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Running retention manager","timestamp":"2022-06-22T11:19:55.789Z","version":"5.3.3"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Running task manager","timestamp":"2022-06-22T11:19:55.789Z","version":"5.3.3"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Running subscription manager","timestamp":"2022-06-22T11:19:55.855Z","version":"5.3.3"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Running rule manager","timestamp":"2022-06-22T11:20:05.857Z","version":"5.3.3"} {"category":"APP","level":"info","message":"[STREAM] Starting stream processor for Rule manager","timestamp":"2022-06-22T11:20:05.861Z","version":"5.3.3"} {"category":"APP","level":"info","message":"[REDIS] Redis 'Rule manager' client ready","timestamp":"2022-06-22T11:20:05.891Z","version":"5.3.3"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Running history manager","timestamp":"2022-06-22T11:20:06.909Z","version":"5.3.3"} {"category":"APP","level":"info","message":"[STREAM] Starting stream processor for History manager","timestamp":"2022-06-22T11:20:06.910Z","version":"5.3.3"} {"category":"APP","level":"info","message":"[REDIS] Redis 'History manager' client ready","timestamp":"2022-06-22T11:20:06.912Z","version":"5.3.3"}

Actual Output

ModuleNotFoundError: No module named 'stix2'

Additional information

Installed python 3.10.5 and pip stix package installed: /opt/opencti$ python3.10 -m pip list Package Version

stix2 3.0.1

What an install of stix2 looks like: python3.10 -m pip install stix2 Defaulting to user installation because normal site-packages is not writeable Requirement already satisfied: stix2 in /home/user/.local/lib/python3.10/site-packages (3.0.1) Requirement already satisfied: requests in /home/user/.local/lib/python3.10/site-packages (from stix2) (2.27.1) Requirement already satisfied: pytz in /home/user/.local/lib/python3.10/site-packages (from stix2) (2021.1) Requirement already satisfied: stix2-patterns>=1.2.0 in /home/user/.local/lib/python3.10/site-packages (from stix2) (1.3.2) Requirement already satisfied: simplejson in /home/user/.local/lib/python3.10/site-packages (from stix2) (3.17.5) Requirement already satisfied: six in /home/user/.local/lib/python3.10/site-packages (from stix2-patterns>=1.2.0->stix2) (1.16.0) Requirement already satisfied: antlr4-python3-runtime~=4.8.0 in /home/user/.local/lib/python3.10/site-packages (from stix2-patterns>=1.2.0->stix2) (4.8) Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/user/.local/lib/python3.10/site-packages (from requests->stix2) (1.26.5) Requirement already satisfied: charset-normalizer~=2.0.0 in /home/user/.local/lib/python3.10/site-packages (from requests->stix2) (2.0.12) Requirement already satisfied: certifi>=2017.4.17 in /home/user/.local/lib/python3.10/site-packages (from requests->stix2) (2020.6.20) Requirement already satisfied: idna<4,>=2.5 in /home/user/.local/lib/python3.10/site-packages (from requests->stix2) (2.10)

faustus25 commented 2 years ago

Specified the directory to install stix2 and added a Python Path to ~/.bashrc export PYTHONPATH="/home/user/.local/lib/python3.10"

pip3.10 install stix2 --target=/home/user/.local/lib/python3.10/site-packages --upgrade Collecting stix2 Using cached stix2-3.0.1-py2.py3-none-any.whl (177 kB) Collecting simplejson Using cached simplejson-3.17.6-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (137 kB) Collecting stix2-patterns>=1.2.0 Using cached stix2_patterns-2.0.0-py2.py3-none-any.whl (65 kB) Collecting pytz Using cached pytz-2022.1-py2.py3-none-any.whl (503 kB) Collecting requests Using cached requests-2.28.0-py3-none-any.whl (62 kB) Collecting antlr4-python3-runtime~=4.9.0 Using cached antlr4-python3-runtime-4.9.3.tar.gz (117 kB) Preparing metadata (setup.py) ... done Collecting six Using cached six-1.16.0-py2.py3-none-any.whl (11 kB) Collecting idna<4,>=2.5 Using cached idna-3.3-py3-none-any.whl (61 kB) Collecting charset-normalizer~=2.0.0 Using cached charset_normalizer-2.0.12-py3-none-any.whl (39 kB) Collecting urllib3<1.27,>=1.21.1 Using cached urllib3-1.26.9-py2.py3-none-any.whl (138 kB) Collecting certifi>=2017.4.17 Using cached certifi-2022.6.15-py3-none-any.whl (160 kB) Using legacy 'setup.py install' for antlr4-python3-runtime, since package 'wheel' is not installed. Installing collected packages: pytz, antlr4-python3-runtime, urllib3, six, simplejson, idna, charset-normalizer, certifi, stix2-patterns, requests, stix2 Running setup.py install for antlr4-python3-runtime ... done Successfully installed antlr4-python3-runtime-4.9.3 certifi-2022.6.15 charset-normalizer-2.0.12 idna-3.3 pytz-2022.1 requests-2.28.0 simplejson-3.17.6 six-1.16.0 stix2-3.0.1 stix2-patterns-2.0.0 urllib3-1.26.9

Installed wheels package: pip3.10 install wheel --target=/home/user/.local/lib/python3.10/site-packages --upgrade Collecting wheel Downloading wheel-0.37.1-py2.py3-none-any.whl (35 kB) Installing collected packages: wheel Successfully installed wheel-0.37.1

The error remains about OpenCTI not detecting installed stix2 package:

{"category":"APP","level":"info","message":"[CHECK] Redis is alive","timestamp":"2022-06-27T06:16:39.563Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[BRIDGE] Traceback (most recent call last):","timestamp":"2022-06-27T09:16:39.773Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[BRIDGE] File \"src/python/stix2_create_pattern.py\", line 3, in <module>","timestamp":"2022-06-27T09:16:39.774Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[BRIDGE] from stix2 import (","timestamp":"2022-06-27T09:16:39.774Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[BRIDGE] ModuleNotFoundError: No module named 'stix2'","timestamp":"2022-06-27T09:16:39.775Z","version":"5.3.7"} {"category":"APP","error":{"context":{},"message":"ModuleNotFoundError: No module named 'stix2'","name":"Error","stack":"Error: ModuleNotFoundError: No module named 'stix2'\n at av.parseError (/sdb1/opt/opencti/build/node_modules/python-shell/index.ts:366:21)\n at terminateIfNeeded (/sdb1/opt/opencti/build/node_modules/python-shell/index.ts:235:32)\n at ChildProcess.<anonymous> (/sdb1/opt/opencti/build/node_modules/python-shell/index.ts:226:13)\n at ChildProcess.emit (node:events:527:28)\n at Process.ChildProcess._handle.onexit (node:internal/child_process:291:12)\n ----- Python Traceback -----\n File \"src/python/stix2_create_pattern.py\", line 3, in <module>\n from stix2 import ("},"level":"error","message":"[OPENCTI] Platform start fail","timestamp":"2022-06-27T06:16:39.804Z","version":"5.3.7"}

Any suggestions to get OpenCTI to look for stix2 in the appropriate directory?

faustus25 commented 2 years ago

Solved this by setting the Python environment:

/opt/opencti$ sudo update-alternatives --install /usr/bin/python3 python /usr/bin/python3.8 1 update-alternatives: using /usr/bin/python3.8 to provide /usr/bin/python3 (python) in auto mode opt/opencti$ sudo update-alternatives --install /usr/bin/python3 python /usr/bin/python3.10 2 update-alternatives: using /usr/bin/python3.10 to provide /usr/bin/python3 (python) in auto mode opt/opencti$ sudo update-alternatives --config python There are 2 choices for the alternative python (providing /usr/bin/python3).

Selection Path Priority Status

Press to keep the current choice[*], or type selection number:

If running yarn serv as a regular user you may get this error then:

Give Safe User Permission To Use Port 80:

/opt/opencti$ sudo apt-get install libcap2-bin /opt/opencti$ sudo setcap cap_net_bind_service=+ep readlink -f \which node``

OpenCTI should run normally:

/opt/opencti$ yarn serv Persisted queries are enabled and are using an unbounded cache. Your server is vulnerable to denial of service attacks via memory exhaustion. Set cache: "bounded" or persistedQueries: false in your ApolloServer constructor, or see https://go.apollo.dev/s/cache-backends for other alternatives. {"category":"APP","level":"info","message":"[OPENCTI] Starting platform","timestamp":"2022-06-27T21:35:53.753Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[OPENCTI] Checking dependencies statuses","timestamp":"2022-06-27T21:35:53.756Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[SEARCH ENGINE] Elasticsearch (8.2.3) client selected / runtime sorting enabled","timestamp":"2022-06-27T21:35:53.783Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[CHECK] Search engine is alive","timestamp":"2022-06-27T21:35:53.784Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[CHECK] Minio is alive","timestamp":"2022-06-27T21:35:53.797Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[CHECK] RabbitMQ is alive","timestamp":"2022-06-27T21:35:53.821Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[CHECK] Redis is alive","timestamp":"2022-06-27T21:35:53.829Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[CHECK] Python3 is available","timestamp":"2022-06-27T21:35:54.179Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Initializing cache manager","timestamp":"2022-06-27T21:35:54.180Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[REDIS] Redis 'Pubsub subscriber' client ready","timestamp":"2022-06-27T21:35:54.238Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Cache manager initialized","timestamp":"2022-06-27T21:35:54.240Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[REDIS] Redis 'Client context' client ready","timestamp":"2022-06-27T21:35:54.244Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[INIT] Starting platform initialization","timestamp":"2022-06-27T21:35:54.246Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[INIT] Existing platform detected, initialization...","timestamp":"2022-06-27T21:35:54.254Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[INIT] admin user initialized","timestamp":"2022-06-27T21:35:54.469Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[MIGRATION] Read 0 migrations from the database","timestamp":"2022-06-27T21:35:54.491Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[MIGRATION] Platform already up to date, nothing to migrate","timestamp":"2022-06-27T21:35:54.492Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[MIGRATION] Migration process completed","timestamp":"2022-06-27T21:35:54.493Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[MIGRATION] Platform version updated to 5.3.7","timestamp":"2022-06-27T21:35:54.508Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[INIT] Platform initialization done","timestamp":"2022-06-27T21:35:54.509Z","version":"5.3.7"} Persisted queries are enabled and are using an unbounded cache. Your server is vulnerable to denial of service attacks via memory exhaustion. Set cache: "bounded" or persistedQueries: false in your ApolloServer constructor, or see https://go.apollo.dev/s/cache-backends for other alternatives. {"category":"APP","level":"info","message":"[OPENCTI] API ready on port 80","timestamp":"2022-06-27T21:35:55.282Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Running Expiration manager","timestamp":"2022-06-27T21:35:55.282Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Running retention manager","timestamp":"2022-06-27T21:35:55.283Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Running task manager","timestamp":"2022-06-27T21:35:55.283Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Subscription manager not started (disabled by configuration)","timestamp":"2022-06-27T21:35:55.297Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Running rule manager from 1656365189752-0","timestamp":"2022-06-27T21:36:05.299Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[STREAM] Starting stream processor for Rule manager","timestamp":"2022-06-27T21:36:05.300Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[REDIS] Redis 'Rule manager' client ready","timestamp":"2022-06-27T21:36:05.303Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[OPENCTI-MODULE] Running history manager","timestamp":"2022-06-27T21:36:05.310Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[STREAM] Starting stream processor for History manager","timestamp":"2022-06-27T21:36:05.311Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[REDIS] Redis 'History manager' client ready","timestamp":"2022-06-27T21:36:05.313Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[REDIS] Redis 'Client base' client ready","timestamp":"2022-06-27T21:36:40.147Z","version":"5.3.7"} {"category":"APP","level":"info","message":"[REDIS] Redis 'Pubsub publisher' client ready","timestamp":"2022-06-27T21:37:09.142Z","version":"5.3.7"}