search

OpenCTI-Platform / opencti

Open Cyber Threat Intelligence Platform
https://opencti.io
Other
6.31k stars 932 forks source link

GraphQL initialization fail > Error: Python3 with STIX2 module is missing #545

Closed oz17il closed 4 years ago

oz17il commented 4 years ago

Hi,

With the latest version of OpenCTI (3.0.2), I'm facing the following issue after all requirements were up:

GraphQL initialization fail > Error: Python3 with STIX2 module is missing

Environment Manuale Installation OS (where OpenCTI server runs): DUbuntu 18.0.4 OpenCTI version: OpenCTI 3.0.2 Installation guide: https://opencti-platform.github.io/docs/installation/manual

root@ip-172-1-1-108:/opt/opencti# yarn serv yarn run v1.22.0 $ node --max_old_space_size=8192 build/index.js info: 🚀 OpenCTI started in production mode with embedded file info: [PRE-CHECK] > Grakn is alive info: [PRE-CHECK] > Elasticsearch is alive info: [PRE-CHECK] > Minio is alive error: [OPENCTI] GraphQL initialization fail > Error: Python3 with STIX2 module is missing at s (/opt/opencti/build/index.js:1:72995) at processTicksAndRejections (internal/process/task_queues.js:97:5) error Command failed with exit code 1. info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

SamuelHassine commented 4 years ago

@oz17il : can you try to execute the following: python3 utils/stix2/stix2_extract_observables.py check?

oz17il commented 4 years ago

@SamuelHassine

root@ip-172-1-1-108:/opt/opencti# python3 src/utils/stix2/stix2_extract_observables.py Traceback (most recent call last): File "src/utils/stix2/stix2_extract_observables.py", line 3, in import stix2 File "/usr/local/lib/python3.6/dist-packages/stix2/init.py", line 28, in from .datastore.filesystem import ( File "/usr/local/lib/python3.6/dist-packages/stix2/datastore/filesystem.py", line 11, in from stix2 import v20, v21 File "/usr/local/lib/python3.6/dist-packages/stix2/v20/init.py", line 34, in from .sdo import ( File "/usr/local/lib/python3.6/dist-packages/stix2/v20/sdo.py", line 6, in from stix2patterns.validator import run_validator File "/usr/local/lib/python3.6/dist-packages/stix2patterns/validator.py", line 9, in from antlr4 import InputStream File "/usr/local/lib/python3.6/dist-packages/antlr4/init.py", line 5, in from antlr4.BufferedTokenStream import TokenStream File "/usr/local/lib/python3.6/dist-packages/antlr4/BufferedTokenStream.py", line 18, in from antlr4.error.Errors import IllegalStateException File "/usr/local/lib/python3.6/dist-packages/antlr4/error/Errors.py", line 5, in from antlr4.atn.Transition import PredicateTransition File "/usr/local/lib/python3.6/dist-packages/antlr4/atn/Transition.py", line 19, in from builtin import unicode ModuleNotFoundError: No module named 'builtin'

MathRig commented 4 years ago

Hi, same issue here, manual installation too with the same guide :

OS: CentOS 7 Python 3.6.8 OpenCTI 3.0.3

when doing the requirement install, all is good:

pip3.6 install -r requirements.txt Requirement already satisfied: antlr4-python2-runtime==4.7.1 in /usr/local/lib/python3.6/site-packages (from -r requirements.txt (line 1)) Requirement already satisfied: stix2==1.3.1 in /usr/local/lib/python3.6/site-packages (from -r requirements.txt (line 2)) Requirement already satisfied: stix2-validator==1.1.2 in /usr/local/lib/python3.6/site-packages (from -r requirements.txt (line 3)) Requirement already satisfied: pycti==3.0.3 in /usr/local/lib/python3.6/site-packages (from -r requirements.txt (line 4)) Requirement already satisfied: pytz in /usr/local/lib/python3.6/site-packages (from stix2==1.3.1->-r requirements.txt (line 2)) Requirement already satisfied: stix2-patterns in /usr/local/lib/python3.6/site-packages (from stix2==1.3.1->-r requirements.txt (line 2)) Requirement already satisfied: requests in /usr/local/lib/python3.6/site-packages (from stix2==1.3.1->-r requirements.txt (line 2)) Requirement already satisfied: six in /usr/local/lib/python3.6/site-packages (from stix2==1.3.1->-r requirements.txt (line 2)) Requirement already satisfied: simplejson in /usr/local/lib/python3.6/site-packages (from stix2==1.3.1->-r requirements.txt (line 2)) Requirement already satisfied: python-dateutil in /usr/local/lib/python3.6/site-packages (from stix2==1.3.1->-r requirements.txt (line 2)) Requirement already satisfied: requests-cache in /usr/local/lib/python3.6/site-packages (from stix2-validator==1.1.2->-r requirements.txt (line 3)) Requirement already satisfied: colorama in /usr/local/lib/python3.6/site-packages (from stix2-validator==1.1.2->-r requirements.txt (line 3)) Requirement already satisfied: jsonschema>=2.5.1 in /usr/local/lib/python3.6/site-packages (from stix2-validator==1.1.2->-r requirements.txt (line 3)) Requirement already satisfied: appdirs in /usr/local/lib/python3.6/site-packages (from stix2-validator==1.1.2->-r requirements.txt (line 3)) Requirement already satisfied: PyYAML in /usr/local/lib64/python3.6/site-packages (from pycti==3.0.3->-r requirements.txt (line 4)) Requirement already satisfied: datefinder in /usr/local/lib/python3.6/site-packages (from pycti==3.0.3->-r requirements.txt (line 4)) Requirement already satisfied: pika in /usr/local/lib/python3.6/site-packages (from pycti==3.0.3->-r requirements.txt (line 4)) Requirement already satisfied: deprecated in /usr/local/lib64/python3.6/site-packages (from pycti==3.0.3->-r requirements.txt (line 4)) Requirement already satisfied: python-magic in /usr/local/lib/python3.6/site-packages (from pycti==3.0.3->-r requirements.txt (line 4)) Requirement already satisfied: antlr4-python3-runtime~=4.8.0; python_version >= "3" in /usr/local/lib/python3.6/site-packages (from stix2-patterns->stix2==1.3.1->-r requirements.txt (line 2)) Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/site-packages (from requests->stix2==1.3.1->-r requirements.txt (line 2)) Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/site-packages (from requests->stix2==1.3.1->-r requirements.txt (line 2)) Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/site-packages (from requests->stix2==1.3.1->-r requirements.txt (line 2)) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/site-packages (from requests->stix2==1.3.1->-r requirements.txt (line 2)) Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.6/site-packages (from jsonschema>=2.5.1->stix2-validator==1.1.2->-r requirements.txt (line 3)) Requirement already satisfied: setuptools in /usr/lib/python3.6/site-packages (from jsonschema>=2.5.1->stix2-validator==1.1.2->-r requirements.txt (line 3)) Requirement already satisfied: pyrsistent>=0.14.0 in /usr/local/lib64/python3.6/site-packages (from jsonschema>=2.5.1->stix2-validator==1.1.2->-r requirements.txt (line 3)) Requirement already satisfied: importlib-metadata; python_version < "3.8" in /usr/local/lib/python3.6/site-packages (from jsonschema>=2.5.1->stix2-validator==1.1.2->-r requirements.txt (line 3)) Requirement already satisfied: regex>=2017.02.08 in /usr/local/lib64/python3.6/site-packages (from datefinder->pycti==3.0.3->-r requirements.txt (line 4)) Requirement already satisfied: wrapt<2,>=1.10 in /usr/local/lib/python3.6/site-packages (from deprecated->pycti==3.0.3->-r requirements.txt (line 4)) Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/site-packages (from importlib-metadata; python_version < "3.8"->jsonschema>=2.5.1->stix2-validator==1.1.2->-r requirements.txt (line 3))

but the command python stix2_extract_observable.py gives me: Traceback (most recent call last): File "stix2_extract_observables.py", line 3, in import stix2 File "/usr/local/lib/python3.6/site-packages/stix2/init.py", line 28, in from .datastore.filesystem import ( File "/usr/local/lib/python3.6/site-packages/stix2/datastore/filesystem.py", line 11, in from stix2 import v20, v21 File "/usr/local/lib/python3.6/site-packages/stix2/v20/init.py", line 34, in from .sdo import ( File "/usr/local/lib/python3.6/site-packages/stix2/v20/sdo.py", line 6, in from stix2patterns.validator import run_validator File "/usr/local/lib/python3.6/site-packages/stix2patterns/validator.py", line 9, in from antlr4 import InputStream File "/usr/local/lib/python3.6/site-packages/antlr4/init.py", line 5, in from antlr4.BufferedTokenStream import TokenStream File "/usr/local/lib/python3.6/site-packages/antlr4/BufferedTokenStream.py", line 18, in from antlr4.error.Errors import IllegalStateException File "/usr/local/lib/python3.6/site-packages/antlr4/error/Errors.py", line 5, in from antlr4.atn.Transition import PredicateTransition File "/usr/local/lib/python3.6/site-packages/antlr4/atn/Transition.py", line 19, in from builtin import unicode ModuleNotFoundError: No module named 'builtin'

and yarn serv gives : yarn run v1.22.4 $ node --max_old_space_size=8192 build/index.js info: 🚀 OpenCTI started in production mode with embedded file info: [PRE-CHECK] > Grakn is alive info: [PRE-CHECK] > Elasticsearch is alive info: [PRE-CHECK] > Minio is alive error: [OPENCTI] GraphQL initialization fail > Error: Python3 with STIX2 module is missing at s (/opt/opencti/opencti/build/index.js:1:75102) at processTicksAndRejections (internal/process/task_queues.js:97:5) error Command failed with exit code 1. info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Any ideas ?

MathRig commented 4 years ago

Some Update.

I downloaded the 4.7.2 version of antlr4 from https://pypi.org/project/antlr4-python3-runtime/4.7.2/#files and replaced the python package in /usr/local/lib/python3.6/site-packages/antlr4 with the downloaded one (/opt/antlr4-python3-runtime-4.7.2/src/antlr4).

Now when I run python stix2_extract_observables.py, it seems to work fine: {"status": "error", "message": "Missing argument to the Python script"}

But now when I restart the launch of OpenCTI, I have an issue with Grakn...

$ yarn serv
yarn run v1.22.4 $ node --max_old_space_size=8192 build/index.js info: 🚀 OpenCTI started in production mode with embedded file error: [GRAKN] Seems down error: [OPENCTI] GraphQL initialization fail > Error: Grakn seems down at q (/opt/opencti/opencti/build/index.js:1:5536) at processTicksAndRejections (internal/process/task_queues.js:97:5) error Command failed with exit code 1. info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

I don't understand because nothing changed except the new version of antlr4. Grakn server is still running fine.

MathRig commented 4 years ago

Sorry, the second error is my bad. I forgot to declare the HOMEPATH /opt/grakn in my profile. It's ok now:

$ yarn serv yarn run v1.22.4 $ node --max_old_space_size=8192 build/index.js info: 🚀 OpenCTI started in production mode with embedded file info: [PRE-CHECK] > Grakn is alive info: [PRE-CHECK] > Elasticsearch is alive info: [PRE-CHECK] > Minio is alive info: [PRE-CHECK] > Python3 is available info: [INIT] > Grakn schema loaded info: [INIT] > Elasticsearch indexes loaded info: [MIGRATION] > Starting migration process info: [MIGRATION] > Fresh platform detected, creating migration structure info: [MIGRATION] > Migration process completed, platform is up to date info: [INIT] > Initialization of settings and basic elements info: [INIT] > Platform default initialized info: [ADMIN_SETUP] admin user initialized info: OPENCTI Ready on port 4000