OpenCTI-Platform / opencti

Open Cyber Threat Intelligence Platform
https://opencti.io
Other
6.37k stars 941 forks source link

Yarn serv display "no module named 'eql'" #4415

Closed alpaysar closed 8 months ago

alpaysar commented 1 year ago

Description

When I run yarn serv command it displays me an error "no module named 'eql'" whereas this module is already installed

Environment

  1. OS Centos 8
  2. OpenCTI version: 5.10.2
  3. OpenCTI client: python

image

alpaysar commented 1 year ago

Could you please help me.

Thanks

tomibennett commented 1 year ago

I'm assuming you're doing a manual installation.

Did you installed dependencies listed in opencti-platform/opencti-graphql/src/python/requirements.txt?

From the repository root:

pip install -r opencti-platform/opencti-graphql/src/python/requirements.txt

You can have a look in the install python modules section of the documentation as well.

alpaysar commented 1 year ago

Yes it's a manual installation and I already installed dependencies listed in opencti-platform/opencti-graphql/src/python/requirements.txt

tomibennett commented 1 year ago

Can you share the output of pip install -r opencti-platform/opencti-graphql/src/python/requirements.txt please?

alpaysar commented 1 year ago

My requirements.txt file is on this path open_CTI/opencti/src/python/requirements.txt

Here is the output of pip3 install -r open_CTI/opencti/src/python/requirements.txt

[root@localhost /]# pip3 install -r open_CTI/opencti/src/python/requirements.txt Requirement already satisfied: pycti==5.10.2 in /usr/local/lib/python3.9/site-packages (from -r open_CTI/opencti/src/python/requirements.txt (line 1)) (5.10.2) Requirement already satisfied: parsuricata==0.4.1 in /usr/local/lib/python3.9/site-packages (from -r open_CTI/opencti/src/python/requirements.txt (line 2)) (0.4.1) Requirement already satisfied: plyara==2.1.1 in /usr/local/lib/python3.9/site-packages (from -r open_CTI/opencti/src/python/requirements.txt (line 3)) (2.1.1) Requirement already satisfied: sigmatools==0.23.1 in /usr/lib/python3.9/site-packages (from -r open_CTI/opencti/src/python/requirements.txt (line 4)) (0.23.1) Requirement already satisfied: jsonpatch==1.33 in /usr/local/lib/python3.9/site-packages (from -r open_CTI/opencti/src/python/requirements.txt (line 5)) (1.33) Requirement already satisfied: eql==0.9.17 in /usr/local/lib/python3.9/site-packages (from -r open_CTI/opencti/src/python/requirements.txt (line 6)) (0.9.17) Requirement already satisfied: cachetools~=5.3.0 in /usr/local/lib/python3.9/site-packages (from pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (5.3.1) Requirement already satisfied: setuptools~=68.2.1 in /usr/local/lib/python3.9/site-packages (from pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (68.2.2) Requirement already satisfied: pyyaml~=6.0 in /usr/local/lib64/python3.9/site-packages (from pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (6.0.1) Requirement already satisfied: python-magic~=0.4.27 in /usr/local/lib/python3.9/site-packages (from pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (0.4.27) Requirement already satisfied: stix2~=3.0.1 in /usr/local/lib/python3.9/site-packages (from pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (3.0.1) Requirement already satisfied: python-json-logger~=2.0.4 in /usr/local/lib/python3.9/site-packages (from pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (2.0.7) Requirement already satisfied: datefinder~=0.7.3 in /usr/local/lib/python3.9/site-packages (from pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (0.7.3) Requirement already satisfied: pika~=1.3.1 in /usr/local/lib/python3.9/site-packages (from pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (1.3.2) Requirement already satisfied: requests~=2.31.0 in /usr/local/lib/python3.9/site-packages (from pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (2.31.0) Requirement already satisfied: filigran-sseclient~=1.0.0 in /usr/local/lib/python3.9/site-packages (from pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (1.0.1) Requirement already satisfied: lark-parser<0.13.0,>=0.12.0 in /usr/local/lib/python3.9/site-packages (from parsuricata==0.4.1->-r open_CTI/opencti/src/python/requirements.txt (line 2)) (0.12.0) Requirement already satisfied: ply>=3.11 in /usr/local/lib/python3.9/site-packages (from plyara==2.1.1->-r open_CTI/opencti/src/python/requirements.txt (line 3)) (3.11) Requirement already satisfied: pymisp in /usr/local/lib/python3.9/site-packages (from sigmatools==0.23.1->-r open_CTI/opencti/src/python/requirements.txt (line 4)) (2.4.176) Requirement already satisfied: progressbar2 in /usr/local/lib/python3.9/site-packages (from sigmatools==0.23.1->-r open_CTI/opencti/src/python/requirements.txt (line 4)) (4.2.0) Requirement already satisfied: ruamel.yaml in /usr/local/lib/python3.9/site-packages (from sigmatools==0.23.1->-r open_CTI/opencti/src/python/requirements.txt (line 4)) (0.17.32) Requirement already satisfied: termcolor in /usr/local/lib/python3.9/site-packages (from sigmatools==0.23.1->-r open_CTI/opencti/src/python/requirements.txt (line 4)) (2.3.0) Requirement already satisfied: jsonpointer>=1.9 in /usr/local/lib/python3.9/site-packages (from jsonpatch==1.33->-r open_CTI/opencti/src/python/requirements.txt (line 5)) (2.4) Requirement already satisfied: pytz in /usr/local/lib/python3.9/site-packages (from datefinder~=0.7.3->pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (2023.3.post1) Requirement already satisfied: regex>=2017.02.08 in /usr/local/lib64/python3.9/site-packages (from datefinder~=0.7.3->pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (2023.8.8) Requirement already satisfied: python-dateutil>=2.4.2 in /usr/local/lib/python3.9/site-packages (from datefinder~=0.7.3->pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (2.8.2) Requirement already satisfied: six in /usr/local/lib/python3.9/site-packages (from filigran-sseclient~=1.0.0->pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (1.16.0) Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib64/python3.9/site-packages (from requests~=2.31.0->pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (3.2.0) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/site-packages (from requests~=2.31.0->pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (3.4) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/site-packages (from requests~=2.31.0->pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (2023.7.22) Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.9/site-packages (from requests~=2.31.0->pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (2.0.4) Requirement already satisfied: stix2-patterns>=1.2.0 in /usr/local/lib/python3.9/site-packages (from stix2~=3.0.1->pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (2.0.0) Requirement already satisfied: simplejson in /usr/local/lib64/python3.9/site-packages (from stix2~=3.0.1->pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (3.19.1) Requirement already satisfied: python-utils>=3.0.0 in /usr/local/lib/python3.9/site-packages (from progressbar2->sigmatools==0.23.1->-r open_CTI/opencti/src/python/requirements.txt (line 4)) (3.7.0) Requirement already satisfied: jsonschema<5.0.0,>=4.19.0 in /usr/local/lib/python3.9/site-packages (from pymisp->sigmatools==0.23.1->-r open_CTI/opencti/src/python/requirements.txt (line 4)) (4.19.0) Requirement already satisfied: deprecated<2.0.0,>=1.2.14 in /usr/local/lib/python3.9/site-packages (from pymisp->sigmatools==0.23.1->-r open_CTI/opencti/src/python/requirements.txt (line 4)) (1.2.14) Requirement already satisfied: publicsuffixlist<0.11.0.0,>=0.10.0.20230828 in /usr/local/lib/python3.9/site-packages (from pymisp->sigmatools==0.23.1->-r open_CTI/opencti/src/python/requirements.txt (line 4)) (0.10.0.20230919) Requirement already satisfied: ruamel.yaml.clib>=0.2.7 in /usr/local/lib64/python3.9/site-packages (from ruamel.yaml->sigmatools==0.23.1->-r open_CTI/opencti/src/python/requirements.txt (line 4)) (0.2.7) Requirement already satisfied: wrapt<2,>=1.10 in /usr/local/lib64/python3.9/site-packages (from deprecated<2.0.0,>=1.2.14->pymisp->sigmatools==0.23.1->-r open_CTI/opencti/src/python/requirements.txt (line 4)) (1.15.0) Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.9/site-packages (from jsonschema<5.0.0,>=4.19.0->pymisp->sigmatools==0.23.1->-r open_CTI/opencti/src/python/requirements.txt (line 4)) (2023.7.1) Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib64/python3.9/site-packages (from jsonschema<5.0.0,>=4.19.0->pymisp->sigmatools==0.23.1->-r open_CTI/opencti/src/python/requirements.txt (line 4)) (0.10.3) Requirement already satisfied: attrs>=22.2.0 in /usr/local/lib/python3.9/site-packages (from jsonschema<5.0.0,>=4.19.0->pymisp->sigmatools==0.23.1->-r open_CTI/opencti/src/python/requirements.txt (line 4)) (23.1.0) Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.9/site-packages (from jsonschema<5.0.0,>=4.19.0->pymisp->sigmatools==0.23.1->-r open_CTI/opencti/src/python/requirements.txt (line 4)) (0.30.2) Requirement already satisfied: typing-extensions>3.10.0.2 in /usr/local/lib/python3.9/site-packages (from python-utils>=3.0.0->progressbar2->sigmatools==0.23.1->-r open_CTI/opencti/src/python/requirements.txt (line 4)) (4.8.0) Requirement already satisfied: antlr4-python3-runtime~=4.9.0 in /usr/local/lib/python3.9/site-packages (from stix2-patterns>=1.2.0->stix2~=3.0.1->pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (4.9.3) WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv WARNING: You are using pip version 22.0.4; however, version 23.2.1 is available. You should consider upgrading via the '/usr/bin/python3.9 -m pip install --upgrade pip' command.

tomibennett commented 1 year ago

Seems right. Though, I wonder about that opencti path's of yours. If you are on the root of the repository it should look like opencti-platform/opencti-graphql/src/python instead of opencti/src/python. But that surely not the issue here.

Ok then, is your PATH correctly configured? What is the output of

python -c 'import eql; print(eql)' 

It should be something like this

<module 'eql' from '/path/to/python/install/directory/python/3.10.10/lib/python3.10/site-packages/eql/__init__.py'>
alpaysar commented 1 year ago

I installed Python 3.9 on my server, so when I run the command python3.9 -c 'import eql; print(eql)' the package is visible: [root@localhost opencti]# python3.9 -c 'import eql; print(eql)' <module 'eql' from '/usr/local/lib/python3.9/site-packages/eql/init.py'>

but when I run the command: python3 -c 'import eql; print(eql)'

It isn't visible: [root@localhost opencti]# python3 -c 'import eql; print(eql)' Traceback (most recent call last): File "", line 1, in ModuleNotFoundError: No module named 'eql'

richard-julien commented 1 year ago

Can you try to execute commands in this order ?

  1. yarn install
  2. yarn build
  3. yarn serv
tomibennett commented 1 year ago

Assuming, you have two python on your system, python3 a python version that existed in the system and python3.9 the one you installed. I guess OpenCTI finds python3 first and work with it, although you want it to work with python3.9 instead.

You can try to:

alpaysar commented 1 year ago

I made an alias for all possible python and it still fails:

image

Unfortunately using python3 doesn't let me use latest version of OpenCTI.

tomibennett commented 1 year ago

I see otherwise, it's working... Well, at least the python issue :)

Here, you've got a configuration issue making ElasticSearch unable to instantiate a new client. It's probably missing an url or something. Did you copied config/default.json from config/production.json as stated in the install main platform section of the documentation?

alpaysar commented 1 year ago

Sorry I was on wrong path, not the good screenshot 😅 image

It still doesn't work...

tomibennett commented 1 year ago

Well,

Just to try it out, does the following works?

$ PATH=$(which python3.9):$PATH
$ yarn serv

Also can you output the following commands ?


$ which python3.9
$ which python3
$ echo $PATH
$ yarn node -e "console.log(process.env['PATH'])"
alpaysar commented 1 year ago

It doesn't work unfortunately

alpaysar commented 1 year ago

image

tomibennett commented 1 year ago

Sorry, I meant from a clean session (without the aliases), to be sure that python versions are different.

It's possible that although we force the use of 3.9 that the dependencies list used are still from your other python.

If indeed there is two python, try to use the version that came with your system instead of the one you installed (with or without uninstalling the 3.9). After moving to a new session, you can use python3 to install dependencies pip3 install -r path/to/src/python/requirements.txt.

Also, the PATH overriding won't work better, but it worth the shot.

tomibennett commented 1 year ago

If you don't make any progress with python, I can advise you to try the docker version instead which removes the need to manage dependencies. Head to doc - using docker

alpaysar commented 1 year ago

when I install dependencies with pip3 instead of pip3.9, I can't install latest version of openCTI, it is limited to the 4.4.1 version.

I succeeded to deploy opencti using docker but I need to install docker-ce package to do it. And when I install this package it downgrades containers-common package beacause the latest version of docker-ce package doesn't support latest version of containers-common package. It causes a failure on our vulnerability scan because containers-common package become not supported.

Do you have a solution for it please?

tomibennett commented 1 year ago

when I install dependencies with pip3 instead of pip3.9, I can't install latest version of openCTI, it is limited to the 4.4.1 version.

And if you upgrade your python using your system package manager? We use python 3.11 and pip 23 in development, and also node >= 20 FYI.

If you prefer using your own python version instead, I suggest relying on tools such as asdf or pyenv or plenty of others... which take care of PATH management for you, thus shadowing the system-wide installed python.

Let me know how that goes.

I succeeded to deploy opencti using docker but I need to install docker-ce package to do it. And when I install this package it downgrades containers-common package beacause the latest version of docker-ce package doesn't support latest version of containers-common package. It causes a failure on our vulnerability scan because containers-common package become not supported.

Do you have a solution for it please?

Not really, I'm not too familiar with docker ecosystem and even less with its dependency management on linux distribution. I advise you to ask your distribution or docker community for that one.

nino-filigran commented 10 months ago

@alpaysar did this discussion helped you out?

alpaysar commented 10 months ago

@nino-filigran unfortunately no

nino-filigran commented 10 months ago

@alpaysar I assume you did also try to reach out slack, right?

alpaysar commented 10 months ago

@nino-filigran nope, I installed it with dockers then

nino-filigran commented 10 months ago

@alpaysar it would worth still reaching out on slack (you can join here), since we also have users who have their own implementation and can potentially help. I'll also look on my side if I can provide some additional help on this ticket.

alpaysar commented 10 months ago

@nino-filigran thank you, I will proceed it if needed, there is no problem with dockers for now

nino-filigran commented 8 months ago

@alpaysar I'm now closing since ticket since it seems that you do not have any issue anymore. Feel free to re-open it if needs be.