Closed alpaysar closed 8 months ago
Could you please help me.
Thanks
I'm assuming you're doing a manual installation.
Did you installed dependencies listed in opencti-platform/opencti-graphql/src/python/requirements.txt
?
From the repository root:
pip install -r opencti-platform/opencti-graphql/src/python/requirements.txt
You can have a look in the install python modules section of the documentation as well.
Yes it's a manual installation and I already installed dependencies listed in opencti-platform/opencti-graphql/src/python/requirements.txt
Can you share the output of pip install -r opencti-platform/opencti-graphql/src/python/requirements.txt
please?
My requirements.txt file is on this path open_CTI/opencti/src/python/requirements.txt
Here is the output of pip3 install -r open_CTI/opencti/src/python/requirements.txt
[root@localhost /]# pip3 install -r open_CTI/opencti/src/python/requirements.txt Requirement already satisfied: pycti==5.10.2 in /usr/local/lib/python3.9/site-packages (from -r open_CTI/opencti/src/python/requirements.txt (line 1)) (5.10.2) Requirement already satisfied: parsuricata==0.4.1 in /usr/local/lib/python3.9/site-packages (from -r open_CTI/opencti/src/python/requirements.txt (line 2)) (0.4.1) Requirement already satisfied: plyara==2.1.1 in /usr/local/lib/python3.9/site-packages (from -r open_CTI/opencti/src/python/requirements.txt (line 3)) (2.1.1) Requirement already satisfied: sigmatools==0.23.1 in /usr/lib/python3.9/site-packages (from -r open_CTI/opencti/src/python/requirements.txt (line 4)) (0.23.1) Requirement already satisfied: jsonpatch==1.33 in /usr/local/lib/python3.9/site-packages (from -r open_CTI/opencti/src/python/requirements.txt (line 5)) (1.33) Requirement already satisfied: eql==0.9.17 in /usr/local/lib/python3.9/site-packages (from -r open_CTI/opencti/src/python/requirements.txt (line 6)) (0.9.17) Requirement already satisfied: cachetools~=5.3.0 in /usr/local/lib/python3.9/site-packages (from pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (5.3.1) Requirement already satisfied: setuptools~=68.2.1 in /usr/local/lib/python3.9/site-packages (from pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (68.2.2) Requirement already satisfied: pyyaml~=6.0 in /usr/local/lib64/python3.9/site-packages (from pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (6.0.1) Requirement already satisfied: python-magic~=0.4.27 in /usr/local/lib/python3.9/site-packages (from pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (0.4.27) Requirement already satisfied: stix2~=3.0.1 in /usr/local/lib/python3.9/site-packages (from pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (3.0.1) Requirement already satisfied: python-json-logger~=2.0.4 in /usr/local/lib/python3.9/site-packages (from pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (2.0.7) Requirement already satisfied: datefinder~=0.7.3 in /usr/local/lib/python3.9/site-packages (from pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (0.7.3) Requirement already satisfied: pika~=1.3.1 in /usr/local/lib/python3.9/site-packages (from pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (1.3.2) Requirement already satisfied: requests~=2.31.0 in /usr/local/lib/python3.9/site-packages (from pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (2.31.0) Requirement already satisfied: filigran-sseclient~=1.0.0 in /usr/local/lib/python3.9/site-packages (from pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (1.0.1) Requirement already satisfied: lark-parser<0.13.0,>=0.12.0 in /usr/local/lib/python3.9/site-packages (from parsuricata==0.4.1->-r open_CTI/opencti/src/python/requirements.txt (line 2)) (0.12.0) Requirement already satisfied: ply>=3.11 in /usr/local/lib/python3.9/site-packages (from plyara==2.1.1->-r open_CTI/opencti/src/python/requirements.txt (line 3)) (3.11) Requirement already satisfied: pymisp in /usr/local/lib/python3.9/site-packages (from sigmatools==0.23.1->-r open_CTI/opencti/src/python/requirements.txt (line 4)) (2.4.176) Requirement already satisfied: progressbar2 in /usr/local/lib/python3.9/site-packages (from sigmatools==0.23.1->-r open_CTI/opencti/src/python/requirements.txt (line 4)) (4.2.0) Requirement already satisfied: ruamel.yaml in /usr/local/lib/python3.9/site-packages (from sigmatools==0.23.1->-r open_CTI/opencti/src/python/requirements.txt (line 4)) (0.17.32) Requirement already satisfied: termcolor in /usr/local/lib/python3.9/site-packages (from sigmatools==0.23.1->-r open_CTI/opencti/src/python/requirements.txt (line 4)) (2.3.0) Requirement already satisfied: jsonpointer>=1.9 in /usr/local/lib/python3.9/site-packages (from jsonpatch==1.33->-r open_CTI/opencti/src/python/requirements.txt (line 5)) (2.4) Requirement already satisfied: pytz in /usr/local/lib/python3.9/site-packages (from datefinder~=0.7.3->pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (2023.3.post1) Requirement already satisfied: regex>=2017.02.08 in /usr/local/lib64/python3.9/site-packages (from datefinder~=0.7.3->pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (2023.8.8) Requirement already satisfied: python-dateutil>=2.4.2 in /usr/local/lib/python3.9/site-packages (from datefinder~=0.7.3->pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (2.8.2) Requirement already satisfied: six in /usr/local/lib/python3.9/site-packages (from filigran-sseclient~=1.0.0->pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (1.16.0) Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib64/python3.9/site-packages (from requests~=2.31.0->pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (3.2.0) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/site-packages (from requests~=2.31.0->pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (3.4) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/site-packages (from requests~=2.31.0->pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (2023.7.22) Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.9/site-packages (from requests~=2.31.0->pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (2.0.4) Requirement already satisfied: stix2-patterns>=1.2.0 in /usr/local/lib/python3.9/site-packages (from stix2~=3.0.1->pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (2.0.0) Requirement already satisfied: simplejson in /usr/local/lib64/python3.9/site-packages (from stix2~=3.0.1->pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (3.19.1) Requirement already satisfied: python-utils>=3.0.0 in /usr/local/lib/python3.9/site-packages (from progressbar2->sigmatools==0.23.1->-r open_CTI/opencti/src/python/requirements.txt (line 4)) (3.7.0) Requirement already satisfied: jsonschema<5.0.0,>=4.19.0 in /usr/local/lib/python3.9/site-packages (from pymisp->sigmatools==0.23.1->-r open_CTI/opencti/src/python/requirements.txt (line 4)) (4.19.0) Requirement already satisfied: deprecated<2.0.0,>=1.2.14 in /usr/local/lib/python3.9/site-packages (from pymisp->sigmatools==0.23.1->-r open_CTI/opencti/src/python/requirements.txt (line 4)) (1.2.14) Requirement already satisfied: publicsuffixlist<0.11.0.0,>=0.10.0.20230828 in /usr/local/lib/python3.9/site-packages (from pymisp->sigmatools==0.23.1->-r open_CTI/opencti/src/python/requirements.txt (line 4)) (0.10.0.20230919) Requirement already satisfied: ruamel.yaml.clib>=0.2.7 in /usr/local/lib64/python3.9/site-packages (from ruamel.yaml->sigmatools==0.23.1->-r open_CTI/opencti/src/python/requirements.txt (line 4)) (0.2.7) Requirement already satisfied: wrapt<2,>=1.10 in /usr/local/lib64/python3.9/site-packages (from deprecated<2.0.0,>=1.2.14->pymisp->sigmatools==0.23.1->-r open_CTI/opencti/src/python/requirements.txt (line 4)) (1.15.0) Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.9/site-packages (from jsonschema<5.0.0,>=4.19.0->pymisp->sigmatools==0.23.1->-r open_CTI/opencti/src/python/requirements.txt (line 4)) (2023.7.1) Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib64/python3.9/site-packages (from jsonschema<5.0.0,>=4.19.0->pymisp->sigmatools==0.23.1->-r open_CTI/opencti/src/python/requirements.txt (line 4)) (0.10.3) Requirement already satisfied: attrs>=22.2.0 in /usr/local/lib/python3.9/site-packages (from jsonschema<5.0.0,>=4.19.0->pymisp->sigmatools==0.23.1->-r open_CTI/opencti/src/python/requirements.txt (line 4)) (23.1.0) Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.9/site-packages (from jsonschema<5.0.0,>=4.19.0->pymisp->sigmatools==0.23.1->-r open_CTI/opencti/src/python/requirements.txt (line 4)) (0.30.2) Requirement already satisfied: typing-extensions>3.10.0.2 in /usr/local/lib/python3.9/site-packages (from python-utils>=3.0.0->progressbar2->sigmatools==0.23.1->-r open_CTI/opencti/src/python/requirements.txt (line 4)) (4.8.0) Requirement already satisfied: antlr4-python3-runtime~=4.9.0 in /usr/local/lib/python3.9/site-packages (from stix2-patterns>=1.2.0->stix2~=3.0.1->pycti==5.10.2->-r open_CTI/opencti/src/python/requirements.txt (line 1)) (4.9.3) WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv WARNING: You are using pip version 22.0.4; however, version 23.2.1 is available. You should consider upgrading via the '/usr/bin/python3.9 -m pip install --upgrade pip' command.
Seems right. Though, I wonder about that opencti path's of yours. If you are on the root of the repository it should look like opencti-platform/opencti-graphql/src/python
instead of opencti/src/python
. But that surely not the issue here.
Ok then, is your PATH correctly configured? What is the output of
python -c 'import eql; print(eql)'
It should be something like this
<module 'eql' from '/path/to/python/install/directory/python/3.10.10/lib/python3.10/site-packages/eql/__init__.py'>
I installed Python 3.9 on my server, so when I run the command python3.9 -c 'import eql; print(eql)' the package is visible: [root@localhost opencti]# python3.9 -c 'import eql; print(eql)' <module 'eql' from '/usr/local/lib/python3.9/site-packages/eql/init.py'>
but when I run the command: python3 -c 'import eql; print(eql)'
It isn't visible:
[root@localhost opencti]# python3 -c 'import eql; print(eql)'
Traceback (most recent call last):
File "
Can you try to execute commands in this order ?
Assuming, you have two python on your system, python3
a python version that existed in the system and python3.9
the one you installed.
I guess OpenCTI finds python3
first and work with it, although you want it to work with python3.9
instead.
You can try to:
python3
instead of python3.9
, install dependencies and work with it instead. python3.9
before python3
in your PATH environment variable, so that when node-calls-python
(the library OpenCTI uses) looks for a Python package, it will find python3.9
first and work with itpython3
completely. At your own risk, some packages on your system may depend on it.python3
, so that when python3
is called, really, it's python3.9
that will be executed. In your shell (will only work for a session)
$ alias python3=python3.9
$ yarn serv
I made an alias for all possible python and it still fails:
Unfortunately using python3 doesn't let me use latest version of OpenCTI.
I see otherwise, it's working... Well, at least the python issue :)
Here, you've got a configuration issue making ElasticSearch unable to instantiate a new client. It's probably missing an url or something. Did you copied config/default.json
from config/production.json
as stated in the install main platform section of the documentation?
Sorry I was on wrong path, not the good screenshot 😅
It still doesn't work...
Well,
Just to try it out, does the following works?
$ PATH=$(which python3.9):$PATH
$ yarn serv
Also can you output the following commands ?
$ which python3.9
$ which python3
$ echo $PATH
$ yarn node -e "console.log(process.env['PATH'])"
It doesn't work unfortunately
Sorry, I meant from a clean session (without the aliases), to be sure that python versions are different.
It's possible that although we force the use of 3.9 that the dependencies list used are still from your other python.
If indeed there is two python, try to use the version that came with your system instead of the one you installed (with or without uninstalling the 3.9).
After moving to a new session, you can use python3
to install dependencies pip3 install -r path/to/src/python/requirements.txt
.
Also, the PATH overriding won't work better, but it worth the shot.
If you don't make any progress with python, I can advise you to try the docker version instead which removes the need to manage dependencies. Head to doc - using docker
when I install dependencies with pip3 instead of pip3.9, I can't install latest version of openCTI, it is limited to the 4.4.1 version.
I succeeded to deploy opencti using docker but I need to install docker-ce package to do it. And when I install this package it downgrades containers-common package beacause the latest version of docker-ce package doesn't support latest version of containers-common package. It causes a failure on our vulnerability scan because containers-common package become not supported.
Do you have a solution for it please?
when I install dependencies with pip3 instead of pip3.9, I can't install latest version of openCTI, it is limited to the 4.4.1 version.
And if you upgrade your python using your system package manager? We use python 3.11 and pip 23 in development, and also node >= 20 FYI.
If you prefer using your own python version instead, I suggest relying on tools such as asdf or pyenv or plenty of others... which take care of PATH management for you, thus shadowing the system-wide installed python.
Let me know how that goes.
I succeeded to deploy opencti using docker but I need to install docker-ce package to do it. And when I install this package it downgrades containers-common package beacause the latest version of docker-ce package doesn't support latest version of containers-common package. It causes a failure on our vulnerability scan because containers-common package become not supported.
Do you have a solution for it please?
Not really, I'm not too familiar with docker ecosystem and even less with its dependency management on linux distribution. I advise you to ask your distribution or docker community for that one.
@alpaysar did this discussion helped you out?
@nino-filigran unfortunately no
@alpaysar I assume you did also try to reach out slack, right?
@nino-filigran nope, I installed it with dockers then
@alpaysar it would worth still reaching out on slack (you can join here), since we also have users who have their own implementation and can potentially help. I'll also look on my side if I can provide some additional help on this ticket.
@nino-filigran thank you, I will proceed it if needed, there is no problem with dockers for now
@alpaysar I'm now closing since ticket since it seems that you do not have any issue anymore. Feel free to re-open it if needs be.
Description
When I run yarn serv command it displays me an error "no module named 'eql'" whereas this module is already installed
Environment