search

MISP / misp-modules

Modules for expansion services, enrichment, import and export in MISP and other tools.
http://misp.github.io/misp-modules
GNU Affero General Public License v3.0
344 stars 233 forks source link

Start MISP-Modules Service #465

Closed khalidrehan closed 3 years ago

khalidrehan commented 3 years ago

Hello, I would like to add a note about starting the misp-modules service. Reference to issue #453, and the installation steps.

I applied the following:

sudo apt-get install python3-dev python3-pip libpq5 libjpeg-dev tesseract-ocr libpoppler-cpp-dev imagemagick virtualenv libopencv-dev zbar-tools libzbar0 libzbar-dev libfuzzy-dev build-essential -y
sudo -u www-data virtualenv -p python3 /var/www/MISP/venv
cd /usr/local/src/
sudo git clone https://github.com/MISP/misp-modules.git
sudo chown -R www-data .
cd misp-modules
sudo -u www-data /var/www/MISP/venv/bin/pip3 install -I -r REQUIREMENTS
sudo -u www-data /var/www/MISP/venv/bin/pip3 install .
# Start misp-modules as a service
sudo cp etc/systemd/system/misp-modules.service /etc/systemd/system/
sudo vim /etc/systemd/system/misp-modules.service
# Edit ExecStart=/var/www/MISP/venv/bin/misp-modules -l 127.0.0.1 -s to be **ExecStart=/var/www/MISP/venv/bin/misp-modules**

image

sudo chmod 774 /etc/systemd/system/misp-modules.service
sudo systemctl daemon-reload
sudo systemctl enable --now misp-modules
sudo systemctl start misp-modules

Regards, Khalid.

chrisinmtown commented 3 years ago

Please explain more. I think you're suggesting a change to this line:

Edit ExecStart=/var/www/MISP/venv/bin/misp-modules -l 127.0.0.1 -s to be ExecStart=/var/www/MISP/venv/bin/misp-modules

to drop the 127.0.0.1 argument, is that right? That may be good for your environment but it's not right for everyone.

khalidrehan commented 3 years ago

Please explain more. I think you're suggesting a change to this line:

Edit ExecStart=/var/www/MISP/venv/bin/misp-modules -l 127.0.0.1 -s to be ExecStart=/var/www/MISP/venv/bin/misp-modules

to drop the 127.0.0.1 argument, is that right? That may be good for your environment but it's not right for everyone.

Service did not start with 127.0.0.1 argument. Please, check the reference mentioned [issue #453].

chrisinmtown commented 3 years ago

The requirements for misp-modules have been updated so you might want to try again, but there may still be a problem with Python 3.6, see #463. You also might try Python 3.7 or later.

khalidrehan commented 3 years ago

The requirements for misp-modules have been updated so you might want to try again, but there may still be a problem with Python 3.6, see #463. You also might try Python 3.7 or later.

Thanks for the advice :)) It had to update to Python 3.7: Could not find a version that satisfies the requirement pandas==1.2.1

But now I'm stuck with this error:

  Building wheel for yara-python (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /var/www/MISP/venv/bin/python3.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-tsl8hwnr/yara-python_37ddfe14810d47c8a51195c18c10431c/setup.py'"'"'; __file__='"'"'/tmp/pip-install-tsl8hwnr/yara-python_37ddfe14810d47c8a51195c18c10431c/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-aaj13jo5
       cwd: /tmp/pip-install-tsl8hwnr/yara-python_37ddfe14810d47c8a51195c18c10431c/
  Complete output (16 lines):
  running bdist_wheel
  running build
  running build_ext
  building 'yara' extension
  creating build
  creating build/temp.linux-x86_64-3.7
  creating build/temp.linux-x86_64-3.7/yara
  creating build/temp.linux-x86_64-3.7/yara/libyara
  creating build/temp.linux-x86_64-3.7/yara/libyara/proc
  creating build/temp.linux-x86_64-3.7/yara/libyara/modules
  x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fdebug-prefix-map=/build/python3.7-zVxTFU/python3.7-3.7.9=. -fstack-protector-strong -Wformat -Werror=format-security -g -fdebug-prefix-map=/build/python3.7-zVxTFU/python3.7-3.7.9=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DUSE_LINUX_PROC=1 -DHAVE_MEMMEM=1 -DHASH_MODULE=1 -DHAVE_LIBCRYPTO=1 -Iyara/libyara/include -Iyara/libyara/ -I. -I/usr/include/python3.7m -I/var/www/MISP/venv/include/python3.7m -c yara-python.c -o build/temp.linux-x86_64-3.7/yara-python.o
  yara-python.c:20:10: fatal error: Python.h: No such file or directory
   #include <Python.h>
            ^~~~~~~~~~
  compilation terminated.
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for yara-python
  Running setup.py clean for yara-python
Successfully built antlr4-python3-runtime blockchain colorclass compressed-rtf dnsdb2 dnspython3 ez-setup ezodf future json-log-formatter maxminddb msoffcrypto-tool np olefile oletools pypssl pyrsistent pytesseract python-baseconv python-docx python-pptx shodan socketio-client trustar unicodecsv urlarchiver validators wrapt
Failed to build pdftotext pydeep yara-python
Installing collected packages: pycparser, six, cffi, zipp, typing-extensions, olefile, cryptography, wrapt, urllib3, setuptools, pytz, python-engineio, pyrsistent, pyparsing, pcodedmp, multidict, msoffcrypto-tool, importlib-metadata, idna, easygui, colorclass, chardet, certifi, bidict, attrs, yarl, websocket-client, tzlocal, soupsieve, requests, python-utils, python-socketio, python-dateutil, pyopenssl, pycryptodomex, pillow, oletools, numpy, lark-parser, jsonschema, isodate, imapclient, ebcdic, deprecated, decorator, compressed-rtf, click, async-timeout, xlsxwriter, validators, url-normalize, unicodecsv, tqdm, tornado, tabulate, socketio-client, rtfde, requests-cache, reportlab, redis, rdflib, pyzipper, pyyaml, python-magic, python-baseconv, pymisp, pydeep, pycryptodome, psutil, progressbar2, pandas, maxminddb, lxml, lief, json-log-formatter, httplib2, futures, future, ezodf, ez-setup, extract-msg, enum-compat, dnspython, configparser, colorama, click-plugins, beautifulsoup4, argparse, antlr4-python3-runtime, aiohttp, yara-python, xlrd, wand, vulners, vt-graph-api, uwhois, urlarchiver, trustar, stix2-patterns, sparqlwrapper, socialscan, sigmatools, shodan, pyzbar, python-pptx, python-docx, pytesseract, pypssl, pypdns, pyonyphe, pyipasnhistory, pyintel471, pygeoip, pyeupi, pydnstrails, pybgpranking, pdftotext, passivetotal, pandas-ods-reader, opencv-python, odtreader, oauth2, np, misp-modules, markdownify, maclookup, jbxapi, geoip2, domaintools-api, dnspython3, dnsdb2, clamd, blockchain, backscatter, assemblyline-client, apiosintds
    Running setup.py install for pydeep ... error
    ERROR: Command errored out with exit status 1:
     command: /var/www/MISP/venv/bin/python3.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-tsl8hwnr/pydeep_46e73119e27a48c3be09050d003a06cf/setup.py'"'"'; __file__='"'"'/tmp/pip-install-tsl8hwnr/pydeep_46e73119e27a48c3be09050d003a06cf/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-v_r8n2ap/install-record.txt --single-version-externally-managed --compile --install-headers /var/www/MISP/venv/include/site/python3.7/pydeep
         cwd: /tmp/pip-install-tsl8hwnr/pydeep_46e73119e27a48c3be09050d003a06cf/
    Complete output (12 lines):
    running install
    running build
    running build_ext
    building 'pydeep' extension
    creating build
    creating build/temp.linux-x86_64-3.7
    x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fdebug-prefix-map=/build/python3.7-zVxTFU/python3.7-3.7.9=. -fstack-protector-strong -Wformat -Werror=format-security -g -fdebug-prefix-map=/build/python3.7-zVxTFU/python3.7-3.7.9=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/local/include/ -I/usr/include/python3.7m -I/var/www/MISP/venv/include/python3.7m -c pydeep.c -o build/temp.linux-x86_64-3.7/pydeep.o
    pydeep.c:1:10: fatal error: Python.h: No such file or directory
     #include <Python.h>
              ^~~~~~~~~~
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /var/www/MISP/venv/bin/python3.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-tsl8hwnr/pydeep_46e73119e27a48c3be09050d003a06cf/setup.py'"'"'; __file__='"'"'/tmp/pip-install-tsl8hwnr/pydeep_46e73119e27a48c3be09050d003a06cf/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-v_r8n2ap/install-record.txt --single-version-externally-managed --compile --install-headers /var/www/MISP/venv/include/site/python3.7/pydeep Check the logs for full command output.

Can I do it without wheel?

Regards, Khalid.

adulau commented 3 years ago

You are missing the Python dev package.

sudo apt-get install python3-dev

chrisinmtown commented 3 years ago

@adulau he may be missing packages but IMHO the first thing to check here, before installing a bunch of other packages, is the version of pip (pip3) and upgrade to the latest pip3. Sometime fairly recently pip3 learned how to install binary packages (wheels) in a new way. Previous versions of pip3 would download source and attempt to compile it into a wheel. Sorry to be vague, it's not clear to me exactly what changed.

khalidrehan commented 3 years ago

apt-get install python3-dev

python3-dev is already installed as a basic requirement, but still did not pass the error.

It worked, but with python3.7-dev for my case.. Thank you @adulau 👍 

root@c393a9cc377f:/usr/local/src/misp-modules# sudo -u www-data /var/www/MISP/venv/bin/pip3.7 install yara-python
WARNING: The directory '/root/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting yara-python
  Downloading yara-python-4.0.4.tar.gz (405 kB)
     |████████████████████████████████| 405 kB 442 kB/s
Building wheels for collected packages: yara-python
  Building wheel for yara-python (setup.py) ... done
  Created wheel for yara-python: filename=yara_python-4.0.4-cp37-cp37m-linux_x86_64.whl size=539901 sha256=e76252db18748de4ee71b51743b8736e9c116a9a979ae567ba947a810b2cdb16
  Stored in directory: /tmp/pip-ephem-wheel-cache-7miq2te0/wheels/18/8a/d7/921b4c10755a49710f84e058ebd2d275aafc2e8d621c51c887
Successfully built yara-python
Installing collected packages: yara-python
Successfully installed yara-python-4.0.4
adulau commented 3 years ago

You're welcome. Glad to see it working.

waredot commented 4 weeks ago

Oct 04 22:45:41 vultr systemd[1]: misp-modules.service: Main process exited, code=exited, status=1/FAILURE Oct 04 22:45:41 vultr systemd[1]: misp-modules.service: Failed with result 'exit-code'. Oct 04 22:45:41 vultr systemd[1]: misp-modules.service: Consumed 1.725s CPU time. Oct 04 22:45:46 vultr systemd[1]: misp-modules.service: Scheduled restart job, restart counter is at 13. Oct 04 22:45:46 vultr systemd[1]: Stopped MISP Modules. Oct 04 22:45:46 vultr systemd[1]: misp-modules.service: Consumed 1.725s CPU time. Oct 04 22:45:46 vultr systemd[1]: Started MISP Modules. Oct 04 22:45:47 vultr python3[136046]: ERROR:root:No module named 'dnstrails' Oct 04 22:45:47 vultr python3[136046]: Traceback (most recent call last): Oct 04 22:45:47 vultr python3[136046]: File "/var/www/MISP/app/files/scripts/misp-modules/misp_modules/init.py", line 94, in Oct 04 22:45:47 vultr python3[136046]: from .modules import # noqa Oct 04 22:45:47 vultr python3[136046]: File "/var/www/MISP/app/files/scripts/misp-modules/misp_modules/modules/init.py", line 1, in Oct 04 22:45:47 vultr python3[136046]: from .expansion import # noqa Oct 04 22:45:47 vultr python3[136046]: File "/var/www/MISP/app/files/scripts/misp-modules/misp_modules/modules/expansion/securitytrails.py", line 6, in Oct 04 22:45:47 vultr python3[136046]: from dnstrails import APIError Oct 04 22:45:47 vultr python3[136046]: ModuleNotFoundError: No module named 'dnstrails' Oct 04 22:45:47 vultr python3[136046]: /var/www/MISP/app/files/scripts/misp-modules/venv/bin/python3: No module named misp_modules.main; 'misp_modules' is a package and cannot be directly executed Oct 04 22:45:47 vultr systemd[1]: misp-modules.service: Main process exited, code=exited, status=1/FAILURE