Unable to Upload Files and Missing STIX and Cybox libraries

[saeeda12]

Work environment

Questions	Answers
Type of issue	Support
OS version (server)	Ubuntu 18.04
OS version (client)	Win 8
PHP version	7.2.15-0ubuntu0.18.04.1
MISP version / git hash	v2.4.102 (98cb4e63d419d9fd6811ae214f67c977c8c40903)
Browser	Chrome

Expected behavior

Be able to upload XML, JSON, STIX format files

Actual behavior

Returns either Internal error or Could not import STIX document: Issues executing the ingestion script or invalid input. Please ask your administrator to check whether the dependencies for STIX are met via the diagnostic tool.

Steps to reproduce the behavior

Attempting to upload any of those file formats.

Logs, screenshots, configuration dump, ...

Further, in the Diagnostics page in MISP, I see the below errors, even though Pydeep PyMISP, STIX, lief, magic, maec, mixbox, cybox were all installed per the instructions (https://github.com/MISP/MISP/blob/2.4/docs/INSTALL.ubuntu1804.md)

[saeeda12]

The output of the app/tmp/logs/exec-errors.log is below:

Traceback (most recent call last):
  File "/var/www/MISP/app/files/scripts/stix2misp.py", line 25, in <module>
    import stix.extensions.marking.ais
ModuleNotFoundError: No module named 'stix'

From error.log:

2019-02-20 15:15:31 Error: Fatal Error (1): Allowed memory size of 2097152 bytes exhausted (tried to allocate 32768 bytes) in [/var/www/MISP/app/Controller/EventsController.php, line 4186]
2019-02-20 15:15:31 Error: [InternalErrorException] Internal Server Error
Request URL: /
Stack Trace:
#0 /var/www/MISP/app/Lib/cakephp/lib/Cake/Error/ErrorHandler.php(212): ErrorHandler::handleFatalError(1, 'Allowed memory ...', '/var/www/MISP/a...', 4186)
#1 /var/www/MISP/app/Lib/cakephp/lib/Cake/Core/App.php(970): ErrorHandler::handleError(1, 'Allowed memory ...', '/var/www/MISP/a...', 4186, Array)
#2 /var/www/MISP/app/Lib/cakephp/lib/Cake/Core/App.php(943): App::_checkFatalError()
#3 [internal function]: App::shutdown()
#4 {main}

I did run the below from the installation script:

for key in upload_max_filesize post_max_size max_execution_time max_input_time memory_limit
  do
      sudo sed -i "s/^\($key\).*/\1 = $(eval echo \${$key})/" $PHP_INI
  done

I also don't have anything listed in my /usr/local/lib/python2.7/dist-packages and /usr/local/lib/python3.6/dist-packages folders.

Should I have used pip3 to install the libraries?

[adulau]

You should use pip3 to install all the libraries related to STIX. You should also check if your submodules are properly sync and updated.

[saeeda12]

Ah, okay. So when I try uninstalling stix, cybox, mixbox, pymisp, using pip3 it says it's not even installed. I reinstalled everything using pip3, but still my /usr/local/lib/python3.6/dist-packages is empty and I actually can't even get to my MISP instance now with a "the page isn't working" error.

Submodules are updated, ran the below again.

sudo -u www-data git submodule update --init --recursive
sudo -u www-data git submodule foreach --recursive git config core.filemode false
sudo -u www-data git config core.filemode false

[adulau]

Could you run sudo -u www-data git submodule sync before your previous commands.

[saeeda12]

Hi @adulau, I ran the sync command and then the three submodule update commands again.

misp@misp:/var/www/MISP$ sudo -u www-data git submodule sync
Synchronizing submodule url for 'INSTALL/dependencies/Console_CommandLine'
Synchronizing submodule url for 'INSTALL/dependencies/Crypt_GPG'
Synchronizing submodule url for 'INSTALL/dependencies/Net_GeoIP'
Synchronizing submodule url for 'Plugin/DebugKit'
Synchronizing submodule url for 'PyMISP'
Synchronizing submodule url for 'app/Lib/cakephp'
Synchronizing submodule url for 'app/Lib/random_compat'
Synchronizing submodule url for 'app/files/misp-galaxy'
Synchronizing submodule url for 'app/files/misp-objects'
Synchronizing submodule url for 'app/files/noticelists'
Synchronizing submodule url for 'app/files/taxonomies'
Synchronizing submodule url for 'app/files/warninglists'
Synchronizing submodule url for 'cti-python-stix2'
Synchronizing submodule url for 'misp-vagrant'
misp@misp:/var/www/MISP$
misp@misp:/var/www/MISP$ sudo -u www-data git submodule update --init --recursive
misp@misp:/var/www/MISP$
misp@misp:/var/www/MISP$ sudo -u www-data git submodule foreach --recursive git config core.filemode false
Entering 'INSTALL/dependencies/Console_CommandLine'
Entering 'INSTALL/dependencies/Crypt_GPG'
Entering 'INSTALL/dependencies/Net_GeoIP'
Entering 'Plugin/DebugKit'
Entering 'PyMISP'
Entering 'PyMISP/pymisp/data/misp-objects'
Entering 'app/Lib/cakephp'
Entering 'app/Lib/random_compat'
Entering 'app/files/misp-galaxy'
Entering 'app/files/misp-objects'
Entering 'app/files/noticelists'
Entering 'app/files/taxonomies'
Entering 'app/files/warninglists'
Entering 'cti-python-stix2'
Entering 'misp-vagrant'
misp@misp:/var/www/MISP$
misp@misp:/var/www/MISP$ sudo -u www-data git config core.filemode false
misp@misp:/var/www/MISP$

However, I am still unable to browse to my misp instance. Getting an HTTP ERROR 500 with "this page isn't working." My config.php and core.php files look okay, but not sure what to look at to fix this.

I am running PHP version 7.2, but in the /app/Plugin/CakeResque/Config/config.php file, it says PHP version 5?

[adulau]

What's mentioned in the error logs? MISP-DIR/app/tmp/logs/error.log?

[saeeda12]

Should I try the steps in #3747?

Yup, error logs are same as above:

Output of the app/tmp/logs/exec-errors.log is below:

Traceback (most recent call last):
  File "/var/www/MISP/app/files/scripts/stix2misp.py", line 25, in <module>
    import stix.extensions.marking.ais
ModuleNotFoundError: No module named 'stix'

From error.log:

2019-02-25 13:26:37 Error: Fatal Error (1): Allowed memory size of 2097152 bytes exhausted (tried to allocate 32768 bytes) in [/var/www/MISP/app/Controller/EventsController.php, line 4101]
2019-02-25 13:26:37 Error: [InternalErrorException] Internal Server Error
Request URL: /
Stack Trace:
#0 /var/www/MISP/app/Lib/cakephp/lib/Cake/Error/ErrorHandler.php(212): ErrorHandler::handleFatalError(1, 'Allowed memory ...', '/var/www/MISP/a...', 4101)
#1 /var/www/MISP/app/Lib/cakephp/lib/Cake/Core/App.php(970): ErrorHandler::handleError(1, 'Allowed memory ...', '/var/www/MISP/a...', 4101, Array)
#2 /var/www/MISP/app/Lib/cakephp/lib/Cake/Core/App.php(943): App::_checkFatalError()
#3 [internal function]: App::shutdown()
#4 {main}

[saeeda12]

Update: I thought I had set the memory_limit = 512M in php.ini, but it got cleared, now I can get to my misp instance again, so that part is solved.

[saeeda12]

@adulau

When I try uninstalling:

misp@misp:/var/www/MISP/app/files/scripts$ pip3 uninstall stix
Cannot uninstall requirement stix, not installed
misp@misp:/var/www/MISP/app/files/scripts$ pip3 uninstall cybox
Cannot uninstall requirement cybox, not installed
misp@misp:/var/www/MISP/app/files/scripts$ pip3 uninstall mixbox
Cannot uninstall requirement mixbox, not installed

Should I delete anything in this folder before re-installing the modules?

misp@misp:/var/www/MISP/app/files/scripts$ ls -la
total 240
drwxr-s--- 11 www-data www-data  4096 Feb 25 19:08 .
drwxr-s--- 10 www-data www-data  4096 Feb 25 21:24 ..
-rwxr-x---  1 www-data www-data     0 Feb 15 00:01 empty
-rwxr-x---  1 www-data www-data  2146 Feb 15 00:01 generate_file_objects.py
-rwxr-x---  1 www-data www-data 86570 Feb 15 00:01 misp2stix.py
-rwxr-x---  1 www-data www-data 10944 Feb 15 00:01 misp_framing.py
drwxr-s---  2 www-data www-data  4096 Feb 20 16:27 mispzmq
drwxr-s---  5 www-data www-data  4096 Feb 15 00:02 mixbox
drwxr-s---  2 www-data www-data  4096 Feb 20 16:16 __pycache__
drwxr-s---  6 www-data www-data  4096 Feb 15 00:02 python-cybox
drwxr-s---  7 www-data www-data  4096 Feb 15 00:02 python-maec
drwxr-s---  6 www-data www-data  4096 Feb 15 00:02 python-stix
-rwxr-x---  1 www-data www-data   242 Feb 15 00:01 selftest.php
drwxr-s---  2 www-data www-data  4096 Feb 21 16:48 stix2
-rwxr-x---  1 www-data www-data  4743 Feb 15 00:01 stix2misp_mapping.py
-rwxr-x---  1 www-data www-data 61597 Feb 15 00:01 stix2misp.py
-rwxr-x---  1 www-data www-data  2218 Feb 15 00:01 stixsighting2misp.py
drwxr-s---  2 www-data www-data  4096 Feb 15 00:01 stixtest
-rwxr-x---  1 www-data www-data  1083 Feb 15 00:01 stixtest.py
-rwxr-x---  1 www-data www-data  6254 Feb 15 00:01 test_payload.txt
drwxr-s---  2 www-data www-data  4096 Feb 25 21:24 tmp

In issue 3747, the user iamtmehdi deleted the /var/www/MISP/PyMISP folder.

Then I should redo these steps from the installation script (using pip3 instead)?

# install mixbox to accommodate the new STIX dependencies:
  sudo -H -u www-data git clone https://github.com/CybOXProject/mixbox.git
  cd ${PATH_TO_MISP}/app/files/scripts/mixbox
  sudo -H -u www-data ${PATH_TO_MISP}/venv/bin/pip install .
  cd ${PATH_TO_MISP}/app/files/scripts/python-cybox
  sudo -H -u www-data ${PATH_TO_MISP}/venv/bin/pip install .
  cd ${PATH_TO_MISP}/app/files/scripts/python-stix
  sudo -H -u www-data ${PATH_TO_MISP}/venv/bin/pip install .
  cd $PATH_TO_MISP/app/files/scripts/python-maec
  sudo -H -u www-data ${PATH_TO_MISP}/venv/bin/pip install .
  # install STIX2.0 library to support STIX 2.0 export:
  cd ${PATH_TO_MISP}/cti-python-stix2
  sudo -H -u www-data ${PATH_TO_MISP}/venv/bin/pip install .

  # install PyMISP
  cd ${PATH_TO_MISP}/PyMISP
  sudo -H -u www-data ${PATH_TO_MISP}/venv/bin/pip install .

[saeeda12]

@adulau Per #3377 (https://github.com/MISP/MISP/issues/3377), when I try python3 setup.py install --record file-to-record-param.txt, I get the below:

running install
running bdist_egg
running egg_info
creating cybox.egg-info
error: could not create 'cybox.egg-info': Permission denied

I've made sure all my permissions/ ownership is correct. Any ideas?

[saeeda12]

Okay, so the above issue's suggestion did work, just needed that sudo. Running python3 stixtest.py showed none of my modules installed, so I did:

cd /var/www/MISP/app/files/scripts/
sudo rm -rf python-cybox
sudo rm -rf python-stix
sudo rm -rf python-maec

sudo -u www-data -H git clone https://github.com/CybOXProject/python-cybox.git
cd /var/www/MISP/app/files/scripts/python-cybox   # Also needed to fix permissions/ ownership
sudo python3 setup.py install
pip3 install stix
pip3 install pymisp

# And did it again for maec
sudo -H -u www-data git clone https://github.com/MAECProject/python-maec.git
cd /var/www/MISP/app/files/scripts/python-maec    # Also needed to fix permissions/ ownership
sudo python3 setup.py install
pip3 install stix
pip3 install pymisp

# And again for stix2
cd ${PATH_TO_MISP}/app/files/scripts
sudo -H -u www-data git clone https://github.com/STIXProject/python-stix.git #Ran this again, just in case
cd /var/www/MISP/cti-python-stix2      # Permissions all looked good
sudo python3 setup.py install
pip3 install stix
pip3 install pymisp

Then, running python3 stixtest.py again now shows (yay):

python3 stixtest.py
{"success": 1, "stix": "1.2.0.6", "cybox": "2.1.0.17", "mixbox": "1.0.3", "maec": "4.1.0.14", "stix2": "1.1.1", "pymisp": "2.4.102"}

BUT, the Diagnostics page still shows these errors:

[saeeda12]

When I tried to run setup.py in the PyMISP dir, I got errors on "ReportLab"?

/var/www/MISP/PyMISP$ sudo python3 setup.py install
ReportLab cannot be imported. Please verify that ReportLab is installed on the system.
Traceback (most recent call last):
  File "setup.py", line 7, in <module>
    import pymisp
  File "/var/www/MISP/PyMISP/pymisp/__init__.py", line 45, in <module>
    from .tools import reportlab_generator  # noqa
  File "/var/www/MISP/PyMISP/pymisp/tools/reportlab_generator.py", line 52, in <module>
    class Flowable_Tag(Flowable):
NameError: name 'Flowable' is not defined

And for pydeep, lief, and python-magic, where can I check that they are installed? I re-ran the below with pip3, but no recognition in Diagnostics page:

# install pydeep
  $SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip3 install git+https://github.com/kbandla/pydeep.git

  # install lief
  $SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip3 install https://github.com/lief-project/packages/raw/lief-master-latest/pylief-0.9.0.dev.zip

  # install python-magic
  $SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip3 install python-magic

[saeeda12]

Still stuck on this unfortunately - anyone have any suggestions?

[SteveClement]

Dear

You have to be careful whether you install the Python modules in the Virtualenv or not.

Do the following command on your instance. My value is: "value": "/var/www/MISP/venv/bin/python", Meaning MISP will use the virtualenv.

sudo -u www-data /var/www/MISP/app/Console/cake Admin getSetting MISP.python_bin |tail -n +7 |jq
{
  "level": 1,
  "description": "It is highly recommended to install all the python dependencies in a virtualenv. The recommended location is: /var/www/MISP/venv",
  "value": "/var/www/MISP/venv/bin/python",
  "errorMessage": "",
  "null": false,
  "test": "testForBinExec",
  "beforeHook": "beforeHookBinExec",
  "type": "string",
  "tab": "MISP",
  "setting": "MISP.python_bin"
}

[saeeda12]

Hi @SteveClement

Ah, so it appears somehow my value is not set correctly. How can I change this?

 sudo -u www-data /var/www/MISP/app/Console/cake Admin getSetting MISP.python_bin |tail -n +7 |jq
{
  "level": 1,
  "description": "It is highly recommended to install all the python dependencies in a virtualenv. The recommended location is: /var/www/MISP/venv",
  "value": false,
  "errorMessage": "Value not set.",
  "null": false,
  "test": "testForBinExec",
  "beforeHook": "beforeHookBinExec",
  "type": "string",
  "error": 1,
  "tab": "MISP",
  "setting": "MISP.python_bin"
}

[saeeda12]

I'm looking at everything below Initialize MISP configuration and set some defaults from https://misp.github.io/MISP/INSTALL.ubuntu1804/ - which has more instructions after Step 9 that the install script on Github doesn't.

So I had to run the below:

# The default install is Python in a virtualenv, setting accordingly
  $SUDO_WWW $CAKE Admin setSetting "MISP.python_bin" "${PATH_TO_MISP}/venv/bin/python"

for my settings to match @SteveClement 's. And now, my Diagnostics page is passing:

BUT, still when I try to import an XML file for Stix 1.1.1, 2.0 or the MISP (when I choose this option, it returns "an internal error has occurred" in the UI), it still returns the original error that started this issue.

[saeeda12]

Might be something to do with these errors?

Module System
This tool tests the various module systems and whether they are reachable based on the module settings.

Enrichment module system…System not enabled
Import module system…System not enabled
Export module system…System not enabled
Cortex module system…System not enabled

So, I had to enable and set those settings - also found at https://misp.github.io/MISP/INSTALL.ubuntu1804/ - now as the connection is refused, do I just need to open up port 6666 from my MISP instance or is something else required to successfully import files?

[saeeda12]

Hi, can anyone assist with getting the import/ export modules functioning?

[saeeda12]

I am able to successfully upload a sample JSON file in STIX 2.0 format - but still not XML or JSON with the other import options - MISP standard format and STIX 1.1.1

[saeeda12]

Alternatively (to import an event via API found via #1298), is there a flag or option I can add in this command so that it includes the original imported file as an attachment in the event (in the attributes)?

curl -i -H "Accept: application/xml" -H "content-type: application/xml" -H "Authorization: [insert key here]" --data @[filename] -X POST [misp_url]/events

[saeeda12]

Hello - I am still unable to import XML files or even some JSON files using the "Import from..." option in the event actions page. Not sure what the issue is.

[saeeda12]

Hi @SteveClement , so I've updated my MISP instance to v2.4.107 (2a2e7aebcd4b5150c231775ae36d775bf3489b44)

And per the stixtest.py:

/var/www/MISP/app/files/scripts$ python3 stixtest.py
{"success": 1, "stix": "1.2.0.6", "cybox": "2.1.0.17", "mixbox": "1.0.3", "maec": "4.1.0.14", "stix2": "1.1.2", "pymisp": "2.4.103"}

My libraries are up to date.

But in the Diagnostics page, it says my STIX2 version is still at 1.1.1

Trying to upload the stix2_indicators_test.json file in the app/files/scripts/stixtest directory as an imported event doesn't work still. Error says "Could not import STIX document: Issues executing the ingestion script or invalid input. Please check whether the dependencies for STIX are met via the diagnostic tool." But uploading the stix2_observables_test.json works perfectly?