How to get minemeld-ansible to install on Ubuntu 20.04

[da667]

Hey there, this is less of a trouble ticket and more of "If you're stuck using this, and you MUST get it working on Ubuntu 20.04, here are the general steps I followed to do so."

Now bear in mind, this project still uses python 2, pip version 2, and probably woefully updated libraries, etc. as a part of the installation process. Also bear in mind that python has been stating that for about a year now that the pip2 repos could be removed at any time, since 2.x is no longer supported anywhere.

Anyway, with that being said, this is a cliffnotes dump on what I needed to do to install minemeld on Ubuntu 20.04 and get it into a mostly operational state:

pre-reqs:

apt-get install python2.7-dev libffi-dev libssl-dev make
ln -s /usr/bin/python2.7 /usr/bin/python

To install pip 2.7

wget https://bootstrap.pypa.io/pip/2.7/get-pip.py
python2.7 get-pip.py

You need ansible to use the minemeld ansible script: pip install ansible

Now, grab this repo: git clone https://github.com/PaloAltoNetworks/minemeld-ansible

change the following fields in local.yml:

    minemeld_core_version: master
    minemeld_webui_version: master
    minemeld_prototypes_version: master

Ubuntu 20.04-specific ansible things: navigate to minemeld-ansible/roles/infrastructure/vars and run: cp Ubuntu-18.04.yml Ubuntu-20.04.yml

navigate to minemeld-ansible/roles/minemeld/vars and run: cp Ubuntu-18.04.yml Ubuntu-20.04.yml

To kick off the install, run: ansible-playbook -K -i 127.0.0.1, local.yml

note: for ubuntu 20.04, I had to include this in the ansible-playbook command: ansible_python_interpreter=/usr/bin/python2

After the install, If you get the following error from nginx (which you almost certainly will):

nginx: [emerg] SSL_CTX_use_certificate("/etc/nginx/minemeld.cer") failed (SSL: error:140AB18F:SSL routines:SSL_CTX_use_certificate:ee key too small)

Run these commands:

cd /etc/nginx
openssl req -x509 -nodes -newkey rsa:4096 -keyout minemeld.pem -out minemeld.cer -sha256 -days 365

The commands above will create a self-signed SSL cert with parameters that nginx can actually use.

Now, there's another problem ujson and newer versions of glibc that modern Linux distros use. The easiest way to solve this problem is to modify the following files:

/opt/minemeld/local/library/constraints.txt
/opt/minemeld/engine/core/minemeld_core.egg-info/requires.txt
/opt/minemeld/engine/core/requirements.txt

find the line that requests ujson 1.34, and change that line, in all three files to ujson version 2.0.3. After doing that, run:

source /opt/minemeld/engine/current/bin/activate
pip install ujson
exit

This should place you in the python virtual environment for minemeld, allow you to install ujson, then exit the virtual environment.

You might also run into a problem where redis isn't creating /var/run/redis.sock confirm that redis service is enabled:

systemctl enable redis.service

stop/start the redis service:

systemctl stop redis.service
systemctl start redis.service

Finally, I also experienced a problem after installation where gunicorn was running, but I couldn't interact with the webUI, even after running:

systemctl restart minemeld.service

Don't be afraid to run:

killall gunicorn

or to reboot the service to clear the fail state.

At this point, I was able to log in and interact with the web interface.

I hope this braindump helps you, and may you find a way to migrate off of this abandonware.

[ANeilan]

hey just a thing i vaguely remember from ubuntu 20.04, instead of symlinking /usr/bin/python2.7 to /usr/bin/python, there should still be a package (aptly named) called python-is-python2

so just install it with sudo apt install python-is-python2

[amateo]

Great post! Thank you very much!!!!

[augustodoug]

I'm having a problem running "ansible-playbook -K -i 127.0.0.1, local.yml" and I'm getting the following response:

TASK [minemeld : requirements] ***** fatal: [127.0.0.1]: FAILED! => {"changed": false, "cmd": ["/opt/minemeld/engine/current/bin/pip2", "install", "-r", "/opt/minemeld/engine/core/requirements.txt"], "msg": "stdout: Requirement already satisfied: pip>=9.0.1 in /opt/minemeld/engine/current/lib/python2.7/site-packages (from -r /opt/minemeld/engine/core/requirements.txt (line 1)) (20.3.4)\nCollecting amqp==1.4.6\n Using cached amqp-1.4.6-py2-none-any.whl (49 kB)\nCollecting gevent==1.0.2\n Using cached gevent-1.0.2.tar.gz (1.7 MB)\nCollecting greenlet==0.4.7\n Using cached greenlet-0.4.7.zip (77 kB)\nCollecting hiredis==0.2.0\n Using cached hiredis-0.2.0-cp27-cp27mu-manylinux1_x86_64.whl (45 kB)\nCollecting PyYAML==5.4\n Using cached PyYAML-5.4-cp27-cp27mu-manylinux1_x86_64.whl (574 kB)\nCollecting redis==2.10.5\n Using cached redis-2.10.5-py2.py3-none-any.whl (60 kB)\nCollecting requests==2.20.0\n Using cached requests-2.20.0-py2.py3-none-any.whl (60 kB)\nCollecting plyvel==0.9\n Using cached plyvel-0.9.tar.gz (121 kB)\nCollecting netaddr==0.7.18\n Using cached netaddr-0.7.18-py2.py3-none-any.whl (1.5 MB)\nCollecting jmespath==0.7.1\n Using cached jmespath-0.7.1-py2.py3-none-any.whl (19 kB)\nCollecting click==4.1\n Using cached click-4.1-py2.py3-none-any.whl (62 kB)\nCollecting pan-python==0.10.0\n Using cached pan-python-0.10.0.tar.gz (95 kB)\nCollecting stix==1.1.1.8\n Using cached stix-1.1.1.8-py2.py3-none-any.whl (263 kB)\nCollecting cybox==2.1.0.17\n Using cached cybox-2.1.0.17-py2.py3-none-any.whl (734 kB)\nCollecting six==1.11.0\n Using cached six-1.11.0-py2.py3-none-any.whl (10 kB)\nCollecting lxml==4.6.3\n Using cached lxml-4.6.3-cp27-cp27mu-manylinux1_x86_64.whl (5.5 MB)\nCollecting stix-edh==1.0.0\n Using cached stix_edh-1.0.0-py2.py3-none-any.whl (25 kB)\nCollecting libtaxii==1.1.107\n Using cached libtaxii-1.1.107.tar.gz (112 kB)\nCollecting pytz==2015.4\n Using cached pytz-2015.4-py2.py3-none-any.whl (475 kB)\nCollecting certifi\n Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB)\nCollecting ujson==1.34\n Using cached ujson-1.34.tar.gz (188 kB)\nCollecting filelock==2.0.4\n Using cached filelock-2.0.4.tar.gz (5.0 kB)\nCollecting sleekxmpp==1.3.1\n Using cached sleekxmpp-1.3.1.tar.gz (831 kB)\nCollecting beautifulsoup4==4.4.1\n Using cached beautifulsoup4-4.4.1-py2-none-any.whl (81 kB)\nCollecting cifsdk==2.0.14\n Using cached cifsdk-2.0.14.tar.gz (30 kB)\nCollecting lz4==2.2.1\n Using cached lz4-2.2.1-cp27-cp27mu-manylinux1_x86_64.whl (395 kB)\nCollecting networkx==1.11\n Using cached networkx-1.11-py2.py3-none-any.whl (1.3 MB)\nCollecting unicodecsv==0.14.1\n Using cached unicodecsv-0.14.1.tar.gz (10 kB)\nCollecting Werkzeug==0.12.2\n Using cached Werkzeug-0.12.2-py2.py3-none-any.whl (312 kB)\nCollecting pyzmq==16.0.3\n Using cached pyzmq-16.0.3-cp27-cp27mu-manylinux1_x86_64.whl (3.0 MB)\nCollecting stix2-patterns==1.1.0\n Using cached stix2_patterns-1.1.0-py2.py3-none-any.whl (35 kB)\nCollecting idna<2.8,>=2.5\n Using cached idna-2.7-py2.py3-none-any.whl (58 kB)\nCollecting urllib3<1.25,>=1.21.1\n Using cached urllib3-1.24.3-py2.py3-none-any.whl (118 kB)\nCollecting chardet<3.1.0,>=3.0.2\n Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)\nCollecting mixbox>=1.0.1\n Using cached mixbox-1.0.5-py2.py3-none-any.whl (47 kB)\nCollecting python-dateutil\n Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)\nCollecting pytricia>=0.9.0\n Using cached pytricia-1.0.2.tar.gz (20 kB)\nCollecting prettytable>=0.7.2\n Using cached prettytable-1.0.1-py2.py3-none-any.whl (22 kB)\nCollecting arrow>=0.15.2\n Using cached arrow-0.17.0-py2.py3-none-any.whl (50 kB)\nCollecting pytest>=2.7.0\n Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)\nCollecting ipaddress>=1.0.16\n Using cached ipaddress-1.0.23-py2.py3-none-any.whl (18 kB)\nCollecting future\n Using cached future-0.18.2.tar.gz (829 kB)\nCollecting decorator>=3.4.0\n Using cached decorator-4.4.2-py2.py3-none-any.whl (9.2 kB)\nCollecting antlr4-python2-runtime>=4.7; python_version < \"3\"\n Using cached antlr4-python2-runtime-4.10.1.tar.gz (113 kB)\n\n:stderr: DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.\n ERROR: Command errored out with exit status 1:\n command: /opt/minemeld/engine/current/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '\"'\"'/tmp/pip-install-OAQVhq/antlr4-python2-runtime/setup.py'\"'\"'; file='\"'\"'/tmp/pip-install-OAQVhq/antlr4-python2-runtime/setup.py'\"'\"';f=getattr(tokenize, '\"'\"'open'\"'\"', open)(file);code=f.read().replace('\"'\"'\r\n'\"'\"', '\"'\"'\n'\"'\"');f.close();exec(compile(code, file, '\"'\"'exec'\"'\"'))' egg_info --egg-base /tmp/pip-pip-egg-info-Y6C7JJ\n cwd: /tmp/pip-install-OAQVhq/antlr4-python2-runtime/\n Complete output (6 lines):\n Traceback (most recent call last):\n File \"\", line 1, in \n File \"/tmp/pip-install-OAQVhq/antlr4-python2-runtime/setup.py\", line 13\n description=f'ANTLR {v} runtime for Python 2.7.12'\n ^\n SyntaxError: invalid syntax\n ----------------------------------------\nERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.\n"}

PLAY RECAP ***** 127.0.0.1 : ok=32 changed=1 unreachable=0 failed=1 skipped=2 rescued=0 ignored=0

[stevenjw0228001]

Getting error:

sudo python2.7 get-pip.py ERROR: This script does not work on Python 2.7 The minimum supported Python version is 3.7. Please use https://bootstrap.pypa.io/pip/2.7/get-pip.py instead.

[stevenjw0228001]

TASK [minemeld : minemeld virtualenv] **** fatal: [127.0.0.1]: FAILED! => {"changed": true, "cmd": ["virtualenv", "/opt/minemeld/engine/current", "-p", "/usr/bin/python2.7"], "delta": "0:00:00.167742", "end": "2024-01-25 21:00:15.772830", "msg": "non-zero return code", "rc": 1, "start": "2024-01-25 21:00:15.605088", "stderr": "", "stderr_lines": [], "stdout": "RuntimeError: failed to query /usr/bin/python2.7 with code 1 err: ' File \"/usr/local/lib/python3.8/dist-packages/virtualenv/discovery/py_info.py\", line 24\n return list(OrderedDict.fromkeys([\"\", os.environ.get(\"PATHEXT\", \"\").lower().split(os.pathsep)]))\n ^\nSyntaxError: invalid syntax\n'", "stdout_lines": ["RuntimeError: failed to query /usr/bin/python2.7 with code 1 err: ' File \"/usr/local/lib/python3.8/dist-packages/virtualenv/discovery/py_info.py\", line 24\n return list(OrderedDict.fromkeys([\"\", os.environ.get(\"PATHEXT\", \"\").lower().split(os.pathsep)]))\n