Error of misp-modules service after package upgrade

[Tyrell20]

Hello,

after the last upgrade of the misp-modules package to the "misp-modules-2.4.150-1.el7.x86_64" version on RHEL 7.9, the service does not start:

Redirecting to /bin/systemctl status misp-modules.service
● misp-modules.service - MISP enhancement modules
   Loaded: loaded (/etc/systemd/system/misp-modules.service; enabled; vendor preset: disabled)
   Active: failed (Result: signal) since Mon 2021-11-22 12:27:25 CET; 20min ago
  Process: 9669 ExecStart=/var/www/cgi-bin/misp-modules-venv/bin/misp-modules (code=killed, signal=ABRT)
 Main PID: 9669 (code=killed, signal=ABRT)

Nov 22 12:27:25 ws001cip misp-modules[9669]: /lib64/libpython3.6m.so.1.0(+0x15173a)[0x7ffff7a0573a]
Nov 22 12:27:25 ws001cip misp-modules[9669]: /lib64/libpython3.6m.so.1.0(+0x151b53)[0x7ffff7a05b53]
Nov 22 12:27:25 ws001cip misp-modules[9669]: /lib64/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x3a7)[0x7ffff79fa147]
Nov 22 12:27:25 ws001cip misp-modules[9669]: /lib64/libpython3.6m.so.1.0(_PyFunction_FastCallDict+0x152)[0x7ffff7a06c42]
Nov 22 12:27:25 ws001cip misp-modules[9669]: /lib64/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x10e)[0x7ffff795be7e]
Nov 22 12:27:25 ws001cip misp-modules[9669]: /lib64/libpython3.6m.so.1.0(_PyObject_CallMethodIdObjArgs+0xf4)[0x7ffff795d194]
Nov 22 12:27:25 ws001cip misp-modules[9669]: /lib64/libpython3.6m.so.1.0(PyImport_ImportModuleLevelObject+0x23f)[0x7ffff7a1e99f]
Nov 22 12:27:25 ws001cip systemd[1]: misp-modules.service: main process exited, code=killed, status=6/ABRT
Nov 22 12:27:25 ws001cip systemd[1]: Unit misp-modules.service entered failed state.
Nov 22 12:27:25 ws001cip systemd[1]: misp-modules.service failed.

On log file we have:

Nov 22 12:50:21 hostname systemd: Started MISP enhancement modules.
Nov 22 12:50:23 hostname misp-modules: *** Error in `/var/www/cgi-bin/misp-modules-venv/bin/python3': free(): invalid pointer: 0x00007fffd95a5c80 ***
Nov 22 12:50:23 hostname misp-modules: ======= Backtrace: =========
Nov 22 12:50:23 hostname misp-modules: /lib64/libc.so.6(+0x81329)[0x7ffff6c42329]
Nov 22 12:50:23 hostname misp-modules: /lib64/libstdc++.so.6(_ZNSt6locale5_Impl16_M_install_facetEPKNS_2idEPKNS_5facetE+0x142)[0x7fffd930f192]
Nov 22 12:50:23 hostname misp-modules: /lib64/libstdc++.so.6(_ZNSt6locale5_ImplC1Em+0x1e3)[0x7fffd930f5e3]
Nov 22 12:50:23 hostname misp-modules: /lib64/libstdc++.so.6(+0x71555)[0x7fffd9310555]
Nov 22 12:50:23 hostname misp-modules: /lib64/libpthread.so.0(+0x620b)[0x7ffff769e20b]
Nov 22 12:50:23 hostname misp-modules: /lib64/libstdc++.so.6(+0x715a1)[0x7fffd93105a1]
Nov 22 12:50:23 hostname misp-modules: /lib64/libstdc++.so.6(_ZNSt6localeC2Ev+0x13)[0x7fffd93105e3]
Nov 22 12:50:23 hostname misp-modules: /lib64/libstdc++.so.6(_ZNSt8ios_base4InitC2Ev+0xbc)[0x7fffd930d43c]
Nov 22 12:50:23 hostname misp-modules: /var/www/cgi-bin/misp-modules-venv/lib64/python3.6/site-packages/cv2/cv2.cpython-36m-x86_64-linux-gnu.so(+0x2f4b20)[0x7fffdd16cb20]

Could you please help us?

Thank you

[amuehlem]

HI Tyrell20

Looks like this segfault is generated by one of the included modules. Unfortunately it's quite complicated to see which one generates the troubles :-(

I'll have to check with the developers of the misp-modules how we can identify the module generating the troubles

[amuehlem]

I could identify the 3 modules causing the issues.

ocr_enrich.py pdf_enrich.py qrcode.py

disabling these modules resolves the segfault issue

cd /var/www/cgi-bin/misp-modules-venv/lib/python3.6/site-packages/misp_modules-1.0-py3.6.egg/misp_modules/modules/expansion
mv ocr_enrich.py ocr_enrich.py.disabled
mv pdf_enrich.py pdf_enrich.py.disabled
mv qrcode.py qrcode.py.disabled

Now I can start to look for the reason why this modules cause the segfault

[JoePJisc]

Also seeing on a clean install disabling ocr_enrich.py, pdf_enrich.py, and qrcode.py worked for me.

[Tyrell20]

I could identify the 3 modules causing the issues.

ocr_enrich.py pdf_enrich.py qrcode.py

disabling these modules resolves the segfault issue

cd /var/www/cgi-bin/misp-modules-venv/lib/python3.6/site-packages/misp_modules-1.0-py3.6.egg/misp_modules/modules/expansion
mv ocr_enrich.py ocr_enrich.py.disabled
mv pdf_enrich.py pdf_enrich.py.disabled
mv qrcode.py qrcode.py.disabled

Now I can start to look for the reason why this modules cause the segfault

It works for me. Thank you

[amuehlem]

Looks like the problem is cause by the numpy and opencv_python libraries. I could recompile them manually and now all the modules seem to start without the segmentation fault.

I'll have to test a bit more to see how I can compile this libraries and add them to the RPMs.

[amuehlem]

The problem is that some python modules need python > 3.6. I've added a misp-python rpm which provides python 3.9 and like this the modules are now working.

[JoePJisc]

Modules now seem fine for me on a clean install on RHEL8 as well as upgrades on RHEL7.

[alainsilva]

I have the same problem. I use Python 3.9 version.

/var/www/cgi-bin/misp-modules-venv/lib/python3.9/site-packages/misp_modules-1.0-py3.9.egg/misp_modules/modules/expansion [root@myserver expansion]# ls .disa ocr_enrich.py.disabled pdf_enrich.py.disabled qrcode.py.disabled

[root@myserver expansion]# systemctl status misp-modules ● misp-modules.service - MISP enhancement modules Loaded: loaded (/etc/systemd/system/misp-modules.service; enabled; vendor preset: disabled) Active: activating (auto-restart) (Result: core-dump) since Mon 2022-04-04 14:37:29 -03; 11s ago Process: 15964 ExecStart=/var/www/cgi-bin/misp-modules-venv/bin/misp-modules (code=dumped, signal=ABRT) Main PID: 15964 (code=dumped, signal=ABRT)

[amuehlem]

the modules should be disabled in /var/www/cgi-bin/misp-modules-venv/lib/python3.9/site-packages/misp_modules/modules/expansion, not within the /var/www/cgi-bin/misp-modules-venv/lib/python3.9/site-packages/misp_modules-1.0-py3.9.egg/misp_modules/modules/expansion

[amuehlem]

see #64

[alainsilva]

I don't believe man! Works fine! Thanks a lot!

Results:

[root@myserver expansion]# systemctl status misp-modules ● misp-modules.service - MISP enhancement modules Loaded: loaded (/etc/systemd/system/misp-modules.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2022-04-06 11:23:23 -03; 7s ago Main PID: 24109 (misp-modules) CGroup: /system.slice/misp-modules.service └─24109 /var/www/cgi-bin/misp-modules-venv/bin/python3 /var/www/cgi-bin/misp-modules-venv/bin/misp-modules

[JoePJisc]

Still seeing issues with 2.4.159 RHEL7

/var/log/messages is full of lines like:

Jul  7 14:40:36 misp systemd: misp-modules.service holdoff time over, scheduling restart.
Jul  7 14:40:37 misp misp-modules: ERROR:root:No module named '_bz2'
Jul  7 14:40:37 misp misp-modules: Traceback (most recent call last):
Jul  7 14:40:37 misp misp-modules: File "/var/www/cgi-bin/misp-modules-venv/lib/python3.9/site-packages/misp_modules/__init__.py", line 41, in <module>
Jul  7 14:40:37 misp misp-modules: from .modules import *  # noqa
Jul  7 14:40:37 misp misp-modules: File "/var/www/cgi-bin/misp-modules-venv/lib/python3.9/site-packages/misp_modules/modules/__init__.py", line 1, in <module>
Jul  7 14:40:37 misp misp-modules: from .expansion import *  # noqa
Jul  7 14:40:37 misp misp-modules: File "/var/www/cgi-bin/misp-modules-venv/lib/python3.9/site-packages/misp_modules/modules/expansion/virustotal.py", line 3, in <module>
Jul  7 14:40:37 misp misp-modules: import vt
Jul  7 14:40:37 misp misp-modules: File "/var/www/cgi-bin/misp-modules-venv/lib/python3.9/site-packages/vt/__init__.py", line 15, in <module>
Jul  7 14:40:37 misp misp-modules: from .client import *
Jul  7 14:40:37 misp misp-modules: File "/var/www/cgi-bin/misp-modules-venv/lib/python3.9/site-packages/vt/client.py", line 21, in <module>
Jul  7 14:40:37 misp misp-modules: from .feed import Feed
Jul  7 14:40:37 misp misp-modules: File "/var/www/cgi-bin/misp-modules-venv/lib/python3.9/site-packages/vt/feed.py", line 23, in <module>
Jul  7 14:40:37 misp misp-modules: import bz2
Jul  7 14:40:37 misp misp-modules: File "/var/www/cgi-bin/misp-python/lib/python3.9/bz2.py", line 18, in <module>
Jul  7 14:40:37 misp misp-modules: from _bz2 import BZ2Compressor, BZ2Decompressor
Jul  7 14:40:37 misp misp-modules: ModuleNotFoundError: No module named '_bz2'
Jul  7 14:40:37 misp misp-modules: 2022-07-07 14:40:37,126 - misp-modules - INFO - Launch MISP modules server from current directory.
Jul  7 14:40:37 misp misp-modules: INFO:misp-modules:Launch MISP modules server from current directory.
Jul  7 14:40:37 misp misp-modules: 2022-07-07 14:40:37,128 - misp-modules - INFO - Helpers loaded cache.py
Jul  7 14:40:37 misp misp-modules: INFO:misp-modules:Helpers loaded cache.py
Jul  7 14:40:37 misp misp-modules: 2022-07-07 14:40:37,130 - misp-modules - INFO - MISP modules rbl imported
Jul  7 14:40:37 misp misp-modules: INFO:misp-modules:MISP modules rbl imported
Jul  7 14:40:37 misp misp-modules: 2022-07-07 14:40:37,151 - misp-modules - INFO - MISP modules apiosintds imported
Jul  7 14:40:37 misp misp-modules: INFO:misp-modules:MISP modules apiosintds imported
Jul  7 14:40:37 misp misp-modules: 2022-07-07 14:40:37,152 - misp-modules - INFO - MISP modules apivoid imported
Jul  7 14:40:37 misp misp-modules: INFO:misp-modules:MISP modules apivoid imported
Jul  7 14:40:37 misp misp-modules: 2022-07-07 14:40:37,163 - misp-modules - INFO - MISP modules mwdb imported
Jul  7 14:40:37 misp misp-modules: INFO:misp-modules:MISP modules mwdb imported
Jul  7 14:40:37 misp misp-modules: 2022-07-07 14:40:37,208 - misp-modules - INFO - MISP modules assemblyline_query imported
Jul  7 14:40:37 misp misp-modules: INFO:misp-modules:MISP modules assemblyline_query imported
Jul  7 14:40:37 misp misp-modules: *** Error in `/var/www/cgi-bin/misp-modules-venv/bin/python3': free(): invalid pointer: 0x00007fd029688c80 ***
Jul  7 14:40:37 misp misp-modules: ======= Backtrace: =========
...
Jul  7 14:40:37 misp misp-modules: ======= Memory map: ========
...
Jul  7 14:40:37 misp systemd: misp-modules.service: main process exited, code=killed, status=6/ABRT
... Memory map contineus ...
Jul  7 14:40:37 misp systemd: Unit misp-modules.service entered failed state.
... Memory map contineus ...

It would be good if you could add some syslog and logrotate config to the RPM to redirect things like this to their own log files.

[JoePJisc]

Ah, appears the three disabled modules got reenabled somehow, fixed with:

cd /var/www/cgi-bin/misp-modules-venv/lib/python3.9/site-packages/misp_modules/modules/expansion
sudo mv ocr_enrich.py ocr_enrich.py.disabled
sudo mv pdf_enrich.py pdf_enrich.py.disabled
sudo mv qrcode.py qrcode.py.disabled
sudo systemctl restart misp-modules

[amuehlem]

If your using the latest misp-modules it nolonger be needed to deactivate this modules. You should have misp-python, misp-modules and some depencies (misp-*). This packages offer Python39 and the three modules in question should work with the updated python version without generating any coredumps.

[JoePJisc]

Even with the updated misp-python package you just released for #71, I'm seeing this error, looks like bz2 needs to be installed in the misp-modules virtualenv.

error includes: ModuleNotFoundError: No module named '_bz2'

All up to date:

yum list installed | grep misp
faup.x86_64                      1.6-1.el7                       @misp
gtcaca.x86_64                    0+gitb05ed3c-2.el7              @misp
misp.x86_64                      2.4.159-1.el7                   @misp
misp-gcc-libs.x86_64             9.4.0-10.el7                    @misp
misp-gmp-libs.x86_64             6.2.1-7.el7                     @misp
misp-isl-libs.x86_64             0.24-7.el7                      @misp
misp-modules.x86_64              2.4.159-1.el7                   @misp
misp-mpc-libs.x86_64             1.2.1-5.el7                     @misp
misp-mpfr-libs.x86_64            4.1.0-5.el7                     @misp
misp-python.x86_64               3.9.13-4.el7                    @misp
misp-python-virtualenv.x86_64    2.4.159-1.el7                   @misp
misp-release.noarch              1.1-2.el7                       @misp
python-setuptools.noarch         36.6.0-1.el7                    @misp

[amuehlem]

did you restart the misp-modules? I can't see the missing _bz2 module error after upgrading misp-python and restarting the misp-modules.