search

remg427 / misp42splunk

A Splunk app to use MISP in background
GNU Lesser General Public License v3.0
109 stars 30 forks source link

MISP 4.0.1 setup failure #192

Closed ag-michael closed 3 years ago

ag-michael commented 3 years ago

On Splunk 7.3.x (specifically 7.3.0), after MISP42 is installed,the configuration page loops. this is correlated with the following splunkd.log error:

02-23-2021 17:24:24.997 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':  ImportError: No module named lib2to3.pgen2.parse
02-23-2021 17:24:24.997 -0600 ERROR AdminManagerExternal - External handler failed with code '1' and output: ''.  See splunkd.log for stderr output.
02-23-2021 17:24:24.997 -0600 ERROR AdminManagerExternal - External handler failed with code '1' and output: ''.  See splunkd.log for stderr output.
02-23-2021 17:24:24.997 -0600 ERROR AdminManagerExternal - External handler failed with code '1' and output: ''.  See splunkd.log for stderr output.
02-23-2021 17:24:24.997 -0600 ERROR AdminManagerExternal - External handler failed with code '1' and output: ''.  See splunkd.log for stderr output.
02-23-2021 17:24:25.006 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':  Traceback (most recent call last):
02-23-2021 17:24:25.006 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':    File "/Applications/Splunk/bin/runScript.py", line 78, in <module>
02-23-2021 17:24:25.007 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':      execfile(REAL_SCRIPT_NAME)
02-23-2021 17:24:25.007 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':    File "/Applications/Splunk/etc/apps/misp42splunk/bin/misp42splunk_rh_settings.py", line 4, in <module>
02-23-2021 17:24:25.007 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':      from splunktaucclib.rest_handler.endpoint import (
02-23-2021 17:24:25.008 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':    File "/Applications/Splunk/etc/apps/misp42splunk/lib/aob_py2/splunktaucclib/rest_handler/endpoint/validator.py", line 8, in <module>
02-23-2021 17:24:25.008 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':      from past.builtins import basestring
02-23-2021 17:24:25.009 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':    File "/Applications/Splunk/lib/python2.7/site-packages/past/__init__.py", line 88, in <module>
02-23-2021 17:24:25.009 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':      from past.translation import install_hooks as autotranslate
02-23-2021 17:24:25.009 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':    File "/Applications/Splunk/lib/python2.7/site-packages/past/translation/__init__.py", line 41, in <module>
02-23-2021 17:24:25.010 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':      from lib2to3.pgen2.parse import ParseError
02-23-2021 17:24:25.010 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':  ImportError: No module named lib2to3.pgen2.parse

This was run on a text OS X box, but an identical error is seen on a Linux development server as well.

remg427 commented 3 years ago

Hi, some libraries are missing under aob_py2 and aob_py3 (cleaning of default package was too strong on your env) could you test package under version 4.0.2 and tell me if everything is fixed?

ag-michael commented 3 years ago

@remg427 I'm getting the exact same error after removing 4.0.1 and installing 4.0.2 (confirmed by looking at the app manifest, downloaded from https://raw.githubusercontent.com/remg427/misp42splunk/4.0.2/misp42splunk.tar.gz). Was there a change in python versions (upgrade to 3.x) when moving from misp42 3.x.x to 4.x.x?

EDIT: --version on /Applications/Splunk/bin/python in my splunk shows '2.7.15' for my python version.

remg427 commented 3 years ago

from version 3.2.3 to 3.3.0+ I removed almost all libraries shipped with AddOnBuilder because of warnings on AppInspect. so I guess some other libraries are still missing in your env. Could you extract message from splunkd to see where is the issue now (in first log extract, past folder was missing) ?

EDIT- or I create another version with all lib back

ag-michael commented 3 years ago

@remg427 It looks like the same error to me:

02-24-2021 17:00:00.226 -0600 INFO  ExecProcessor - setting reschedule_ms=3599775, for command=python /Applications/Splunk/etc/apps/splunk_instrumentation/bin/instrumentation.py
02-24-2021 17:48:11.685 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':  Traceback (most recent call last):
02-24-2021 17:48:11.687 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':    File "/Applications/Splunk/bin/runScript.py", line 78, in <module>
02-24-2021 17:48:11.688 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':      execfile(REAL_SCRIPT_NAME)
02-24-2021 17:48:11.689 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':    File "/Applications/Splunk/etc/apps/misp42splunk/bin/misp42splunk_rh_settings.py", line 4, in <module>
02-24-2021 17:48:11.689 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':      from splunktaucclib.rest_handler.endpoint import (
02-24-2021 17:48:11.689 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':    File "/Applications/Splunk/etc/apps/misp42splunk/lib/aob_py2/splunktaucclib/rest_handler/endpoint/validator.py", line 8, in <module>
02-24-2021 17:48:11.690 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':      from past.builtins import basestring
02-24-2021 17:48:11.692 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':    File "/Applications/Splunk/etc/apps/misp42splunk/lib/aob_py2/past/__init__.py", line 88, in <module>
02-24-2021 17:48:11.694 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':      from past.translation import install_hooks as autotranslate
02-24-2021 17:48:11.694 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':    File "/Applications/Splunk/etc/apps/misp42splunk/lib/aob_py2/past/translation/__init__.py", line 41, in <module>
02-24-2021 17:48:11.695 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':      from lib2to3.pgen2.parse import ParseError
02-24-2021 17:48:11.696 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':  ImportError: No module named lib2to3.pgen2.parse
02-24-2021 17:48:11.697 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':  Traceback (most recent call last):
02-24-2021 17:48:11.698 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':    File "/Applications/Splunk/bin/runScript.py", line 78, in <module>
02-24-2021 17:48:11.699 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':      execfile(REAL_SCRIPT_NAME)
02-24-2021 17:48:11.702 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':    File "/Applications/Splunk/etc/apps/misp42splunk/bin/misp42splunk_rh_settings.py", line 4, in <module>
02-24-2021 17:48:11.703 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':      from splunktaucclib.rest_handler.endpoint import (
02-24-2021 17:48:11.703 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':    File "/Applications/Splunk/etc/apps/misp42splunk/lib/aob_py2/splunktaucclib/rest_handler/endpoint/validator.py", line 8, in <module>
02-24-2021 17:48:11.704 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':      from past.builtins import basestring
02-24-2021 17:48:11.705 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':    File "/Applications/Splunk/etc/apps/misp42splunk/lib/aob_py2/past/__init__.py", line 88, in <module>
02-24-2021 17:48:11.706 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':      from past.translation import install_hooks as autotranslate
02-24-2021 17:48:11.707 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':    File "/Applications/Splunk/etc/apps/misp42splunk/lib/aob_py2/past/translation/__init__.py", line 41, in <module>
02-24-2021 17:48:11.708 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':      from lib2to3.pgen2.parse import ParseError
02-24-2021 17:48:11.710 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':  ImportError: No module named lib2to3.pgen2.parse
02-24-2021 17:48:11.711 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':  Traceback (most recent call last):
02-24-2021 17:48:11.712 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':    File "/Applications/Splunk/bin/runScript.py", line 78, in <module>
02-24-2021 17:48:11.713 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':      execfile(REAL_SCRIPT_NAME)
02-24-2021 17:48:11.714 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':    File "/Applications/Splunk/etc/apps/misp42splunk/bin/misp42splunk_rh_settings.py", line 4, in <module>
02-24-2021 17:48:11.714 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':      from splunktaucclib.rest_handler.endpoint import (
02-24-2021 17:48:11.715 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':    File "/Applications/Splunk/etc/apps/misp42splunk/lib/aob_py2/splunktaucclib/rest_handler/endpoint/validator.py", line 8, in <module>
02-24-2021 17:48:11.716 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':      from past.builtins import basestring
02-24-2021 17:48:11.716 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':    File "/Applications/Splunk/etc/apps/misp42splunk/lib/aob_py2/past/__init__.py", line 88, in <module>
02-24-2021 17:48:11.717 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':      from past.translation import install_hooks as autotranslate
02-24-2021 17:48:11.718 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':    File "/Applications/Splunk/etc/apps/misp42splunk/lib/aob_py2/past/translation/__init__.py", line 41, in <module>
02-24-2021 17:48:11.718 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':      from lib2to3.pgen2.parse import ParseError
02-24-2021 17:48:11.719 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':  ImportError: No module named lib2to3.pgen2.parse
02-24-2021 17:48:11.729 -0600 ERROR AdminManagerExternal - External handler failed with code '1' and output: ''.  See splunkd.log for stderr output.
02-24-2021 17:48:11.729 -0600 ERROR AdminManagerExternal - External handler failed with code '1' and output: ''.  See splunkd.log for stderr output.
02-24-2021 17:48:11.735 -0600 ERROR AdminManagerExternal - External handler failed with code '1' and output: ''.  See splunkd.log for stderr output.
02-24-2021 17:48:11.741 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':  Traceback (most recent call last):
02-24-2021 17:48:11.742 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':    File "/Applications/Splunk/bin/runScript.py", line 78, in <module>
02-24-2021 17:48:11.742 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':      execfile(REAL_SCRIPT_NAME)
02-24-2021 17:48:11.743 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':    File "/Applications/Splunk/etc/apps/misp42splunk/bin/misp42splunk_rh_instances.py", line 4, in <module>
02-24-2021 17:48:11.744 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':      from splunktaucclib.rest_handler.endpoint import (
02-24-2021 17:48:11.744 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':    File "/Applications/Splunk/etc/apps/misp42splunk/lib/aob_py2/splunktaucclib/rest_handler/endpoint/validator.py", line 8, in <module>
02-24-2021 17:48:11.745 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':      from past.builtins import basestring
02-24-2021 17:48:11.746 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':    File "/Applications/Splunk/etc/apps/misp42splunk/lib/aob_py2/past/__init__.py", line 88, in <module>
02-24-2021 17:48:11.746 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':      from past.translation import install_hooks as autotranslate
02-24-2021 17:48:11.747 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':    File "/Applications/Splunk/etc/apps/misp42splunk/lib/aob_py2/past/translation/__init__.py", line 41, in <module>
02-24-2021 17:48:11.747 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':      from lib2to3.pgen2.parse import ParseError
02-24-2021 17:48:11.748 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':  ImportError: No module named lib2to3.pgen2.parse
02-24-2021 17:48:11.749 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':  Traceback (most recent call last):
02-24-2021 17:48:11.750 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':    File "/Applications/Splunk/bin/runScript.py", line 78, in <module>
02-24-2021 17:48:11.750 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':      execfile(REAL_SCRIPT_NAME)
02-24-2021 17:48:11.751 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':    File "/Applications/Splunk/etc/apps/misp42splunk/bin/misp42splunk_rh_settings.py", line 4, in <module>
02-24-2021 17:48:11.752 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':      from splunktaucclib.rest_handler.endpoint import (
02-24-2021 17:48:11.753 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':    File "/Applications/Splunk/etc/apps/misp42splunk/lib/aob_py2/splunktaucclib/rest_handler/endpoint/validator.py", line 8, in <module>
02-24-2021 17:48:11.753 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':      from past.builtins import basestring
02-24-2021 17:48:11.754 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':    File "/Applications/Splunk/etc/apps/misp42splunk/lib/aob_py2/past/__init__.py", line 88, in <module>
02-24-2021 17:48:11.754 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':      from past.translation import install_hooks as autotranslate
02-24-2021 17:48:11.755 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':    File "/Applications/Splunk/etc/apps/misp42splunk/lib/aob_py2/past/translation/__init__.py", line 41, in <module>
02-24-2021 17:48:11.756 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':      from lib2to3.pgen2.parse import ParseError
02-24-2021 17:48:11.757 -0600 ERROR ScriptRunner - stderr from '/Applications/Splunk/bin/python /Applications/Splunk/bin/runScript.py setup':  ImportError: No module named lib2to3.pgen2.parse
02-24-2021 17:48:11.758 -0600 ERROR AdminManagerExternal - External handler failed with code '1' and output: ''.  See splunkd.log for stderr output.
02-24-2021 17:48:11.764 -0600 ERROR AdminManagerExternal - External handler failed with code '1' and output: ''.  See splunkd.log for stderr output.
ag-michael commented 3 years ago

@remg427 I solved the problem! I downloaded lib2to3 from cpython's 2.7.15 branch and put Lib/lib2to3 under misp42splunk/lib and the config page loads, and it lets me define the MISP server. Would it be possible to include this library and have it be used for Splunk 7.x?

I think the problem is that somehow Splunk managed to ship their python interpreter without lib2to3,because even on 2.7.x python, I can see this library is included.

I have only tested this on a Splunk instance I put together to help resolve this specific issue, I still need a published app to test in our development environment.

remg427 commented 3 years ago

Hi thanks will do over weekend Do you need version on splunkbase or git is enough?

-- Sent with K-9 Mail.

ag-michael commented 3 years ago

@remg427 Splunk base would be best, thank you!

ag-michael commented 3 years ago

@remg427 any update on this issue?

remg427 commented 3 years ago

Overlooked appinspect report now 4.0.2 is visible -- Sent with K-9 Mail.

ag-michael commented 3 years ago

@remg427 , Thank you so much, I've tested and implemented this.