cffi compatibility issue in 0.1.36: theHive + magic +

[hollowimage]

building in docker from node:alpine

Requirement already satisfied: pycparser in /usr/lib/python2.7/site-packages/pycparser-2.19-py2.7.egg (from cffi==1.7.0->python-libmagic) (2.19)
elastalert 0.1.36 has requirement cffi>=1.11.5, but you'll have cffi 1.7.0 which is incompatible.
Installing collected packages: cffi, python-libmagic

So i added

pip uninstall cffi --yes && \
pip install cffi>=1.11.5 && \

to my dockerfile, but then

python-libmagic 0.4.0 has requirement cffi==1.7.0, but you'll have cffi 1.11.5 which is incompatible.

It seems like there's a conflict in the requirements.txt, and it looks like it arrived with the addition of theHive stuff. https://github.com/Yelp/elastalert/pull/1889 if i am pulling master.zip

rolling back to https://github.com/Yelp/elastalert/archive/v0.1.35.zip fixes the build problem.

in practice this ends up manifesting as the compiler throwing during initial run

2018-10-25 11:33:01,721 DEBG 'elastalert' stderr output:
/usr/lib/python2.7/site-packages/magic/__pycache__/_cffi__x14a33f6ax54cebdac.c:212:19: fatal error: magic.h: No such file or directory
 #include <magic.h>
                   ^

2018-10-25 11:33:01,722 DEBG 'elastalert' stderr output:
compilation terminated.

2018-10-25 11:33:01,730 DEBG 'elastalert' stderr output:
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/opt/elastalert/elastalert/elastalert.py", line 21, in <module>
    from alerts import DebugAlerter
  File "elastalert/alerts.py", line 31, in <module>
    from thehive4py.api import TheHiveApi
  File "/usr/lib/python2.7/site-packages/thehive4py-1.5.1-py2.7.egg/thehive4py/api.py", line 7, in <module>
    import magic
  File "/usr/lib/python2.7/site-packages/magic/__init__.py", line 1, in <module>
    from . import ffi
  File "/usr/lib/python2.7/site-packages/magic/ffi.py", line 27, in <module>
    ext_package="magic")
  File "/usr/lib/python2.7/site-packages/cffi/api.py", line 451, in verify
    lib = self.verifier.load_library()
  File "/usr/lib/python2.7/site-packages/cffi/verifier.py", line 104, in load_library
    self._compile_module()
  File "/usr/lib/python2.7/site-packages/cffi/verifier.py", line 201, in _compile_module
    outputfilename = ffiplatform.compile(tmpdir, self.get_extension())
  File "/usr/lib/python2.7/site-packages/cffi/ffiplatform.py", line 22, in compile
    outputfilename = _build(tmpdir, ext, compiler_verbose, debug)
  File "/usr/lib/python2.7/site-packages/cffi/ffiplatform.py", line 58, in _build
    raise VerificationError('%s: %s' % (e.__class__.__name__, e))
cffi.error.VerificationError: CompileError: command 'gcc' failed with exit status 1

Here is the rough summary of the dockerfile:

FROM node:alpine

RUN apk update && \
    apk upgrade && \
    apk add --update --no-cache ca-certificates openssl-dev openssl libffi-dev python2 python2-dev py2-pip py2-yaml gcc musl-dev tzdata openntpd bash linux-headers

RUN python setup.py install && \
    pip install --upgrade pip && \
    pip install -e . && \
    pip install awscli && \
    pip install jsonschema && \
    pip uninstall twilio --yes && \
    pip install twilio==6.0.0 && \
    pip uninstall python-libmagic --yes && \
    pip install python-libmagic && \

I figured maybe the last two steps above were not needed, so i tried wtihout them (thinking i was forcing wrong libmagic pull), but then, even though with the requirements.txt

2018-10-25 11:48:33,286 DEBG 'elastalert' stderr output:
Traceback (most recent call last):

2018-10-25 11:48:33,287 DEBG 'elastalert' stderr output:
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main

2018-10-25 11:48:33,287 DEBG 'elastalert' stderr output:

2018-10-25 11:48:33,288 DEBG 'elastalert' stderr output:
"__main__", fname, loader, pkg_name)

2018-10-25 11:48:33,288 DEBG 'elastalert' stderr output:
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code

2018-10-25 11:48:33,288 DEBG 'elastalert' stderr output:

2018-10-25 11:48:33,289 DEBG 'elastalert' stderr output:
exec code in run_globals

2018-10-25 11:48:33,289 DEBG 'elastalert' stderr output:
  File "/opt/elastalert/elastalert/elastalert.py", line 21, in <module>

2018-10-25 11:48:33,290 DEBG 'elastalert' stderr output:

2018-10-25 11:48:33,290 DEBG 'elastalert' stderr output:
from alerts import DebugAlerter

2018-10-25 11:48:33,291 DEBG 'elastalert' stderr output:
  File "elastalert/alerts.py", line 31, in <module>

2018-10-25 11:48:33,291 DEBG 'elastalert' stderr output:

2018-10-25 11:48:33,300 DEBG 'elastalert' stderr output:
from thehive4py.api import TheHiveApi
  File "/usr/lib/python2.7/site-packages/thehive4py-1.5.1-py2.7.egg/thehive4py/api.py", line 7, in <module>
    import magic
  File "/usr/lib/python2.7/site-packages/python_magic-0.4.15-py2.7.egg/magic.py", line 181, in <module>
    raise ImportError('failed to find libmagic.  Check your installation')
ImportError: failed to find libmagic.  Check your installation

rolling back to https://github.com/Yelp/elastalert/archive/v0.1.35.zip works without issue.

[xuing]

I also have this problem.

[xuing]

Check out https://github.com/pidydx/libmagicwin64 for installing libmagic on windows. then I solved the problem.