Installing with pip fails because of ssdeep

[JohannesTK]

Installing through pip (sudo pip install thug) fails because installing ssdeep fails.

Env: Ubuntu 14.04.5 LTS x64.

pip.log for thug: ` Downloading/unpacking pymongo==3.3.0 (from thug) Downloading pymongo-3.3.0.tar.gz (494kB): 494kB downloaded Running setup.py (path:/tmp/pip_build_root/pymongo/setup.py) egg_info for package pymongo Downloading/unpacking ssdeep==3.1.1 (from thug) Downloading ssdeep-3.1.1.tar.gz (385kB): 385kB downloaded Running setup.py (path:/tmp/pip_build_root/ssdeep/setup.py) egg_info for package ssdeep

Installed /tmp/pip_build_root/ssdeep/.eggs/cffi-1.9.1-py2.7-linux-x86_64.egg
Searching for pycparser
Reading https://pypi.python.org/simple/pycparser/
Downloading https://pypi.python.org/packages/be/64/1bb257ffb17d01f4a38d7ce686809a736837ad4371bcc5c42ba7a715c3ac/pycparser-2.17.tar.gz#md5=ca98dcb50bc1276f230118f6af5a40c7
Best match: pycparser 2.17
Processing pycparser-2.17.tar.gz
Writing /tmp/easy_install-y3Opwr/pycparser-2.17/setup.cfg
Running pycparser-2.17/setup.py -q bdist_egg --dist-dir /tmp/easy_install-y3Opwr/pycparser-2.17/egg-dist-tmp-YzeKd6
warning: no previously-included files matching 'yacctab.*' found under directory 'tests'
warning: no previously-included files matching 'lextab.*' found under directory 'tests'
warning: no previously-included files matching 'yacctab.*' found under directory 'examples'
warning: no previously-included files matching 'lextab.*' found under directory 'examples'
zip_safe flag not set; analyzing archive contents...
pycparser.ply.yacc: module references __file__
pycparser.ply.yacc: module MAY be using inspect.getsourcefile
pycparser.ply.yacc: module MAY be using inspect.stack
pycparser.ply.ygen: module references __file__
pycparser.ply.lex: module references __file__
pycparser.ply.lex: module MAY be using inspect.getsourcefile
Extracting pycparser-2.17-py2.7.egg to /tmp/pip_build_root/ssdeep/.eggs

Installed /tmp/pip_build_root/ssdeep/.eggs/pycparser-2.17-py2.7.egg
ssdeep/__pycache__/_ssdeep_cffi_8a9054b9x627c7d55.c:209:23: fatal error: fuzzy.h: No such file or directory
     #include "fuzzy.h"
                       ^
compilation terminated.
Traceback (most recent call last):
  File "<string>", line 17, in <module>
  File "/tmp/pip_build_root/ssdeep/setup.py", line 123, in <module>
    ext_package="ssdeep",
  File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "<string>", line 15, in replacement_run
  File "build/bdist.linux-x86_64/egg/setuptools/command/egg_info.py", line 306, in find_sources
  File "build/bdist.linux-x86_64/egg/setuptools/command/egg_info.py", line 533, in run
  File "build/bdist.linux-x86_64/egg/setuptools/command/egg_info.py", line 569, in add_defaults
  File "build/bdist.linux-x86_64/egg/setuptools/command/py36compat.py", line 34, in add_defaults
  File "build/bdist.linux-x86_64/egg/setuptools/command/sdist.py", line 133, in _add_defaults_python
  File "/usr/lib/python2.7/distutils/cmd.py", line 312, in get_finalized_command
    cmd_obj.ensure_finalized()
  File "/usr/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
    self.finalize_options()
  File "build/bdist.linux-x86_64/egg/setuptools/command/build_py.py", line 34, in finalize_options
  File "/usr/lib/python2.7/distutils/command/build_py.py", line 46, in finalize_options
    ('force', 'force'))
  File "/usr/lib/python2.7/distutils/cmd.py", line 298, in set_undefined_options
    src_cmd_obj.ensure_finalized()
  File "/usr/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
    self.finalize_options()
  File "/tmp/pip_build_root/ssdeep/setup.py", line 18, in finalize_options
    self.distribution.ext_modules = get_ext_modules()
  File "/tmp/pip_build_root/ssdeep/setup.py", line 59, in get_ext_modules
    binding.verify()
  File "ssdeep/binding.py", line 125, in verify
    libraries=self._libraries,
  File "/tmp/pip_build_root/ssdeep/.eggs/cffi-1.9.1-py2.7-linux-x86_64.egg/cffi/api.py", line 437, in verify
    lib = self.verifier.load_library()
  File "/tmp/pip_build_root/ssdeep/.eggs/cffi-1.9.1-py2.7-linux-x86_64.egg/cffi/verifier.py", line 113, in load_library
    self._compile_module()
  File "/tmp/pip_build_root/ssdeep/.eggs/cffi-1.9.1-py2.7-linux-x86_64.egg/cffi/verifier.py", line 210, in _compile_module
    outputfilename = ffiplatform.compile(tmpdir, self.get_extension())
  File "/tmp/pip_build_root/ssdeep/.eggs/cffi-1.9.1-py2.7-linux-x86_64.egg/cffi/ffiplatform.py", line 29, in compile
    outputfilename = _build(tmpdir, ext, compiler_verbose, debug)
  File "/tmp/pip_build_root/ssdeep/.eggs/cffi-1.9.1-py2.7-linux-x86_64.egg/cffi/ffiplatform.py", line 65, in _build
    raise VerificationError('%s: %s' % (e.__class__.__name__, e))
cffi.ffiplatform.VerificationError: CompileError: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Complete output from command python setup.py egg_info:

Installed /tmp/pip_build_root/ssdeep/.eggs/cffi-1.9.1-py2.7-linux-x86_64.egg

Searching for pycparser

Reading https://pypi.python.org/simple/pycparser/

Downloading https://pypi.python.org/packages/be/64/1bb257ffb17d01f4a38d7ce686809a736837ad4371bcc5c42ba7a715c3ac/pycparser-2.17.tar.gz#md5=ca98dcb50bc1276f230118f6af5a40c7

Best match: pycparser 2.17

Processing pycparser-2.17.tar.gz

Writing /tmp/easy_install-y3Opwr/pycparser-2.17/setup.cfg

Running pycparser-2.17/setup.py -q bdist_egg --dist-dir /tmp/easy_install-y3Opwr/pycparser-2.17/egg-dist-tmp-YzeKd6

warning: no previously-included files matching 'yacctab.*' found under directory 'tests'

warning: no previously-included files matching 'lextab.*' found under directory 'tests'

warning: no previously-included files matching 'yacctab.*' found under directory 'examples'

warning: no previously-included files matching 'lextab.*' found under directory 'examples'

zip_safe flag not set; analyzing archive contents...

pycparser.ply.yacc: module references file

pycparser.ply.yacc: module MAY be using inspect.getsourcefile

pycparser.ply.yacc: module MAY be using inspect.stack

pycparser.ply.ygen: module references file

pycparser.ply.lex: module references file

pycparser.ply.lex: module MAY be using inspect.getsourcefile

creating /tmp/pip_build_root/ssdeep/.eggs/pycparser-2.17-py2.7.egg

Extracting pycparser-2.17-py2.7.egg to /tmp/pip_build_root/ssdeep/.eggs

Installed /tmp/pip_build_root/ssdeep/.eggs/pycparser-2.17-py2.7.egg

running egg_info

creating pip-egg-info/ssdeep.egg-info

writing requirements to pip-egg-info/ssdeep.egg-info/requires.txt

writing pip-egg-info/ssdeep.egg-info/PKG-INFO

writing top-level names to pip-egg-info/ssdeep.egg-info/top_level.txt

writing dependency_links to pip-egg-info/ssdeep.egg-info/dependency_links.txt

writing manifest file 'pip-egg-info/ssdeep.egg-info/SOURCES.txt'

ssdeep/pycache/_ssdeep_cffi_8a9054b9x627c7d55.c:209:23: fatal error: fuzzy.h: No such file or directory

 #include "fuzzy.h"

                   ^

compilation terminated.

Traceback (most recent call last):

File "", line 17, in

File "/tmp/pip_build_root/ssdeep/setup.py", line 123, in

ext_package="ssdeep",

File "/usr/lib/python2.7/distutils/core.py", line 151, in setup

dist.run_commands()

File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands

self.run_command(cmd)

File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command

cmd_obj.run()

File "", line 15, in replacement_run

File "build/bdist.linux-x86_64/egg/setuptools/command/egg_info.py", line 306, in find_sources

File "build/bdist.linux-x86_64/egg/setuptools/command/egg_info.py", line 533, in run

File "build/bdist.linux-x86_64/egg/setuptools/command/egg_info.py", line 569, in add_defaults

File "build/bdist.linux-x86_64/egg/setuptools/command/py36compat.py", line 34, in add_defaults

File "build/bdist.linux-x86_64/egg/setuptools/command/sdist.py", line 133, in _add_defaults_python

File "/usr/lib/python2.7/distutils/cmd.py", line 312, in get_finalized_command

cmd_obj.ensure_finalized()

File "/usr/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized

self.finalize_options()

File "build/bdist.linux-x86_64/egg/setuptools/command/build_py.py", line 34, in finalize_options

File "/usr/lib/python2.7/distutils/command/build_py.py", line 46, in finalize_options

('force', 'force'))

File "/usr/lib/python2.7/distutils/cmd.py", line 298, in set_undefined_options

src_cmd_obj.ensure_finalized()

File "/usr/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized

self.finalize_options()

File "/tmp/pip_build_root/ssdeep/setup.py", line 18, in finalize_options

self.distribution.ext_modules = get_ext_modules()

File "/tmp/pip_build_root/ssdeep/setup.py", line 59, in get_ext_modules

binding.verify()

File "ssdeep/binding.py", line 125, in verify

libraries=self._libraries,

File "/tmp/pip_build_root/ssdeep/.eggs/cffi-1.9.1-py2.7-linux-x86_64.egg/cffi/api.py", line 437, in verify

lib = self.verifier.load_library()

File "/tmp/pip_build_root/ssdeep/.eggs/cffi-1.9.1-py2.7-linux-x86_64.egg/cffi/verifier.py", line 113, in load_library

self._compile_module()

File "/tmp/pip_build_root/ssdeep/.eggs/cffi-1.9.1-py2.7-linux-x86_64.egg/cffi/verifier.py", line 210, in _compile_module

outputfilename = ffiplatform.compile(tmpdir, self.get_extension())

File "/tmp/pip_build_root/ssdeep/.eggs/cffi-1.9.1-py2.7-linux-x86_64.egg/cffi/ffiplatform.py", line 29, in compile

outputfilename = _build(tmpdir, ext, compiler_verbose, debug)

File "/tmp/pip_build_root/ssdeep/.eggs/cffi-1.9.1-py2.7-linux-x86_64.egg/cffi/ffiplatform.py", line 65, in _build

raise VerificationError('%s: %s' % (e.__class__.__name__, e))

cffi.ffiplatform.VerificationError: CompileError: command 'x86_64-linux-gnu-gcc' failed with exit status 1

---

Cleaning up... Command python setup.py egg_info failed with error code 1 in /tmp/pip_build_root/ssdeep Storing debug log for failure in /home/vagrant/.pip/pip.log `

Trying: pip install ssdeep will produce the same output.

It is because the build will fail if the ssdeep library isn't installed. The issue can be fixed with rather ease. From python-ssdeep Github: $ BUILD_LIB=1 pip install ssdeep (https://github.com/DinoTools/python-ssdeep)

[buffer]

Well setup already cares to properly set BUILD_LIB. I am not a Ubuntu user but it seems like you miss libfuzzy-dev. Please take a look at

https://github.com/buffer/thug/blob/master/docker/Dockerfile

and double check if all the packages listed at lines 36-53 are installed on your system and then retry.

[JohannesTK]

Ack. Got it. Thanks for the quick reply and the great work!

[buffer]

Thanks for the notification. Please let me know it that solves the issue.

[JohannesTK]

Yes, got it solved. An updated version of Thug bootstrap script can be found from: https://github.com/PayloadSecurity/VxCommunity/blob/master/bash/thuginstallation.sh

Cheers!