Installation Ubuntu

[kevthehermit]

Ubuntu Server 14.04

OS - Ubuntu 14.04 Server Install SSDeep from source

thehermit@viper:~/viper$ ssdeep -V
2.11

thehermit@viper:~/viper$ sudo pip install pydeep
Downloading/unpacking pydeep
  Downloading pydeep-0.2.tar.gz
  Running setup.py (path:/tmp/pip_build_root/pydeep/setup.py) egg_info for package pydeep

Installing collected packages: pydeep
  Running setup.py install for pydeep
    building 'pydeep' extension
    x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c pydeep.c -o build/temp.linux-x86_64-2.7/pydeep.o
    pydeep.c: In function ‘pydeep_hash_buf’:
    pydeep.c:52:15: warning: variable ‘inputStringBuffer’ set but not used [-Wunused-but-set-variable]
         PyObject *inputStringBuffer = NULL;
                   ^
    x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/pydeep.o -lfuzzy -o build/lib.linux-x86_64-2.7/pydeep.so
    /usr/bin/ld: //usr/local/lib/libfuzzy.a(fuzzy.o): relocation R_X86_64_32S against `.rodata.str1.8' can not be used when making a shared object; recompile with -fPIC
    //usr/local/lib/libfuzzy.a: error adding symbols: Bad value
    collect2: error: ld returned 1 exit status
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip_build_root/pydeep/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-g9iFHO-record/install-record.txt --single-version-externally-managed --compile:
    running install

running build

running build_ext

building 'pydeep' extension

creating build

creating build/temp.linux-x86_64-2.7

x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c pydeep.c -o build/temp.linux-x86_64-2.7/pydeep.o

pydeep.c: In function ‘pydeep_hash_buf’:

pydeep.c:52:15: warning: variable ‘inputStringBuffer’ set but not used [-Wunused-but-set-variable]

     PyObject *inputStringBuffer = NULL;

               ^

creating build/lib.linux-x86_64-2.7

x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/pydeep.o -lfuzzy -o build/lib.linux-x86_64-2.7/pydeep.so

/usr/bin/ld: //usr/local/lib/libfuzzy.a(fuzzy.o): relocation R_X86_64_32S against `.rodata.str1.8' can not be used when making a shared object; recompile with -fPIC

//usr/local/lib/libfuzzy.a: error adding symbols: Bad value

collect2: error: ld returned 1 exit status

error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

----------------------------------------
Cleaning up...
Command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip_build_root/pydeep/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-g9iFHO-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /tmp/pip_build_root/pydeep
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    load_entry_point('pip==1.5.4', 'console_scripts', 'pip')()
  File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 185, in main
    return command.main(cmd_args)
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 161, in main
    text = '\n'.join(complete_log)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 26: ordinal not in range(128)
thehermit@viper:~/viper$

Install from GitHub

thehermit@viper:~/tmp_build/pydeep$ sudo python setup.py install
running install
running build
running build_ext
building 'pydeep' extension
creating build
creating build/temp.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c pydeep.c -o build/temp.linux-x86_64-2.7/pydeep.o
pydeep.c: In function ‘pydeep_hash_buf’:
pydeep.c:52:15: warning: variable ‘inputStringBuffer’ set but not used [-Wunused-but-set-variable]
     PyObject *inputStringBuffer = NULL;
               ^
creating build/lib.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/pydeep.o -lfuzzy -o build/lib.linux-x86_64-2.7/pydeep.so
/usr/bin/ld: //usr/local/lib/libfuzzy.a(fuzzy.o): relocation R_X86_64_32S against `.rodata.str1.8' can not be used when making a shared object; recompile with -fPIC
//usr/local/lib/libfuzzy.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

[kevthehermit]

Rolled ssdeep back to version 2.10 then installed pydeep via pip. - Errors the same but completes install and seems to be useble.

thehermit@viper:~/tmp_build/ssdeep-2.10$ ssdeep -V
2.10
thehermit@viper:~/tmp_build/ssdeep-2.10$ sudo pip install pydeep
Downloading/unpacking pydeep
  Downloading pydeep-0.2.tar.gz
  Running setup.py (path:/tmp/pip_build_root/pydeep/setup.py) egg_info for package pydeep

Installing collected packages: pydeep
  Running setup.py install for pydeep
    building 'pydeep' extension
    x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c pydeep.c -o build/temp.linux-x86_64-2.7/pydeep.o
    pydeep.c: In function ‘pydeep_hash_buf’:
    pydeep.c:52:15: warning: variable ‘inputStringBuffer’ set but not used [-Wunused-but-set-variable]
         PyObject *inputStringBuffer = NULL;
                   ^
    x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/pydeep.o -lfuzzy -o build/lib.linux-x86_64-2.7/pydeep.so

Successfully installed pydeep
Cleaning up...

[kbandla]

@kevthehermit thanks for posting this issue.

Detail

This issue is very specific to ssdeep-2.11. Previous versions of ssdeep used to build static and shared libraries. ssdeep-2.11 only builds static libraries. pydeep was not able to find the shared library when building. The author of ssdeep was made aware of this issue and will soon update ssdeep's Makefile to build shared libraries as well. (Thanks Jesse!)

Avoid reverting to ssdeep-2.10

Reverting back to ssdeep-2.10 should be avoided. ssdeep-2.10 had a bug in the signature generation code. If you have to revert, I recommend using ssdeep-2.9 instead.

Workaround

Till ssdeep-2.11 source is updated, you can try this workaround (from Jesse).

In ssdeep-2.11/Makefile.am, remove the '-static' flag from libfuzzy_la_LDFLAGS in :

-libfuzzy_la_LDFLAGS=-static -no-undefined -version-info 2:0:0
+libfuzzy_la_LDFLAGS=-no-undefined -version-info 2:0:0

After that, run:

autoreconf -fvi

Then do

./configure
make
sudo make install

After that, proceed building and installing pydeep as usual.

I will leave this issue open till there is an update to ssdeep.

[kevthehermit]

Thank you for the continued support of this project :) And for the workaround

Have you considered travis for automated testing - http://docs.travis-ci.com/user/getting-started/

a sample for ssdeep on the viper project - https://github.com/botherder/viper/blob/master/.travis.yml

[kbandla]

Fixed

ssdeep-2.11.1 was released last night. It now builds shared libraries. Please update to it. pydeep should work fine with ssdeep-2.11-1

I will surely look into travis-ci. Thanks!