How to use --extract on Linux-32 ? What extract archive mean?

[K-Rex]

I want to use -c -l to scan, but it scan all the files like .c .h and all, I don't want to scan .svn/ directory. How can I do?

I use : ./scancode -e ../gpl/ I get this error:

Extracting archives... [------------------------------------] 0 Traceback (most recent call last): File "/home/rex/qdu_claro/yuki/tools/scancode-toolkit-1.2.4/bin/scancode", line 9, in load_entry_point('scancode-toolkit==1.2.4', 'console_scripts', 'scancode')() File "/home/rex/qdu_claro/yuki/tools/scancode-toolkit-1.2.4/local/lib/python2.7/site-packages/click/core.py", line 664, in call return self.main(_args, _kwargs) File "/home/rex/qdu_claro/yuki/tools/scancode-toolkit-1.2.4/local/lib/python2.7/site-packages/click/core.py", line 644, in main rv = self.invoke(ctx) File "/home/rex/qdu_claro/yuki/tools/scancode-toolkit-1.2.4/local/lib/python2.7/site-packages/click/core.py", line 837, in invoke return ctx.invoke(self.callback, _ctx.params) File "/home/rex/qdu_claro/yuki/tools/scancode-toolkit-1.2.4/local/lib/python2.7/site-packages/click/core.py", line 464, in invoke return callback(_args, **kwargs) File "/home/rex/qdu_claro/yuki/tools/scancode-toolkit-1.2.4/src/scancode/cli.py", line 258, in scancode extract_with_progress(abs_input, verbose) File "/home/rex/qdu_claro/yuki/tools/scancode-toolkit-1.2.4/src/scancode/cli.py", line 345, in extract_with_progress for xevent in extractions: File "/home/rex/qdu_claro/yuki/tools/scancode-toolkit-1.2.4/local/lib/python2.7/site-packages/click/_termui_impl.py", line 240, in next rv = next(self.iter) File "/home/rex/qdu_claro/yuki/tools/scancode-toolkit-1.2.4/src/scancode/api.py", line 44, in extract_archives from extractcode.extract import extract File "/home/rex/qdu_claro/yuki/tools/scancode-toolkit-1.2.4/src/extractcode/extract.py", line 37, in from extractcode import archive File "/home/rex/qdu_claro/yuki/tools/scancode-toolkit-1.2.4/src/extractcode/archive.py", line 47, in from extractcode import libarchive2 File "/home/rex/qdu_claro/yuki/tools/scancode-toolkit-1.2.4/src/extractcode/libarchive2.py", line 91, in libarchive = load_lib() File "/home/rex/qdu_claro/yuki/tools/scancode-toolkit-1.2.4/src/extractcode/libarchive2.py", line 87, in load_lib raise ImportError('Failed to load libarchive: %(libarchive)r' % locals()) ImportError: Failed to load libarchive: '/home/rex/qdu_claro/yuki/scancode-toolkit-1.2.4/src/extractcode/bin/linux-32/bin/libarchive.so'

There is no libarchive.so in that directory!!! Can anyone help me? Thx...

[pombredanne]

@K-Rex first, the pre-built binaries have not been built for 32-bites Linux... Do you have access to a 64 bits Linux installation instead?

I created a new ticket to ignore .svn dirs and similar ass #34

[pombredanne]

@K-Rex which exact Linux distro do you run?

[K-Rex]

My Linux environment is : Ubuntu 12.04 i386 GNU/Linux.

Can you tell me what -e command do?

I try to use it on WIN7 64bit. It generate housekeeper.a-extract directory (included file and platform_conf.o)

But I can't understand what it do. Thx

[pombredanne]

@K-Rex Thanks for the Linux version info.

The -e or --extract option looks in the input directory and extracts any archive or compressed files found there, such as a zip file, a tarball (or in your example a static library ar archive).

In your example above, housekeeper.a is an archive, so scancode -e extracted the housekeeper.a archive to the housekeeper.a-extract directory where you can find the files that are contained in the original housekeeper.a archive, such as platform_conf.o and posssibly other files.

If you do not have archives in the directory you want to scan or you would prefer to extract these by hand before a scan, you do not need the -e/--extract option.

[pombredanne]

@K-Rex see also scancode --examples for --extract usage examples such as:

Extract all archives found in the 'samples' directory tree:

    scancode --extract samples

Note: If an archive contains other archives, all contained archives will be
extracted recursively. Extraction is done directly in the 'samples' directory,
side-by-side with each archive. Files are extracted in a directory named after
the archive with an '-extract' suffix added to its name, created side-by-side
with the corresponding archive file.

Extract a single archive. Files are extracted in the directory
'samples/arch/zlib.tar.gz-extract/':

    scancode --extract samples/arch/zlib.tar.gz

[pombredanne]

@K-Rex the new https://github.com/nexB/scancode-toolkit/releases/tag/v1.3.0 has experimental binaries for Linux 32 bits and improved help for the --extract option. I will make this an official release when I have you testing feedback

[K-Rex]

I download "scancode-toolkit-1.3.0.tar.bz2" and install it, but it still can't work for Linux 32 bits.

error: ImportError: Failed to load libarchive: '/home/rex/qdu_claro/yuki/tools/scancode-toolkit-1.3.0/src/extractcode/bin/linux-32/bin/libarchive.so'

[pombredanne]

@K-Rex This was another silly un-tested mistake... (I unfortunately have no 32 bits test machine for now). Please try again with the latest from develop here: https://github.com/nexB/scancode-toolkit/archive/develop.tar.gz

[K-Rex]

Uh.. It still can't work on Linux-32

./scancode -e ../../qdu_claro/yuki/tools/gpl/

Traceback (most recent call last): File "/home/rex/test/scancode-toolkit-develop/bin/scancode", line 9, in load_entry_point('scancode-toolkit==1.3.0', 'console_scripts', 'scancode')() File "/home/rex/test/scancode-toolkit-develop/local/lib/python2.7/site-packages/click/core.py", line 664, in call return self.main(_args, _kwargs) File "/home/rex/test/scancode-toolkit-develop/src/scancode/cli.py", line 230, in main standalone_mode=standalone_mode, _extra) File "/home/rex/test/scancode-toolkit-develop/local/lib/python2.7/site-packages/click/core.py", line 644, in main rv = self.invoke(ctx) File "/home/rex/test/scancode-toolkit-develop/local/lib/python2.7/site-packages/click/core.py", line 837, in invoke return ctx.invoke(self.callback, _ctx.params) File "/home/rex/test/scancode-toolkit-develop/local/lib/python2.7/site-packages/click/core.py", line 464, in invoke return callback(_args, _kwargs) File "/home/rex/test/scancode-toolkit-develop/src/scancode/cli.py", line 270, in scancode extract_with_progress(abs_input, verbose) File "/home/rex/test/scancode-toolkit-develop/src/scancode/cli.py", line 360, in extract_with_progress for xevent in extractions: File "/home/rex/test/scancode-toolkit-develop/local/lib/python2.7/site-packages/click/_termui_impl.py", line 240, in next rv = next(self.iter) File "/home/rex/test/scancode-toolkit-develop/src/scancode/api.py", line 44, in extract_archives from extractcode.extract import extract File "/home/rex/test/scancode-toolkit-develop/src/extractcode/extract.py", line 37, in from extractcode import archive File "/home/rex/test/scancode-toolkit-develop/src/extractcode/archive.py", line 47, in from extractcode import libarchive2 File "/home/rex/test/scancode-toolkit-develop/src/extractcode/libarchive2.py", line 91, in libarchive = load_lib() File "/home/rex/test/scancode-toolkit-develop/src/extractcode/libarchive2.py", line 84, in load_lib lib = ctypes.CDLL(libarchive) File "/usr/lib/python2.7/ctypes/init.py", line 365, in init self._handle = _dlopen(self._name, mode)

OSError: liblzma.so.1: cannot open shared object file: No such file or directory

[pombredanne]

@K-Rex You need to install a Linux pre-requisite package that may not be present on Ubuntu 12.04.

Use this:

sudo apt-get update
sudo apt-get install xz-utils

[pombredanne]

FYI, xz-utils provides liblzma

[K-Rex]

My OS had already install xz-utils

$sudo apt-get install xz-utils Reading package lists... Done Building dependency tree Reading state information... Done xz-utils is already the newest version. You might want to run 'apt-get -f install' to correct these: The following packages have unmet dependencies: bison : Depends: m4 but it is not installable Depends: libbison-dev (= 1:2.5.dfsg-2.1) but it is not installable gettext : Depends: libunistring0 but it is not installable Depends: libgettextpo0 (= 0.18.1.1-5ubuntu3) but it is not installable vim-tiny : Depends: vim-common (= 2:7.3.429-2ubuntu2.1) but 2:7.3.429-2ubuntu2 is to be installed E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).

It still has no liblzma.so.1 QQ

[pombredanne]

@K-Rex: ok, I will install a VM to test run extactly on your OS

[K-Rex]

I try to use VM to install Ubuntu 12.04 and test develop.tar.gz and latest version 1.3.1

Error Message: /scancode-toolkit-develop$ ./scancode --help

• Configuring ScanCode ...
• Configuring Python ... Traceback (most recent call last): File "", line 1, in UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 10: ordinal not in range(128) Traceback (most recent call last): File "thirdparty/base/virtualenv.py", line 2363, in main() File "thirdparty/base/virtualenv.py", line 832, in main symlink=options.symlink) File "thirdparty/base/virtualenv.py", line 1004, in create_environment install_wheel(to_install, py_executable, search_dirs) File "thirdparty/base/virtualenv.py", line 969, in install_wheel 'PIP_NO_INDEX': '1' File "thirdparty/base/virtualenv.py", line 910, in call_subprocess % (cmd_desc, proc.returncode)) OSError

Failed to execute command: /usr/bin/python2.7 thirdparty/base/virtualenv.py --never-download --quiet --extra-search-dir=thirdparty/dev --extra-search-dir=thirdparty/base --extra-search-dir=thirdparty/prod /home/aaa/Download/scancode-toolkit-1.3.1/scancode-toolkit-develop ./scancode: line 17: /home/aaa/Download/scancode-toolkit-1.3.1/scancode-toolkit-develop/bin/scancode: No file and no directory.

Is that because PIP?

My environment is : Linux virtual-machine 3.13.0-32-generic #57~precise1-Ubuntu SMP Tue Jul 15 03:50:54 UTC 2014 i686 i686 i386 GNU/Linux

[pombredanne]

@K-Rex since you are trying a VM, can you try to install a 64 bits VM instead? again the 32 bits support is really something experimental I tried for you

[pombredanne]

If possible I would like to focus on 64 bits instead

[pombredanne]

The tests are all passing fine on Ubuntu 12 64 bits FWIW: See https://travis-ci.org/nexB/scancode-toolkit/branches

[K-Rex]

Well, maybe I don't need -e command. It's ok. Thx.

[pombredanne]

@K-Rex I made a last attempt to rebuild libarchive on 32 bits, on Ubuntu 12. Please try again using the latest from develop: https://github.com/nexB/scancode-toolkit/archive/develop.tar.gz

If this does not work, I will instead likely provide a better error handling and disable entirely 32 bits support

[pombredanne]

But if you are using Linux in a VM, please use a 64 bits VM, rather than 32 bits

[K-Rex]

Hey, it can work on Linux-32 bit and VM-32bits, the error is because the $LANG. I change it to en_US.UTF-8, then it can install successfully. And --verbose and -e command can work too. Thank you very much, maybe you can release that version for Linux-32 bits!

[pombredanne]

@K-Rex Thank you for the test!. I am not entirely surprise by that... because the last error you had was really something in a core python tool called pip.