Trouble getting tests running

[eclipseo]

Hello,

I can't seem to manage to get extractcode and typecode to run the tests.

The whole issue is that part of the README.rst:

To install this package with its full capability (where the binaries for 7zip and libarchive are installed), use the full extra option::

pip install extractcode[full]

If you want to use the version of binaries (possibly) provided by your operating system, use the minimal option::

pip install extractcode

In this case, you will need to provide a working and compatible libarchive and 7zip installed and configured in one of these ways such that ExtractCode can find them:

• a typecode-libarchive and typecode-7z plugin: See the standard ones at https://github.com/nexB/scancode-plugins/tree/main/builtins These can either bundle a libarchive library, a 7z executable or expose a system-installed libraries. It does so by providing plugin entry points as scancode_location_provider for extractcode_libarchive that should point to a LocationProviderPlugin subclass with a get_locations() method that must return a mapping with this key:

  • 'extractcode.libarchive.dll': the absolute path to a libarchive shared object/DLL

  See for example:

  • https://github.com/nexB/scancode-plugins/blob/4da5fe8a5ab1c87b9b4af9e54d7ad60e289747f5/builtins/extractcode_libarchive-linux/setup.py#L40
  • https://github.com/nexB/scancode-plugins/blob/4da5fe8a5ab1c87b9b4af9e54d7ad60e289747f5/builtins/extractcode_libarchive-linux/src/extractcode_libarchive/__init__.py#L17

  And in the same way, the scancode_location_provider for extractcode_7zip should point to a LocationProviderPlugin subclass with a get_locations() method that must return a mapping with this key:

  • 'extractcode.sevenzip.exe': the absolute path to a 7zip executable

  See for example:

  • https://github.com/nexB/scancode-plugins/blob/4da5fe8a5ab1c87b9b4af9e54d7ad60e289747f5/builtins/extractcode_7z-linux/setup.py#L40
  • https://github.com/nexB/scancode-plugins/blob/4da5fe8a5ab1c87b9b4af9e54d7ad60e289747f5/builtins/extractcode_7z-linux/src/extractcode_7z/__init__.py#L18

• use environment variables to point to installed binaries:

  • EXTRACTCODE_LIBARCHIVE_PATH: the absolute path to a libarchive DLL
  • EXTRACTCODE_7Z_PATH: the absolute path to a 7zip executable

• a system-installed libarchive and 7zip executable available in the system PATH.

So I am on a distro with libarchive-3.7.1 and p7zip-16.02 installed. Obviously I don't want to bundle with the full option.

I set up:

export EXTRACTCODE_7Z_PATH=%{_bindir}
export EXTRACTCODE_LIBARCHIVE_PATH_ENVVAR=%{_libdir}
%pytest

It seems that libarchive is detected:

=============================== warnings summary ===============================
../../../../usr/lib/python3.12/site-packages/typecode/magic2.py:195
  /usr/lib/python3.12/site-packages/typecode/magic2.py:195: UserWarning: System libmagic found in typical location is used. Install instead a typecode-libmagic plugin for best support.
    warnings.warn(

src/extractcode/libarchive2.py:107: 12 warnings
  /builddir/build/BUILD/extractcode-31.0.0/src/extractcode/libarchive2.py:107: UserWarning: Using "libarchive" library found in a system location. Install instead a extractcode-libarchive plugin for best support.
    warnings.warn(

(same with libmagic)

However nothing works:

==================================== ERRORS ====================================
_________________ ERROR collecting src/extractcode/archive.py __________________
src/extractcode/archive.py:29: in <module>
    from extractcode import libarchive2
<frozen importlib._bootstrap>:1266: in _find_and_load
    ???
<frozen importlib._bootstrap>:1237: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:841: in _load_unlocked
    ???
/usr/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
src/extractcode/libarchive2.py:635: in <module>
    archive_reader = libarchive.archive_read_new
/usr/lib64/python3.12/ctypes/__init__.py:392: in __getattr__
    func = self.__getitem__(name)
/usr/lib64/python3.12/ctypes/__init__.py:397: in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
E   AttributeError: /usr/bin/python3: undefined symbol: archive_read_new
_________________ ERROR collecting src/extractcode/archive.py __________________
src/extractcode/archive.py:29: in <module>
    from extractcode import libarchive2
<frozen importlib._bootstrap>:1266: in _find_and_load
    ???
<frozen importlib._bootstrap>:1237: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:841: in _load_unlocked
    ???
/usr/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
src/extractcode/libarchive2.py:635: in <module>
    archive_reader = libarchive.archive_read_new
/usr/lib64/python3.12/ctypes/__init__.py:392: in __getattr__
    func = self.__getitem__(name)
/usr/lib64/python3.12/ctypes/__init__.py:397: in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
E   AttributeError: /usr/bin/python3: undefined symbol: archive_read_new
_________________ ERROR collecting src/extractcode/extract.py __________________
src/extractcode/extract.py:23: in <module>
    import extractcode.archive
<frozen importlib._bootstrap>:1266: in _find_and_load
    ???
<frozen importlib._bootstrap>:1237: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:841: in _load_unlocked
    ???
/usr/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
src/extractcode/archive.py:29: in <module>
    from extractcode import libarchive2
<frozen importlib._bootstrap>:1266: in _find_and_load
    ???
<frozen importlib._bootstrap>:1237: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:841: in _load_unlocked
    ???
/usr/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
src/extractcode/libarchive2.py:635: in <module>
    archive_reader = libarchive.archive_read_new
/usr/lib64/python3.12/ctypes/__init__.py:392: in __getattr__
    func = self.__getitem__(name)
/usr/lib64/python3.12/ctypes/__init__.py:397: in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
E   AttributeError: /usr/bin/python3: undefined symbol: archive_read_new
_________________ ERROR collecting src/extractcode/extract.py __________________
src/extractcode/extract.py:23: in <module>
    import extractcode.archive
<frozen importlib._bootstrap>:1266: in _find_and_load
    ???
<frozen importlib._bootstrap>:1237: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:841: in _load_unlocked
    ???
/usr/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
src/extractcode/archive.py:29: in <module>
    from extractcode import libarchive2
<frozen importlib._bootstrap>:1266: in _find_and_load
    ???
<frozen importlib._bootstrap>:1237: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:841: in _load_unlocked
    ???
/usr/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
src/extractcode/libarchive2.py:635: in <module>
    archive_reader = libarchive.archive_read_new
/usr/lib64/python3.12/ctypes/__init__.py:392: in __getattr__
    func = self.__getitem__(name)
/usr/lib64/python3.12/ctypes/__init__.py:397: in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
E   AttributeError: /usr/bin/python3: undefined symbol: archive_read_new
_______________ ERROR collecting src/extractcode/libarchive2.py ________________
src/extractcode/libarchive2.py:635: in <module>
    archive_reader = libarchive.archive_read_new
/usr/lib64/python3.12/ctypes/__init__.py:392: in __getattr__
    func = self.__getitem__(name)
/usr/lib64/python3.12/ctypes/__init__.py:397: in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
E   AttributeError: /usr/bin/python3: undefined symbol: archive_read_new
_______________ ERROR collecting src/extractcode/libarchive2.py ________________
src/extractcode/libarchive2.py:635: in <module>
    archive_reader = libarchive.archive_read_new
/usr/lib64/python3.12/ctypes/__init__.py:392: in __getattr__
    func = self.__getitem__(name)
/usr/lib64/python3.12/ctypes/__init__.py:397: in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
E   AttributeError: /usr/bin/python3: undefined symbol: archive_read_new
____________________ ERROR collecting tests/test_archive.py ____________________
tests/test_archive.py:29: in <module>
    from extractcode import archive
<frozen importlib._bootstrap>:1266: in _find_and_load
    ???
<frozen importlib._bootstrap>:1237: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:841: in _load_unlocked
    ???
/usr/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
src/extractcode/archive.py:29: in <module>
    from extractcode import libarchive2
<frozen importlib._bootstrap>:1266: in _find_and_load
    ???
<frozen importlib._bootstrap>:1237: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:841: in _load_unlocked
    ???
/usr/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
src/extractcode/libarchive2.py:635: in <module>
    archive_reader = libarchive.archive_read_new
/usr/lib64/python3.12/ctypes/__init__.py:392: in __getattr__
    func = self.__getitem__(name)
/usr/lib64/python3.12/ctypes/__init__.py:397: in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
E   AttributeError: /usr/bin/python3: undefined symbol: archive_read_new
____________________ ERROR collecting tests/test_archive.py ____________________
tests/test_archive.py:29: in <module>
    from extractcode import archive
<frozen importlib._bootstrap>:1266: in _find_and_load
    ???
<frozen importlib._bootstrap>:1237: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:841: in _load_unlocked
    ???
/usr/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
src/extractcode/archive.py:29: in <module>
    from extractcode import libarchive2
<frozen importlib._bootstrap>:1266: in _find_and_load
    ???
<frozen importlib._bootstrap>:1237: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:841: in _load_unlocked
    ???
/usr/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
src/extractcode/libarchive2.py:635: in <module>
    archive_reader = libarchive.archive_read_new
/usr/lib64/python3.12/ctypes/__init__.py:392: in __getattr__
    func = self.__getitem__(name)
/usr/lib64/python3.12/ctypes/__init__.py:397: in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
E   AttributeError: /usr/bin/python3: undefined symbol: archive_read_new
____________________ ERROR collecting tests/test_extract.py ____________________
tests/test_extract.py:23: in <module>
    from extractcode import extract
<frozen importlib._bootstrap>:1266: in _find_and_load
    ???
<frozen importlib._bootstrap>:1237: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:841: in _load_unlocked
    ???
/usr/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
src/extractcode/extract.py:23: in <module>
    import extractcode.archive
<frozen importlib._bootstrap>:1266: in _find_and_load
    ???
<frozen importlib._bootstrap>:1237: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:841: in _load_unlocked
    ???
/usr/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
src/extractcode/archive.py:29: in <module>
    from extractcode import libarchive2
<frozen importlib._bootstrap>:1266: in _find_and_load
    ???
<frozen importlib._bootstrap>:1237: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:841: in _load_unlocked
    ???
/usr/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
src/extractcode/libarchive2.py:635: in <module>
    archive_reader = libarchive.archive_read_new
/usr/lib64/python3.12/ctypes/__init__.py:392: in __getattr__
    func = self.__getitem__(name)
/usr/lib64/python3.12/ctypes/__init__.py:397: in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
E   AttributeError: /usr/bin/python3: undefined symbol: archive_read_new
____________________ ERROR collecting tests/test_extract.py ____________________
tests/test_extract.py:23: in <module>
    from extractcode import extract
<frozen importlib._bootstrap>:1266: in _find_and_load
    ???
<frozen importlib._bootstrap>:1237: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:841: in _load_unlocked
    ???
/usr/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
src/extractcode/extract.py:23: in <module>
    import extractcode.archive
<frozen importlib._bootstrap>:1266: in _find_and_load
    ???
<frozen importlib._bootstrap>:1237: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:841: in _load_unlocked
    ???
/usr/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
src/extractcode/archive.py:29: in <module>
    from extractcode import libarchive2
<frozen importlib._bootstrap>:1266: in _find_and_load
    ???
<frozen importlib._bootstrap>:1237: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:841: in _load_unlocked
    ???
/usr/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
src/extractcode/libarchive2.py:635: in <module>
    archive_reader = libarchive.archive_read_new
/usr/lib64/python3.12/ctypes/__init__.py:392: in __getattr__
    func = self.__getitem__(name)
/usr/lib64/python3.12/ctypes/__init__.py:397: in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
E   AttributeError: /usr/bin/python3: undefined symbol: archive_read_new
________________ ERROR collecting tests/test_extractcode_api.py ________________
tests/test_extractcode_api.py:16: in <module>
    from extractcode import extract
<frozen importlib._bootstrap>:1266: in _find_and_load
    ???
<frozen importlib._bootstrap>:1237: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:841: in _load_unlocked
    ???
/usr/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
src/extractcode/extract.py:23: in <module>
    import extractcode.archive
<frozen importlib._bootstrap>:1266: in _find_and_load
    ???
<frozen importlib._bootstrap>:1237: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:841: in _load_unlocked
    ???
/usr/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
src/extractcode/archive.py:29: in <module>
    from extractcode import libarchive2
<frozen importlib._bootstrap>:1266: in _find_and_load
    ???
<frozen importlib._bootstrap>:1237: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:841: in _load_unlocked
    ???
/usr/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
src/extractcode/libarchive2.py:635: in <module>
    archive_reader = libarchive.archive_read_new
/usr/lib64/python3.12/ctypes/__init__.py:392: in __getattr__
    func = self.__getitem__(name)
/usr/lib64/python3.12/ctypes/__init__.py:397: in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
E   AttributeError: /usr/bin/python3: undefined symbol: archive_read_new
________________ ERROR collecting tests/test_extractcode_api.py ________________
tests/test_extractcode_api.py:16: in <module>
    from extractcode import extract
<frozen importlib._bootstrap>:1266: in _find_and_load
    ???
<frozen importlib._bootstrap>:1237: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:841: in _load_unlocked
    ???
/usr/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
src/extractcode/extract.py:23: in <module>
    import extractcode.archive
<frozen importlib._bootstrap>:1266: in _find_and_load
    ???
<frozen importlib._bootstrap>:1237: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:841: in _load_unlocked
    ???
/usr/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
src/extractcode/archive.py:29: in <module>
    from extractcode import libarchive2
<frozen importlib._bootstrap>:1266: in _find_and_load
    ???
<frozen importlib._bootstrap>:1237: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:841: in _load_unlocked
    ???
/usr/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
src/extractcode/libarchive2.py:635: in <module>
    archive_reader = libarchive.archive_read_new
/usr/lib64/python3.12/ctypes/__init__.py:392: in __getattr__
    func = self.__getitem__(name)
/usr/lib64/python3.12/ctypes/__init__.py:397: in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
E   AttributeError: /usr/bin/python3: undefined symbol: archive_read_new

archive_read_new is not found so it doesn't find libarchive I believe.

So I may not have understood the README.rst correctly. Are all the part s following "In this case, you will need to provide a working and compatible libarchive and 7zip installed and configured in one of these ways such that ExtractCode can find them:" mandatory or do I have to choose among the options?

Are the plugins mandatory?

Then If I distribute extractool, does the user have to set up EXTRACTCODE_7Z_PATH and EXTRACTCODE_7Z_PATH each time? Is there a way to avoir that without bundling?

[eclipseo]

I have packaged the "system" plugins and it works now.

[pombredanne]

I have packaged the "system" plugins and it works now.

Awesome! You rock.

I am reopening this though as I still want to make sure we have some tests using the latest versions of these system packages !