Fall back issue for lzo/lz4

[OlafHaalstra]

During the upgrade of dissect I get the following error:

Building wheels for collected packages: python-lzo
  Building wheel for python-lzo (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for python-lzo (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [9 lines of output]
      /private/var/folders/70/r3109jrs19vgs2lyzrh6v9840000gp/T/pip-build-env-ymr_bx26/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py:261: UserWarning: Unknown distribution option: 'tests_require'
        warnings.warn(msg)
      lzomodule.c:38:10: fatal error: 'lzo/lzo1.h' file not found
      #include <lzo/lzo1.h>
               ^~~~~~~~~~~~
      1 error generated.
      LZO_DIR: /private/var/folders/70/r3109jrs19vgs2lyzrh6v9840000gp/T/pip-install-9ugxz0a2/python-lzo_f569e17600514119b8325cb4790965fa/lzo-2.10
      include dir: /private/var/folders/70/r3109jrs19vgs2lyzrh6v9840000gp/T/pip-install-9ugxz0a2/python-lzo_f569e17600514119b8325cb4790965fa/lzo-2.10/include
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for python-lzo
Failed to build python-lzo
ERROR: Could not build wheels for python-lzo, which is required to install pyproject.toml-based projects

A) Does the fallback not work? B) What is the recommended approach to circumvent this issue? I assume having the C version available provides optimization / speedup?

[Horofic]

Hey @OlafHaalstra thank you for raising this issue! Internally we got some similar reports. With regards to your questions:

A) The fallback to our own pure-python implementation should still work, since this is taken account of in their respective projects (dissect.btrfs, dissect.squashfs, dissect.util).

B) For now, installing the liblzo2-dev or lzo-devel package manually through your package manager should do the trick.

[pyrco]

Hey @OlafHaalstra, what OS / cpu architecture / python version are you using? For most a pre-build binary wheel should be available, so I'm curious what the combination of things is that runs into this issue.

[pyrco]

Hey @OlafHaalstra, I think I found the underlying reason for your issues. The pyhton-lzo project releases binary wheels for most OS/architecture/Python version combinations, however during their latest release the wheels for all the linux/pyhton combinations faild to build and were never published.

What we are going to do is release a minor version of dissect (3.16.1) where the lz4 and lzo dependencies are moved to extras so they won't be installed by default. See #68.

However, if you want the speed-ups that these extras provide, do follow the pointers in @Horofic's comment at the start of this thread.

[pyrco]

As dissect 3.16.1 is now released, your issue should be fixed. If you run into any other problems regarding lzo/lz4 feel free to reopen this ticket or create a new one.