Building against libarchive on Windows

voltagex commented 1 year ago

Windows 10, VS2022 build tools are installed, Python 3.10.10

> pip install python-libarchive
Collecting python-libarchive
  Using cached python-libarchive-4.2.1.tar.gz (58 kB)
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: python-libarchive
  Building wheel for python-libarchive (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [13 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build\lib.win32-cpython-310
      creating build\lib.win32-cpython-310\libarchive
      copying libarchive\tar.py -> build\lib.win32-cpython-310\libarchive
      copying libarchive\zip.py -> build\lib.win32-cpython-310\libarchive
      copying libarchive\_libarchive.py -> build\lib.win32-cpython-310\libarchive
      copying libarchive\__init__.py -> build\lib.win32-cpython-310\libarchive
      running build_ext
      building 'libarchive.__libarchive' extension
      error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for python-libarchive
  Running setup.py clean for python-libarchive
Failed to build python-libarchive
Installing collected packages: python-libarchive
  Running setup.py install for python-libarchive ... error
  error: subprocess-exited-with-error

  × Running setup.py install for python-libarchive did not run successfully.
  │ exit code: 1
  ╰─> [15 lines of output]
      running install
      C:\git\archive-test\venv\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build\lib.win32-cpython-310
      creating build\lib.win32-cpython-310\libarchive
      copying libarchive\tar.py -> build\lib.win32-cpython-310\libarchive
      copying libarchive\zip.py -> build\lib.win32-cpython-310\libarchive
      copying libarchive\_libarchive.py -> build\lib.win32-cpython-310\libarchive
      copying libarchive\__init__.py -> build\lib.win32-cpython-310\libarchive
      running build_ext
      building 'libarchive.__libarchive' extension
      error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> python-libarchive

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

voltagex commented 1 year ago

In case anyone else stumbles upon this.

The error message directing you to https://visualstudio.microsoft.com/visual-cpp-build-tools/ is either wrong, bad or incomplete.

setup.py build_ext is running "C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe" -latest -prerelease -requiresAny -requires Microsoft.VisualStudio.Component.VC.Tools.x86.x64 -property installationPath -products *

which wasn't returning anything even though the "Visual Studio Build Tools" were supposedly installed (hell, I had a developer command prompt and everything)

https://learn.microsoft.com/en-us/visualstudio/install/workload-component-id-vs-build-tools?view=vs-2022 shows that at the time of writing, you'll want to go to the VS installer and install Microsoft.VisualStudio.Component.VC.Tools.x86.x64 or more likely, tick the box that says MSVC v143 - VS 2022 C++ x64/x86 build tools (Latest)

This looks like it will not fix this particular issue with libarchive, as it needs io.h, which appears to be a non-portable header from Unix. Which is strange, as Microsoft ship something libarchive-shaped in tar.exe on Windows 10.

voltagex commented 1 year ago

Got stuck in hell. io.h is in the ucrt, but the generated C doesn't seem to be compatible with MSVC.

C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(3977): error C2065: 'mode_t': undeclared identifier
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(3977): error C2146: syntax error: missing ';' before identifier 'result'
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(3977): error C2065: 'result': undeclared identifier
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(3986): error C2065: 'result': undeclared identifier
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(3987): error C2065: 'mode_t': undeclared identifier
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(3987): error C2059: syntax error: ')'
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(4296): error C2065: 'mode_t': undeclared identifier
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(4296): error C2146: syntax error: missing ';' before identifier 'result'
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(4296): error C2065: 'result': undeclared identifier
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(4305): error C2065: 'result': undeclared identifier
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(4306): error C2065: 'mode_t': undeclared identifier
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(4306): error C2059: syntax error: ')'
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(4482): error C2065: 'mode_t': undeclared identifier
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(4482): error C2146: syntax error: missing ';' before identifier 'result'
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(4482): error C2065: 'result': undeclared identifier
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(4491): error C2065: 'result': undeclared identifier
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(4492): error C2065: 'mode_t': undeclared identifier
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(4492): error C2059: syntax error: ')'
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(5945): error C2065: 'mode_t': undeclared identifier
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(5945): error C2146: syntax error: missing ';' before identifier 'arg2'
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(5945): error C2065: 'arg2': undeclared identifier
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(5966): error C2065: 'arg2': undeclared identifier
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(5966): error C2065: 'mode_t': undeclared identifier
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(5966): error C2059: syntax error: ')'
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(5969): error C2065: 'arg2': undeclared identifier
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(6231): error C2065: 'mode_t': undeclared identifier
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(6231): error C2146: syntax error: missing ';' before identifier 'arg2'
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(6231): error C2065: 'arg2': undeclared identifier
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(6252): error C2065: 'arg2': undeclared identifier
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(6252): error C2065: 'mode_t': undeclared identifier
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(6252): error C2059: syntax error: ')'
C:\git\archive-test\python-libarchive\libarchive\_libarchive_wrap.c(6255): error C2065: 'arg2': undeclared identifier
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2022\\BuildTools\\VC\\Tools\\MSVC\\14.35.32215\\bin\\HostX86\\x86\\cl.exe' failed with exit code 2

libarchive is able to be compiled on Windows, but I'm not the one to be able to fix this.

voltagex commented 1 year ago

I can get as far as libarchive loading now, but attempting to use it results in

Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "C:\git\archive-test\venv\lib\site-packages\python_libarchive-4.2.1-py3.10-win-amd64.egg\libarchive\__init__.py", line 46, in <module>
    None: (_libarchive.archive_read_support_format_all, None),
AttributeError: module 'libarchive._libarchive' has no attribute 'archive_read_support_format_all'

0001-Hack-for-Windows-support.patch is a very messy hack around setup.py build_ext.

I also regenerated the SWIG bindings and upgraded libarchive, which is probably what's causing me issues.

To get to this point:

Install the build tools, including MSVC v143 - VS 2022 C++ x64/x86 build tools (Latest) as above.
Install the latest Windows 10/11 SDK (for ucrt)
Install vcpkg, and then vcpkg install libarchive:x64-windows
Build the Python extension - I think I got it working by dropping archive.lib in the same path setup.py was running from
Copy all DLLs from vcpkg\installed\x64-windows\bin (at least archive.dll and its dependencies) to site-packages\python_libarchive-4.2.1-py3.10-win-amd64.egg\libarchive (I hope you're using a virtualenv)
- I am assuming there's a proper way to do this in setup.py but I don't know it
- import libarchive

voltagex commented 1 year ago

0001-Slightly-better-hack-for-Windows.patch

This time I didn't accidentally remove half the functions.

smartfile / python-libarchive

Building against libarchive on Windows #38