Python package building rework

Antelox commented 2 weeks ago

Your checklist for this pull request

[ ] I've documented or updated the documentation of every API function and struct this PR changes.
[ ] I've added tests that prove my fix is effective or that my feature works (if possible)

Detailed description

Refactored setup.py to remove hacks regarding packaging of wheels for different platforms, improve and cleanup the code
Updated README.txt
Removed old Makefile and build_wheel.sh scripts
Created a new workflow that takes care of building and testing python packages for different platforms/architectures/python versions
Fixed cstest_py positional argument
Fixed root path detection in integration_tests.py
Added run_tests.py script to run all the python unittests
Added options to specify build type and threads to be used during building
Refreshed the list of files to be copied into src folder for sdist archive

Test plan

There are 6 tests failing for both Linux and Windows x86 arch.

INFO  - Run test: TestInput { arch: mos65xx, options: ['CS_OPT_DETAIL', 'CS_MODE_MOS65XX_W65C02', 'CS_OPT_SYNTAX_MOTOROLA'], addr: 1000, bytes: [ 0x07,0x12,0x27,0x12,0x47,0x12,0x67,0x12,0x87,0x12,0xa7,0x12,0xc7,0x12,0xe7,0x12,0x10,0xfe,0x0f,0x12,0xfd,0x4f,0x12,0xfd,0x8f,0x12,0xfd,0xcf,0x12,0xfd ] }
WARNING - CS_ARCH: Capstone doesn't have the attribute 'mos65xx'
ERROR - Normalized asm-text doesn't match:
decoded:  'bbr0 $12, (null)f0fb60df'
expected: 'bbr0 $12, $1012'

INFO  - TestResult.FAILED

INFO  - Run test: TestInput { arch: mos65xx, options: ['CS_OPT_DETAIL', 'CS_MODE_MOS65XX_65816_LONG_MX', 'CS_OPT_SYNTAX_MOTOROLA'], addr: 1000, bytes: [ 0xa9,0x34,0x12,0xad,0x34,0x12,0xbd,0x34,0x12,0xb9,0x34,0x12,0xaf,0x56,0x34,0x12,0xbf,0x56,0x34,0x12,0xa5,0x12,0xb5,0x12,0xb2,0x12,0xa1,0x12,0xb1,0x12,0xa7,0x12,0xb7,0x12,0xa3,0x12,0xb3,0x12,0xc2,0x00,0xe2,0x00,0x54,0x34,0x12,0x44,0x34,0x12,0x02,0x12 ] }
ERROR - Normalized asm-text doesn't match:
decoded:  'mvn $12, (null)f0fb60df'
expected: 'mvn $12, $34'

INFO  - TestResult.FAILED

INFO  - Run test: TestInput { arch: CS_ARCH_ARM, options: ['CS_MODE_THUMB', 'CS_OPT_DETAIL'], addr: 0, bytes: [ 0x18,0xf8,0x03,0x1e ] }
ERROR - regs_write_count: 2 != 1
INFO  - TestResult.FAILED

INFO  - Run test: TestInput { arch: CS_ARCH_ARM, options: ['CS_MODE_THUMB', 'CS_MODE_MCLASS', 'CS_OPT_DETAIL'], addr: 0, bytes: [ 0x98,0xfc,0x4e,0x08 ] }
ERROR - imm: 4029476313 != 90
INFO  - TestResult.FAILED

INFO  - Run test: TestInput { arch: CS_ARCH_ARM, options: ['CS_MODE_THUMB', 'CS_MODE_V8', 'CS_OPT_DETAIL'], addr: 0, bytes: [ 0x94,0xfd,0x46,0x48 ] }
ERROR - imm: 4029476493 != 270
INFO  - TestResult.FAILED

Closing issues

I suppose it should close:

Rot127 commented 1 week ago

The failing tests are possibly related to https://github.com/capstone-engine/capstone/issues/2416. Both are clearly 32bit system issues.

Rot127 commented 6 days ago

@kabeor This PR opens the question if Capstone supports x86 systems. The tests fail on 32bit systems (at least Windows). I think it is a good idea to support it. Because there are also modern 32bit CPU architectures out there. So having at least a single x86 job is good reduces the chance of bugs also on the other archtectures.

Rot127 commented 6 days ago

Something is still wrong with the naming: https://github.com/capstone-engine/capstone/actions/runs/11813575581/job/32923353685?pr=2538

Antelox commented 6 days ago

Something is still wrong with the naming: https://github.com/capstone-engine/capstone/actions/runs/11813575581/job/32923353685?pr=2538

That is ugly I know, but it's like that because that job is skipped hence the matrix doesn't get created and finally the placeholders don't get replaced

Rot127 commented 6 days ago

   Successfully installed cstest_py-0.1.0 pyyaml-6.0.2
  INFO:tests:Running python D:\a\capstone\capstone/bindings/python/tests/test_all.py
  INFO:tests:#######################
  Traceback (most recent call last):
    File "D:\a\capstone\capstone/bindings/python/tests/test_all.py", line 3, in <module>
      import test_lite
    File "D:\a\capstone\capstone\bindings\python\tests\test_lite.py", line 4, in <module>
      from capstone import *
  ModuleNotFoundError: No module named 'capstone'
  Traceback (most recent call last):
    File "D:\a\capstone\capstone/suite/run_tests.py", line 23, in <module>
      subprocess.run(test.split(" "), check=True)
    File "C:\Users\runneradmin\AppData\Local\pypa\cibuildwheel\Cache\nuget-cpython\python.3.8.10\tools\lib\subprocess.py", line 516, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['python', 'D:\\a\\capstone\\capstone/bindings/python/tests/test_all.py']' returned non-zero exit status 1.
  Error: Command python -m pip install D:\a\capstone\capstone\bindings\python/cstest_py && python D:\a\capstone\capstone/suite/run_tests.py
   failed with code 1.

Rot127 commented 6 days ago

This one is weird: https://github.com/capstone-engine/capstone/actions/runs/11818841771/job/32927697780?pr=2538 Happens also for this job. Let's see after the review of the others if it is fixed.

Antelox commented 6 days ago

This one is weird: https://github.com/capstone-engine/capstone/actions/runs/11818841771/job/32927697780?pr=2538 Happens also for this job. Let's see after the review of the others if it is fixed.

Should be internal GitHub networking issue

Antelox commented 5 days ago

I can't see if this is also the case for all architectures (like Linux + AArch64, Windows + AArch64 or Windows i686). But if all those are free, we should absolutely enable some more runners.

Those are not native runners so it's not the case. Here you can check what are the specs of the runners. Basically all are x64 plus macos-latest one that is AppleSilicon (ARM).

Rot127 commented 5 days ago

@kabeor The run in case of tag creation: https://github.com/Antelox/capstone/actions/runs/11839185045

Rot127 commented 5 days ago

Binaries look good as well:

for f in $(ls ./*/capstone/lib/*); do file $f; done
./capstone-6.0.0a1-cp310-cp310-macosx_10_9_universal2/capstone/lib/libcapstone.dylib: Mach-O universal binary with 2 architectures: [x86_64:\012- Mach-O 64-bit x86_64 dynamically linked shared library, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|NO_REEXPORTED_DYLIBS>] [\012- arm64]
./capstone-6.0.0a1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64/capstone/lib/libcapstone.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, BuildID[sha1]=d139511e718f699bc4ff70698f6e5cd7d2c9d26d, not stripped
./capstone-6.0.0a1-cp310-cp310-win_amd64/capstone/lib/capstone.dll: PE32+ executable (DLL) (GUI) x86-64, for MS Windows, 6 sections
./capstone-6.0.0a1-cp310-cp310-win_arm64/capstone/lib/capstone.dll: PE32+ executable (DLL) (GUI) Aarch64, for MS Windows, 6 sections
./capstone-6.0.0a1-cp39-cp39-macosx_10_9_x86_64/capstone/lib/libcapstone.dylib: Mach-O 64-bit x86_64 dynamically linked shared library, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|NO_REEXPORTED_DYLIBS>
./capstone-6.0.0a1-cp39-cp39-macosx_11_0_arm64/capstone/lib/libcapstone.dylib: Mach-O 64-bit arm64 dynamically linked shared library, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|NO_REEXPORTED_DYLIBS>
./capstone-6.0.0a1-cp39-cp39-musllinux_1_2_x86_64/capstone/lib/libcapstone.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=fd9af6ba25a55df236d41a04d6d4776f7b585cab, with debug_info, not stripped
./capstone-6.0.0a1-cp39-cp39-win_amd64/capstone/lib/capstone.dll: PE32+ executable (DLL) (GUI) x86-64, for MS Windows, 6 sections

Rot127 commented 5 days ago

The sdist build fails though (on Linux). Please remove the issue from the list above or fix it.

Processing ./capstone-6.0.0a1.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: capstone
  Building wheel for capstone (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for capstone (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [69 lines of output]
      /tmp/pip-build-env-jwlqop42/overlay/lib/python3.12/site-packages/setuptools/dist.py:330: InformationOnly: Normalizing '6.0.0alpha1' to '6.0.0a1'
        self.metadata.version = self._normalize_version(self.metadata.version)
      INFO:root:running bdist_wheel
      INFO:root:running build
      INFO:root:running build_py
      INFO:__main__:Building C extensions
      INFO:__main__:Build Directory: /tmp/pip-req-build-dqzt6b7j/src/build_python

      -- The C compiler identification is GNU 14.2.1
      -- The CXX compiler identification is GNU 14.2.1
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Performing Test SUPPORTS_MU
      -- Performing Test SUPPORTS_MU - Success
      -- Performing Test SUPPORTS_SHADOWING
      -- Performing Test SUPPORTS_SHADOWING - Success
      -- Performing Test SUPPORTS_SUNINIT
      -- Performing Test SUPPORTS_SUNINIT - Failed
      Enabling CAPSTONE_ARM_SUPPORT
      Enabling CAPSTONE_AARCH64_SUPPORT
      Enabling CAPSTONE_M68K_SUPPORT
      Enabling CAPSTONE_MIPS_SUPPORT
      Enabling CAPSTONE_PPC_SUPPORT
      Enabling CAPSTONE_SPARC_SUPPORT
      Enabling CAPSTONE_SYSTEMZ_SUPPORT
      Enabling CAPSTONE_XCORE_SUPPORT
      Enabling CAPSTONE_X86_SUPPORT
      Enabling CAPSTONE_TMS320C64X_SUPPORT
      Enabling CAPSTONE_M680X_SUPPORT
      Enabling CAPSTONE_EVM_SUPPORT
      Enabling CAPSTONE_MOS65XX_SUPPORT
      Enabling CAPSTONE_WASM_SUPPORT
      Enabling CAPSTONE_BPF_SUPPORT
      Enabling CAPSTONE_RISCV_SUPPORT
      Enabling CAPSTONE_SH_SUPPORT
      Enabling CAPSTONE_TRICORE_SUPPORT
      Enabling CAPSTONE_ALPHA_SUPPORT
      Enabling CAPSTONE_HPPA_SUPPORT
      Enabling CAPSTONE_LOONGARCH_SUPPORT
      Enabling CAPSTONE_XTENSA_SUPPORT
      CMake Error: File /tmp/pip-req-build-dqzt6b7j/src/capstone.pc.in does not exist.
      CMake Error at CMakeLists.txt:879 (configure_file):
        configure_file Problem configuring file

      CMake Error: File /tmp/pip-req-build-dqzt6b7j/src/capstone-config.cmake.in does not exist.
      CMake Error at /usr/share/cmake/Modules/CMakePackageConfigHelpers.cmake:501 (configure_file):
        configure_file Problem configuring file
      Call Stack (most recent call first):
        CMakeLists.txt:884 (configure_package_config_file)

      CMake Error: File /tmp/pip-req-build-dqzt6b7j/src/cmake_uninstall.cmake.in does not exist.
      CMake Error at CMakeLists.txt:924 (configure_file):
        configure_file Problem configuring file

      -- Configuring incomplete, errors occurred!
      gmake: Makefile: No such file or directory
      gmake: *** No rule to make target 'Makefile'.  Stop.
      error: [Errno 2] No such file or directory: 'libcapstone.so.6'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for capstone
Failed to build capstone
ERROR: Could not build wheels for capstone, which is required to install pyproject.toml-based projects

Antelox commented 5 days ago

The sdist build fails though (on Linux). Please remove the issue from the list above or fix it.

Processing ./capstone-6.0.0a1.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: capstone
  Building wheel for capstone (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for capstone (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [69 lines of output]
      /tmp/pip-build-env-jwlqop42/overlay/lib/python3.12/site-packages/setuptools/dist.py:330: InformationOnly: Normalizing '6.0.0alpha1' to '6.0.0a1'
        self.metadata.version = self._normalize_version(self.metadata.version)
      INFO:root:running bdist_wheel
      INFO:root:running build
      INFO:root:running build_py
      INFO:__main__:Building C extensions
      INFO:__main__:Build Directory: /tmp/pip-req-build-dqzt6b7j/src/build_python

      -- The C compiler identification is GNU 14.2.1
      -- The CXX compiler identification is GNU 14.2.1
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Performing Test SUPPORTS_MU
      -- Performing Test SUPPORTS_MU - Success
      -- Performing Test SUPPORTS_SHADOWING
      -- Performing Test SUPPORTS_SHADOWING - Success
      -- Performing Test SUPPORTS_SUNINIT
      -- Performing Test SUPPORTS_SUNINIT - Failed
      Enabling CAPSTONE_ARM_SUPPORT
      Enabling CAPSTONE_AARCH64_SUPPORT
      Enabling CAPSTONE_M68K_SUPPORT
      Enabling CAPSTONE_MIPS_SUPPORT
      Enabling CAPSTONE_PPC_SUPPORT
      Enabling CAPSTONE_SPARC_SUPPORT
      Enabling CAPSTONE_SYSTEMZ_SUPPORT
      Enabling CAPSTONE_XCORE_SUPPORT
      Enabling CAPSTONE_X86_SUPPORT
      Enabling CAPSTONE_TMS320C64X_SUPPORT
      Enabling CAPSTONE_M680X_SUPPORT
      Enabling CAPSTONE_EVM_SUPPORT
      Enabling CAPSTONE_MOS65XX_SUPPORT
      Enabling CAPSTONE_WASM_SUPPORT
      Enabling CAPSTONE_BPF_SUPPORT
      Enabling CAPSTONE_RISCV_SUPPORT
      Enabling CAPSTONE_SH_SUPPORT
      Enabling CAPSTONE_TRICORE_SUPPORT
      Enabling CAPSTONE_ALPHA_SUPPORT
      Enabling CAPSTONE_HPPA_SUPPORT
      Enabling CAPSTONE_LOONGARCH_SUPPORT
      Enabling CAPSTONE_XTENSA_SUPPORT
      CMake Error: File /tmp/pip-req-build-dqzt6b7j/src/capstone.pc.in does not exist.
      CMake Error at CMakeLists.txt:879 (configure_file):
        configure_file Problem configuring file

      CMake Error: File /tmp/pip-req-build-dqzt6b7j/src/capstone-config.cmake.in does not exist.
      CMake Error at /usr/share/cmake/Modules/CMakePackageConfigHelpers.cmake:501 (configure_file):
        configure_file Problem configuring file
      Call Stack (most recent call first):
        CMakeLists.txt:884 (configure_package_config_file)

      CMake Error: File /tmp/pip-req-build-dqzt6b7j/src/cmake_uninstall.cmake.in does not exist.
      CMake Error at CMakeLists.txt:924 (configure_file):
        configure_file Problem configuring file

      -- Configuring incomplete, errors occurred!
      gmake: Makefile: No such file or directory
      gmake: *** No rule to make target 'Makefile'.  Stop.
      error: [Errno 2] No such file or directory: 'libcapstone.so.6'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for capstone
Failed to build capstone
ERROR: Could not build wheels for capstone, which is required to install pyproject.toml-based projects

Should be fixed with commit: https://github.com/capstone-engine/capstone/pull/2538/commits/82fe006a60eb4035932a0a52c0733a8b8f7238ba

capstone-engine / capstone

Python package building rework #2538