Help in packaging `pyperscan` or debugging build error for AUR

[AkechiShiro]

Hi,

I'm trying to package unblob in the AUR, and I'm currently stuck at building pyperscan.

Logs of a build attempt in a chroot are here : https://gist.github.com/AkechiShiro/0fc6840388183ff16e52fe1f23576752 End of logs show this failure with vectorscan installed in the chroot and compiled from source :

  running: cd "/build/python-pyperscan/src/pyperscan-0.2.0/target/release/build/hyperscan-sys-21227b15d35c8a79/out/build" && CMAKE_PREFIX_PATH="" "cmake" "/build/python-pyperscan/src/pyperscan-0.2.0/local_dependencies/hyperscan-sys/vectorscan" "-DCMAKE_INSTALL_INCLUDEDIR=/build/python-pyperscan/src/pyperscan-0.2.0/target/release/build/hyperscan-sys-21227b15d35c8a79/out/include" "-DFAT_RUNTIME=ON" "-DBUILD_AVX512=ON" "-DCMAKE_INSTALL_PREFIX=/build/python-pyperscan/src/pyperscan-0.2.0/target/release/build/hyperscan-sys-21227b15d35c8a79/out" "-DCMAKE_C_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64 -march=x86-64 -mtune=generic -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection" "-DCMAKE_C_COMPILER=/usr/bin/cc" "-DCMAKE_CXX_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64 -march=x86-64 -mtune=generic -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -Wp,-D_GLIBCXX_ASSERTIONS" "-DCMAKE_CXX_COMPILER=/usr/bin/c++" "-DCMAKE_ASM_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64 -march=x86-64 -mtune=generic -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection" "-DCMAKE_ASM_COMPILER=/usr/bin/cc" "-DCMAKE_BUILD_TYPE=release"

  --- stderr
  CMake Error: The source directory "/build/python-pyperscan/src/pyperscan-0.2.0/local_dependencies/hyperscan-sys/vectorscan" does not exist.
  Specify --help for usage, or press the help button on the CMake GUI.
  thread 'main' panicked at '
  command did not execute successfully, got: exit status: 1

  build script failed, must exit now', /build/.cargo/registry/src/github.com-1ecc6299db9ec823/cmake-0.1.49/src/lib.rs:1104:5
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
💥 maturin failed
  Caused by: Failed to build a native library through cargo
  Caused by: Cargo build finished with "exit status: 101": `PYTHON_SYS_EXECUTABLE="/usr/bin/python" "cargo" "rustc" "--release" "--manifest-path" "/build/python-pyperscan/src/pyperscan-0.2.0/Cargo.toml" "--message-format" "json" "--lib"`
Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/usr/bin/python', '--compatibility', 'off'] returned non-zero exit status 1

ERROR Backend subprocess exited when trying to invoke build_wheel
==> ERROR: A failure occurred in build().
    Aborting...
==> ERROR: Build failed, check /home/redacted/project/opensource/aur/unblob-git/chroot/redacted/build

The WIP script for pyperscan is the following :

_pyname=pyperscan
pkgname=python-${_pyname}
pkgver=0.2.0
pkgrel=1
pkgdesc='A hyperscan binding for Python, which supports vectorscan'
arch=('x86_64')
url="https://github.com/onekey-sec/$_pyname"
license=(MIT)
depends=(python)
conflicts=()
provides=(python-pyperscan)
makedepends=(cmake python-pip python-build python-installer python-wheel python-setuptools 'python-maturin>=0.14' 'python-maturin<0.15' vectorscan)
checkdepends=(python-pytest python-pytest-cov)
source=("https://files.pythonhosted.org/packages/source/${_pyname::1}/$_pyname/$_pyname-$pkgver.tar.gz")
sha256sums=('698f06b23872457f7b499c575b8a3528b637b426c5af77f4df3b97c6b4fd1a9e')

build() {
    cd "$_pyname-$pkgver"
    python -m build --wheel --no-isolation
}

package() {
    cd "$_pyname-$pkgver"
    python -m installer --destdir="$pkgdir" dist/*.whl
}

check()  {
  cd "$_pyname-$pkgver"
  local tmp=$(mktemp -d)
  local site_packages=$(python -c "import site; print(site.getsitepackages()[0])")
  ls -lah *
  python -m installer --destdir="$tmp" dist/$_pyname-$pkgver-py3-none-any.whl
  PYTHONPATH="$tmp$site_packages" python -m pytest tests

  rmdir --ignore-fail-on-non-empty $tmp 
}

[AkechiShiro]

Actually I think I figured it out, still testing, I have added boost as they were needed and I've transitioned to a git submodule clone approach, the tar.gz source code do not contain submodules vectorscan and hyperscan with the pinned commit hashes.

[vlaci]

Yes, the issue is the missing vectorscan sources. You can use the git clone approach, or depend on the arch version of hyperscan instead. Then, you can build a wheel via maturin build --no-default-features (you also need maturin as well) The only downside of this approach, is the loss of aarch64 compatibility.

I also introduced some changes recently: When built from the source tar.gz, it will try to use an installed version of either hyperscan or vectorscan (not yet released on PyPI).

[AkechiShiro]

@vlaci Could you please remove the use of maturin >=14 or < 15 as well ? After that change I will close, I finished packaging pyperscan for the AUR.

[vlaci]

@vlaci Could you please remove the use of maturin >=14 or < 15 as well ? After that change I will close, I finished packaging pyperscan for the AUR.

Updated in #24