NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
17.08k stars 13.39k forks source link

hyperscan: replace with `vectorscan` #310772

Open Atemu opened 3 months ago

Atemu commented 3 months ago

Issue description

Apparently, the great minds at Intel felt the need to turn a regex library proprietary: https://www.phoronix.com/news/Intel-Hyperscan-Now-Proprietary

Shall we replace it with vectorscan (already packaged) like Fedora intends to? It claims to be API-compatible, is committed to FOSS and has cross-platform support (ARM for instance).

Leveraging this could also make some packages compatible cross-platform and/or enable hyperscan configuration for more platforms.

cc @avnik @fpletz @globin @nlewo @tnias @vlaci @0x4A6F @fabaff @magenbluten (maintainers of hyperscan/vectorscan and dependant packages)

vlaci commented 3 months ago

In my experience Vectorscan is binary compatible with Hyperscan's current version, so switching shouldn't be a problem. Although adding ARM support to software built on top of it may require some fine tuning of CMake flags to work. On x86, Vectorscan will have the same performance characteristics as well. I have no experience with the Chimera module though.

On the other hand I don't know if we need to be proactive about this. Existing versions of Hyperscan package will continue to work. So at least there is no time pressure.

tnias commented 3 months ago

Relevant fedora issue: https://bugzilla.redhat.com/show_bug.cgi?id=2263601

Debian has packages for hyperscan (since at least debian 10) and vectorscan (since debian 12), but they currently only have packages depending on hyperscan and nothing depending on vectorscan.

Reverse dependencies for libhyperscan5 and libvectorscan5 in debian:sid ``` % docker run --rm -it debian:sid root@d27655d895c4:/# apt update Get:1 http://deb.debian.org/debian sid InRelease [198 kB] Get:2 http://deb.debian.org/debian sid/main amd64 Packages [9876 kB] Fetched 10.1 MB in 2s (6224 kB/s) Reading package lists... Done Building dependency tree... Done Reading state information... Done 43 packages can be upgraded. Run 'apt list --upgradable' to see them. root@d27655d895c4:/# apt-cache rdepends libvectorscan5 libvectorscan5 Reverse Depends: libvectorscan-dev root@d27655d895c4:/# apt-cache rdepends libhyperscan5 libhyperscan5 Reverse Depends: libhyperscan-dev libvectorscan5 libvectorscan5 libvectorscan5 suricata rspamd kbtin root@d27655d895c4:/# ```
Atemu commented 3 months ago

Debian has packages for hyperscan (since at least debian 10) and vectorscan (since debian 12), but they currently only have packages depending on hyperscan and nothing depending on vectorscan.

Note that we're talking about an imperative FHS distro here; if the so names are the same, one package could simply be substituted with the other as an alternate implementation.
The packages are likely built against hyperscan but users can opt to install vectorscan instead. To facilitate this, it's declared to implement and replace "libhyperscan5":

root@04ba83f99d9d:/# apt-cache show libvectorscan5
Package: libvectorscan5
Source: vectorscan
Version: 5.4.11-2
Installed-Size: 13660
Maintainer: Konstantinos Margaritis <markos@debian.org>
Architecture: amd64
Replaces: libhyperscan5
Provides: libhyperscan5
Depends: sse4.2-support, libc6 (>= 2.32), libgcc-s1 (>= 3.0), libstdc++6 (>= 11)
Pre-Depends: debconf
Conflicts: libhyperscan5
Description: Portable fork of Intel's Hyperscan library
Description-md5: 122e14c3dd052e13b79532de5f832353
Multi-Arch: same
Homepage: https://vectorcamp.gr/vectorscan
Tag: role::shared-lib
Section: libs
Priority: optional
Filename: pool/main/v/vectorscan/libvectorscan5_5.4.11-2_amd64.deb
Size: 2523140
MD5sum: 64e0f0e1d276b86c92821ab69a340ced
SHA256: 77c5265cbebd6d64cf6a1b86a8e47c6c95e7da29b0231adf6662ecaea9c56441