v0.2.0 does not work with Python 3.10

[dfarley1]

We're currently using v0.2.0 since we have a customized version of libhs. Since trying to update to Python 3.10 we're seeing the following error (in a container based on python:3.10-bullseye:

$ python3 -m pip install hyperscan==0.2.0 --no-cache-dir
Defaulting to user installation because normal site-packages is not writeable
Collecting hyperscan==0.2.0
  Downloading hyperscan-0.2.0.tar.gz (13 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: hyperscan
  Building wheel for hyperscan (pyproject.toml) ... done
  Created wheel for hyperscan: filename=hyperscan-0.2.0-cp310-cp310-manylinux_2_31_x86_64.whl size=12566 sha256=c6665898f9cdb3b601ccbc21d0a432838502eb6644ab555e7220cf8e8a5bd341
  Stored in directory: /tmp/pip-ephem-wheel-cache-scq2qfof/wheels/4d/2a/88/2f04c0e9dfd098657e8617bff17d277bfa9eb5febc26b4ba33
Successfully built hyperscan
Installing collected packages: hyperscan
Successfully installed hyperscan-0.2.0
$ python3
Python 3.10.11 (main, Apr  5 2023, 23:35:11) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hyperscan
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/site-packages/hyperscan/__init__.py", line 5, in <module>
    from hyperscan._hyperscan import *
ModuleNotFoundError: No module named 'hyperscan._hyperscan'
>>> 

Upgrading to v0.4.0 fixes the problem, but we can't use it because we require dynamically linking libhs. I suppose this is either a bug report for the old version or a feature request for dynamic linking support in the newer version?

[dfarley1]

cc @betterlch

[betterlch]

cc @betterlch I just know that 0.2.0 works in python3.7~

---

Here are other work about python3.10:

I linked libhs.so as _hyperscn.so, and I get this

ImportError: dynamic module does not define module export function (PyInit__hyperscan)

And then I use

strace python3 -c 'import hyperscan'

It has no more help. I don't how C extending works in python3, so I have no ideas.

[darvid]

just added a new environment variable to enable dynamic linking when building, PYTHON_HYPERSCAN_STATIC (defaults to true). try pulling latest and building with:

PYTHON_HYPERSCAN_STATIC=false poetry install

[betterlch]

just added a new environment variable to enable dynamic linking when building, PYTHON_HYPERSCAN_STATIC (defaults to true). try pulling latest and building with:

PYTHON_HYPERSCAN_STATIC=false poetry install

Something maybe not as you wish:

running build_ext
building 'hyperscan._hyperscan' extension
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/include/openssl11 -I/usr/include/openssl11 -fPIC -I/usr/include/hs -I/usr/include/hs -I/tmp/tmpf0_08_r3/.venv/include -I/usr/local/include/python3.10 -c src/hyperscan/hyperscanmodule.c -o /root/py/python-hyperscan/build/temp.linux-x86_64-cpython-310/src/hyperscan/hyperscanmodule.o
src/hyperscan/hyperscanmodule.c: In function ‘ch_match_handler’:
src/hyperscan/hyperscanmodule.c:124:3: error: ‘for’ loop initial declarations are only allowed in C99 mode
   for (unsigned int i = 0; i < size; i++) {
   ^
src/hyperscan/hyperscanmodule.c:124:3: note: use option -std=c99 or -std=gnu99 to compile your code
src/hyperscan/hyperscanmodule.c: In function ‘Database_compile’:
src/hyperscan/hyperscanmodule.c:270:3: error: ‘for’ loop initial declarations are only allowed in C99 mode
   for (uint64_t i = 0; i < elements; i++) {
   ^
src/hyperscan/hyperscanmodule.c:331:5: error: ‘for’ loop initial declarations are only allowed in C99 mode
     for (uint64_t i = 0; i < elements; i++) {
     ^
src/hyperscan/hyperscanmodule.c:384:9: error: ‘for’ loop initial declarations are only allowed in C99 mode
         for (uint64_t i = 0; i < elements; i++) {
         ^
src/hyperscan/hyperscanmodule.c: In function ‘Database_scan’:
src/hyperscan/hyperscanmodule.c:547:5: error: ‘for’ loop initial declarations are only allowed in C99 mode
     for (uint32_t i = 0; i < num_buffers; i++) {
     ^
error: command '/usr/bin/gcc' failed with exit code 1

Resolved by change code to std format c11

And new error is here:

Preparing build environment with build-system requirements setuptools, poetry>=0.12Building hyperscan (0.4.0)
A setup.py file already exists. Using it.
running build
running build_py
copying src/hyperscan/hyperscanmodule.c -> /root/py/python-hyperscan/build/lib.linux-x86_64-cpython-310/hyperscan
running build_ext
building 'hyperscan._hyperscan' extension
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/include/openssl11 -I/usr/include/openssl11 -fPIC -I/usr/include/hs -I/usr/include/hs -I/tmp/tmp5r6qyfy_/.venv/include -I/usr/local/include/python3.10 -c src/hyperscan/hyperscanmodule.c -o /root/py/python-hyperscan/build/temp.linux-x86_64-cpython-310/src/hyperscan/hyperscanmodule.o
gcc -pthread -shared -L/usr/lib64/openssl11 -lssl -lcrypto -L/usr/lib64/openssl11 -lssl -lcrypto /root/py/python-hyperscan/build/temp.linux-x86_64-cpython-310/src/hyperscan/hyperscanmodule.o /opt/pcre/.libs/libpcre.a /opt/pcre/.libs/libpcre_la-pcre_byte_order.o /opt/pcre/.libs/libpcre_la-pcre_compile.o /opt/pcre/.libs/libpcre_la-pcre_config.o /opt/pcre/.libs/libpcre_la-pcre_dfa_exec.o /opt/pcre/.libs/libpcre_la-pcre_exec.o /opt/pcre/.libs/libpcre_la-pcre_fullinfo.o /opt/pcre/.libs/libpcre_la-pcre_get.o /opt/pcre/.libs/libpcre_la-pcre_globals.o /opt/pcre/.libs/libpcre_la-pcre_jit_compile.o /opt/pcre/.libs/libpcre_la-pcre_maketables.o /opt/pcre/.libs/libpcre_la-pcre_newline.o /opt/pcre/.libs/libpcre_la-pcre_ord2utf8.o /opt/pcre/.libs/libpcre_la-pcre_refcount.o /opt/pcre/.libs/libpcre_la-pcre_string_utils.o /opt/pcre/.libs/libpcre_la-pcre_study.o /opt/pcre/.libs/libpcre_la-pcre_tables.o /opt/pcre/.libs/libpcre_la-pcre_ucd.o /opt/pcre/.libs/libpcre_la-pcre_valid_utf8.o /opt/pcre/.libs/libpcre_la-pcre_version.o /opt/pcre/.libs/libpcre_la-pcre_xclass.o /opt/pcre/.libs/libpcre_la-pcre_chartables.o /opt/pcre/.libs/libpcreposix_la-pcreposix.o /opt/pcre/.libs/libpcrecpp_la-pcrecpp.o /opt/pcre/.libs/libpcrecpp_la-pcre_scanner.o /opt/pcre/.libs/libpcrecpp_la-pcre_stringpiece.o -lchimera -lhs -o /root/py/python-hyperscan/build/lib.linux-x86_64-cpython-310/hyperscan/_hyperscan.cpython-310-x86_64-linux-gnu.so
/usr/bin/ld: /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/libchimera.a(ch_compile.cpp.o): relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/libchimera.a(ch_runtime.c.o): relocation R_X86_64_32S against undefined symbol `mmbit_maxlevel_direct_lut' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/libchimera.a(ch_alloc.c.o): relocation R_X86_64_32 against symbol `malloc@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: error: ld returned 1 exit status
error: command '/usr/bin/gcc' failed with exit code 1

[betterlch]

1. download and make install hypercan 5.4.0 with doc guide
2. use cmake -DCMAKE_POSITION_INDEPENDENT_CODE=ON .. remake hyperscan/build/chimera --> hyperscan/build/lib/libchimera.a
3. add /opt/pcre/ and build pcre-8.4.5 to gen /opt/pcre/.libs for build python-hyperscan
4. poetry build -vvv (maybe need update cmake an add c11 support in python-hyperscan sourcecode)

Done!