search

sergey-dryabzhinsky / python-zstd

Simple python bindings to Yann Collet ZSTD compression library
BSD 2-Clause "Simplified" License
165 stars 27 forks source link

Port to Python 3.11: use Py_SET_SIZE() #70

Closed vstinner closed 2 years ago

vstinner commented 2 years ago

On Python 3.11, "Py_SIZE(result) = cSize;" fails with a compiler error. Since Python 3.9, Py_SET_SIZE() must be used instead:

Add a copy of the pythoncapi_compat.h header file to get Py_SET_SIZE() on Python 3.8 and older: https://github.com/pythoncapi/pythoncapi_compat

vstinner commented 2 years ago

The build error:

building 'zstd' extension
gcc -pthread -Wno-unused-result -Wsign-compare -g -Og -Wall -O0 -fPIC -I/home/vstinner/dev/python-zstd/env/include -I/home/vstinner/python/main/Include -I/home/vstinner/python/main -c src/python-zstd.c -o build/temp.linux-x86_64-3.11-pydebug/src/python-zstd.o -O2 -DVERSION="1.5.0.2" -DZSTD_MULTITHREAD=1 -Izstd/lib -Izstd/lib/common -Izstd/lib/compress -Izstd/lib/decompress -DZSTD_TRACE=0
src/python-zstd.c: In function ‘py_zstd_compress_mt’:
src/python-zstd.c:119:25: error: lvalue required as left operand of assignment
  119 |         Py_SIZE(result) = cSize;
      |                         ^
error: command '/usr/bin/gcc' failed with exit code 1

I'm the author of the Py_SIZE() change in Python 3.11 :-) It has been approved by the Steering Council: https://github.com/python/steering-council/issues/79

I'm also working on a PEP which changes other macros and explains the rationale for the Py_SIZE() change, see: https://www.python.org/dev/peps/pep-0674/

Moreover, I'm the author of the pythoncapi_compat project and its pythoncapi_compat.h header file: https://github.com/pythoncapi/pythoncapi_compat I'm trying to propose it as a standard way to support multiple Python versions with a single C code base. The header provides new C API functions on old Python versions.

vstinner commented 2 years ago

Test with the development branch of Python 3.11:

$ ~/python/main/python -m venv env # Python 3.11
$ env/bin/python setup.py test
(...)
INFO:ZSTD:Python version: 3.11.0a2+ (heads/main:cb2b3c8d35, Dec  2 2021, 11:42:55) [GCC 11.2.1 20210728 (Red Hat 11.2.1-1)]
test_compression_default_level (tests.test_compress.TestZstdCompress) ... ok
test_compression_default_level_default (tests.test_compress.TestZstdCompress) ... ok
test_compression_default_level_zero (tests.test_compress.TestZstdCompress) ... ok
test_compression_level1 (tests.test_compress.TestZstdCompress) ... ok
test_compression_level20 (tests.test_compress.TestZstdCompress) ... ok
test_compression_level6 (tests.test_compress.TestZstdCompress) ... ok
test_compression_multi_thread_many (tests.test_compress.TestZstdCompress) ... ok
test_compression_multi_thread_one (tests.test_compress.TestZstdCompress) ... ok
test_compression_negative_level (tests.test_compress.TestZstdCompress) ... ok
test_compression_negative_level_notdefault (tests.test_compress.TestZstdCompress) ... ok
test_compression_random (tests.test_compress.TestZstdCompress) ... ok
test_compression_wrong_level (tests.test_compress.TestZstdCompress) ... ok
test_library_version (tests.test_version.TestZstdVersion) ... ok
test_library_version_number (tests.test_version.TestZstdVersion) ... ok
test_library_version_number_min (tests.test_version.TestZstdVersion) ... ok
test_module_version (tests.test_version.TestZstdVersion) ... ok

----------------------------------------------------------------------
Ran 16 tests in 0.021s

OK
INFO:ZSTD:Python version: 3.11.0a2+ (heads/main:cb2b3c8d35, Dec  2 2021, 11:42:55) [GCC 11.2.1 20210728 (Red Hat 11.2.1-1)]
test_compression_default_level (tests.test_compress.TestZstdCompress) ... ok
test_compression_default_level_default (tests.test_compress.TestZstdCompress) ... ok
test_compression_default_level_zero (tests.test_compress.TestZstdCompress) ... ok
test_compression_level1 (tests.test_compress.TestZstdCompress) ... ok
test_compression_level20 (tests.test_compress.TestZstdCompress) ... ok
test_compression_level6 (tests.test_compress.TestZstdCompress) ... ok
test_compression_multi_thread_many (tests.test_compress.TestZstdCompress) ... ok
test_compression_multi_thread_one (tests.test_compress.TestZstdCompress) ... ok
test_compression_negative_level (tests.test_compress.TestZstdCompress) ... ok
test_compression_negative_level_notdefault (tests.test_compress.TestZstdCompress) ... ok
test_compression_random (tests.test_compress.TestZstdCompress) ... ok
test_compression_wrong_level (tests.test_compress.TestZstdCompress) ... ok
test_library_version (tests.test_version.TestZstdVersion) ... ok
test_library_version_number (tests.test_version.TestZstdVersion) ... ok
test_library_version_number_min (tests.test_version.TestZstdVersion) ... ok
test_module_version (tests.test_version.TestZstdVersion) ... ok

----------------------------------------------------------------------
Ran 16 tests in 0.009s

OK
sergey-dryabzhinsky commented 2 years ago

Did you check it with python versions 2.7, 3.4-3.10 too?

vstinner commented 2 years ago

Did you check it with python versions 2.7, 3.4-3.10 too?

I tested manually the following versions (thanks Fedora for providing a wide range of Python versions!):

$ git clean -fdx; python2.7 setup.py test
$ git clean -fdx; python3.4 -m venv env; env/bin/python setup.py test
$ git clean -fdx; python3.5 -m venv env; env/bin/python setup.py test
$ git clean -fdx; python3.6 -m venv env; env/bin/python setup.py test
$ git clean -fdx; python3.7 -m venv env; env/bin/python setup.py test
$ git clean -fdx; python3.8 -m venv env; env/bin/python setup.py test
$ git clean -fdx; python3.9 -m venv env; env/bin/python setup.py test
$ git clean -fdx; python3.10 -m venv env; env/bin/python setup.py test
$ git clean -fdx; ~/python/main/python -m venv env; env/bin/python setup.py test # dev version of Python 3.11

=> All builds and tests passed successfully!

pythoncapi_compat.h is compatible with Python 2.7: it has been tested on Windows with Visual Studio and Python 2.7. Mercurial uses it and Mercurial is still supporting Python 2.7 on Windows ;-)

By the way, the setup.py test command emits a warning:

WARNING: Testing via this command is deprecated and will be removed in a future version. Users looking for a generic test entry point independent of test runner are encouraged to use tox.

Using tox would avoid me to have to create the virtual environment and run Python manually for each Python version :-D

sergey-dryabzhinsky commented 2 years ago

Good. Thanks.