search

Blosc / python-blosc

A Python wrapper for the extremely fast Blosc compression library
https://www.blosc.org/python-blosc/python-blosc.html
Other
353 stars 73 forks source link

Support Python 3.11 #298

Closed DimitriPapadopoulos closed 1 year ago

DimitriPapadopoulos commented 2 years ago

Not sure why it fails on macOS.

creating dist
creating 'dist/blosc-1.10.7.dev0-py3.11-macosx-11.0-x86_64.egg' and adding '_skbuild/macosx-11.0-x86_64-3.11/setuptools/bdist.macosx-11.0-x86_64/egg' to it
removing '_skbuild/macosx-11.0-x86_64-3.11/setuptools/bdist.macosx-11.0-x86_64/egg' (and everything under it)
Processing blosc-1.10.7.dev0-py3.11-macosx-11.0-x86_64.egg
creating /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/blosc-1.10.7.dev0-py3.11-macosx-11.0-x86_64.egg
Extracting blosc-1.10.7.dev0-py3.11-macosx-11.0-x86_64.egg to /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages
Adding blosc 1.10.7.dev0 to easy-install.pth file

Installed /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/blosc-1.10.7.dev0-py3.11-macosx-11.0-x86_64.egg
Processing dependencies for blosc==1.10.7.dev0
Searching for blosc==1.10.7.dev0
Reading https://pypi.org/simple/blosc/
/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning:  is an invalid version and will not be supported in a future release
  warnings.warn(
No local packages or working download links found for blosc==1.10.7.dev0
error: Could not find suitable distribution for Requirement.parse('blosc==1.10.7.dev0')
Error: Process completed with exit code 1.
DimitriPapadopoulos commented 2 years ago

Probably need to update .github/workflows/cibuildwheels.yml too.

DimitriPapadopoulos commented 2 years ago

OK, the main difference I see between the Python 3.10 and 3.11 CI logs on macos:

218c216,218
<     /Users/runner/work/python-blosc/python-blosc/_skbuild/macosx-10.16-x86_64-3.10/cmake-build
---
>     /Users/runner/work/python-blosc/python-blosc/_skbuild/macosx-11.0-x86_64-3.11/cmake-build
> /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
>   warnings.warn(
220c220
<     cmake /Users/runner/work/python-blosc/python-blosc -G Ninja -DCMAKE_INSTALL_PREFIX:PATH=/Users/runner/work/python-blosc/python-blosc/_skbuild/macosx-10.16-x86_64-3.10/cmake-install -DPYTHON_VERSION_STRING:STRING=3.10.8 -DSKBUILD:INTERNAL=TRUE -DCMAKE_MODULE_PATH:PATH=/Users/runner/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/skbuild/resources/cmake -DPython3_EXECUTABLE:FILEPATH=/Users/runner/hostedtoolcache/Python/3.10.8/x64/bin/python -DPython3_INCLUDE_DIR:PATH=/Users/runner/hostedtoolcache/Python/3.10.8/x64/include/python3.10 -DPython3_LIBRARY:PATH=/Users/runner/hostedtoolcache/Python/3.10.8/x64/lib/libpython3.10.dylib -DPython3_NumPy_INCLUDE_DIRS:PATH=/Users/runner/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/numpy/core/include -DPython_EXECUTABLE:FILEPATH=/Users/runner/hostedtoolcache/Python/3.10.8/x64/bin/python -DPython_INCLUDE_DIR:PATH=/Users/runner/hostedtoolcache/Python/3.10.8/x64/include/python3.10 -DPython_LIBRARY:PATH=/Users/runner/hostedtoolcache/Python/3.10.8/x64/lib/libpython3.10.dylib -DPython_NumPy_INCLUDE_DIRS:PATH=/Users/runner/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/numpy/core/include -DPYTHON_EXECUTABLE:FILEPATH=/Users/runner/hostedtoolcache/Python/3.10.8/x64/bin/python -DPYTHON_INCLUDE_DIR:PATH=/Users/runner/hostedtoolcache/Python/3.10.8/x64/include/python3.10 -DPYTHON_LIBRARY:PATH=/Users/runner/hostedtoolcache/Python/3.10.8/x64/lib/libpython3.10.dylib -DPYTHON_NumPy_INCLUDE_DIRS:PATH=/Users/runner/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/numpy/core/include -DUSE_SYSTEM_BLOSC:BOOL=OFF -DDEACTIVATE_SSE2:BOOL=OFF -DDEACTIVATE_AVX2:BOOL=ON -DDEACTIVATE_LZ4:BOOL=OFF -DDEACTIVATE_SNAPPY:BOOL=ON -DDEACTIVATE_ZLIB:BOOL=OFF -DDEACTIVATE_ZSTD:BOOL=OFF -DCMAKE_BUILD_TYPE:STRING=Release -DCMAKE_OSX_DEPLOYMENT_TARGET:STRING=10.16 -DCMAKE_OSX_ARCHITECTURES:STRING=x86_64
---
>     cmake /Users/runner/work/python-blosc/python-blosc -G Ninja -DCMAKE_INSTALL_PREFIX:PATH=/Users/runner/work/python-blosc/python-blosc/_skbuild/macosx-11.0-x86_64-3.11/cmake-install -DPYTHON_VERSION_STRING:STRING=3.11.0 -DSKBUILD:INTERNAL=TRUE -DCMAKE_MODULE_PATH:PATH=/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/skbuild/resources/cmake -DPython3_EXECUTABLE:FILEPATH=/Library/Frameworks/Python.framework/Versions/3.11/bin/python -DPython3_INCLUDE_DIR:PATH=/Library/Frameworks/Python.framework/Versions/3.11/include/python3.11 -DPython3_LIBRARY:PATH=/Library/Frameworks/Python.framework/Versions/3.11/lib/libpython3.11.dylib -DPython3_NumPy_INCLUDE_DIRS:PATH=/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/numpy/core/include -DPython_EXECUTABLE:FILEPATH=/Library/Frameworks/Python.framework/Versions/3.11/bin/python -DPython_INCLUDE_DIR:PATH=/Library/Frameworks/Python.framework/Versions/3.11/include/python3.11 -DPython_LIBRARY:PATH=/Library/Frameworks/Python.framework/Versions/3.11/lib/libpython3.11.dylib -DPython_NumPy_INCLUDE_DIRS:PATH=/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/numpy/core/include -DPYTHON_EXECUTABLE:FILEPATH=/Library/Frameworks/Python.framework/Versions/3.11/bin/python -DPYTHON_INCLUDE_DIR:PATH=/Library/Frameworks/Python.framework/Versions/3.11/include/python3.11 -DPYTHON_LIBRARY:PATH=/Library/Frameworks/Python.framework/Versions/3.11/lib/libpython3.11.dylib -DPYTHON_NumPy_INCLUDE_DIRS:PATH=/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/numpy/core/include -DUSE_SYSTEM_BLOSC:BOOL=OFF -DDEACTIVATE_SSE2:BOOL=OFF -DDEACTIVATE_AVX2:BOOL=ON -DDEACTIVATE_LZ4:BOOL=OFF -DDEACTIVATE_SNAPPY:BOOL=ON -DDEACTIVATE_ZLIB:BOOL=OFF -DDEACTIVATE_ZSTD:BOOL=OFF -DCMAKE_BUILD_TYPE:STRING=Release -DCMAKE_OSX_DEPLOYMENT_TARGET:STRING=11.0 -DCMAKE_OSX_ARCHITECTURES:STRING=x86_64
223,227c223,227
< copying blosc/version.py -> _skbuild/macosx-10.16-x86_64-3.10/cmake-install/blosc/version.py
< copying blosc/toplevel.py -> _skbuild/macosx-10.16-x86_64-3.10/cmake-install/blosc/toplevel.py
< copying blosc/_version.py -> _skbuild/macosx-10.16-x86_64-3.10/cmake-install/blosc/_version.py
< copying blosc/__init__.py -> _skbuild/macosx-10.16-x86_64-3.10/cmake-install/blosc/__init__.py
< copying blosc/test.py -> _skbuild/macosx-10.16-x86_64-3.10/cmake-install/blosc/test.py
---
> copying blosc/version.py -> _skbuild/macosx-11.0-x86_64-3.11/cmake-install/blosc/version.py
> copying blosc/toplevel.py -> _skbuild/macosx-11.0-x86_64-3.11/cmake-install/blosc/toplevel.py
> copying blosc/_version.py -> _skbuild/macosx-11.0-x86_64-3.11/cmake-install/blosc/_version.py
> copying blosc/__init__.py -> _skbuild/macosx-11.0-x86_64-3.11/cmake-install/blosc/__init__.py
> copying blosc/test.py -> _skbuild/macosx-11.0-x86_64-3.11/cmake-install/blosc/test.py
230c230
< /Users/runner/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
---
> /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
292,296c292,296
< creating 'dist/blosc-1.10.7.dev0-py3.10-macosx-10.16-x86_64.egg' and adding '_skbuild/macosx-10.16-x86_64-3.10/setuptools/bdist.macosx-10.16-x86_64/egg' to it
< removing '_skbuild/macosx-10.16-x86_64-3.10/setuptools/bdist.macosx-10.16-x86_64/egg' (and everything under it)
< Processing blosc-1.10.7.dev0-py3.10-macosx-10.16-x86_64.egg
< creating /Users/runner/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/blosc-1.10.7.dev0-py3.10-macosx-10.16-x86_64.egg
< Extracting blosc-1.10.7.dev0-py3.10-macosx-10.16-x86_64.egg to /Users/runner/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages
---
> creating 'dist/blosc-1.10.7.dev0-py3.11-macosx-11.0-x86_64.egg' and adding '_skbuild/macosx-11.0-x86_64-3.11/setuptools/bdist.macosx-11.0-x86_64/egg' to it
> removing '_skbuild/macosx-11.0-x86_64-3.11/setuptools/bdist.macosx-11.0-x86_64/egg' (and everything under it)
> Processing blosc-1.10.7.dev0-py3.11-macosx-11.0-x86_64.egg
> creating /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/blosc-1.10.7.dev0-py3.11-macosx-11.0-x86_64.egg
> Extracting blosc-1.10.7.dev0-py3.11-macosx-11.0-x86_64.egg to /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages
299c299
< Installed /Users/runner/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/blosc-1.10.7.dev0-py3.10-macosx-10.16-x86_64.egg
---
> Installed /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/blosc-1.10.7.dev0-py3.11-macosx-11.0-x86_64.egg
301c301,307
< Finished processing dependencies for blosc==1.10.7.dev0
---
> Searching for blosc==1.10.7.dev0
> Reading https://pypi.org/simple/blosc/
> /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning:  is an invalid version and will not be supported in a future release
>   warnings.warn(
> No local packages or working download links found for blosc==1.10.7.dev0
> error: Could not find suitable distribution for Requirement.parse('blosc==1.10.7.dev0')
> Error: Process completed with exit code 1.

As you can see, Python 3.10 runs on macOS 10.16, while Python 3.11 runs on macOS 11.0. I don't know if it's a Python 3.10 vs. 3.11 or a macOS 10 vs. 11 issue, but:

I guess that's the root of the problem.

FrancescAlted commented 1 year ago

Yeah, that's quite puzzling indeed. Curiously enough, I did not have this problem in adding support for Python 3.11 in Python-Blosc2 :-/

DimitriPapadopoulos commented 1 year ago

Then I have to compare requirements in python-blosc and python-blosc2.

Could this be a temporary glitch in the provided containers? Can you re-launch this failing job?

FrancescAlted commented 1 year ago

Yes, I relaunched the jobs several times a few minutes ago, and the error continues to be the same.

DimitriPapadopoulos commented 1 year ago

python-blosc has if(USE_SYSTEM_BLOSC) in blosc/CMakeLists.txt, and I can see a few other differences with blosc2/CMakeLists.txt:

< add_library(blosc_extension MODULE blosc_extension.c)
---
> set(STATIC_LIB ON CACHE BOOL "Build a static version of the blosc library.")
> set(SHARED_LIB ON CACHE BOOL "Build a shared library version of the blosc
>     library.")
> set(CMAKE_POSITION_INDEPENDENT_CODE ON)
> # we want the binaries of the C-Blosc2 library to go into the wheels
> set(BLOSC_INSTALL ON)
> add_subdirectory(c-blosc2)
> include_directories("${CMAKE_CURRENT_SOURCE_DIR}/c-blosc2/include")
3,18d9
< if(USE_SYSTEM_BLOSC)
<     set(CMAKE_MODULE_PATH "${PROJECT_SOURCE_DIR}/cmake")
<     find_package(Blosc REQUIRED)
<     target_include_directories(blosc_extension PRIVATE ${Blosc_INCLUDE_DIRS})
<     target_link_libraries(blosc_extension ${Blosc_LIBRARIES})
< else()
<     set(BUILD_STATIC ON CACHE BOOL "Build a static version of the blosc library.")
<     set(BUILD_SHARED ON CACHE BOOL "Build a shared library version of the blosc library.")
<     set(BUILD_TESTS OFF CACHE BOOL "Build test programs form the blosc compression library")
<     set(BUILD_BENCHMARKS OFF CACHE BOOL "Build benchmark programs form the blosc compression library")
<     set(BLOSC_IS_SUBPROJECT OFF CACHE BOOL "Blosc is subproject")
<     set(CMAKE_POSITION_INDEPENDENT_CODE ON)
<     add_subdirectory(c-blosc)
<     include_directories("${CMAKE_CURRENT_SOURCE_DIR}/c-blosc/blosc")
<     target_link_libraries(blosc_extension blosc_static)
< endif()
19a11,12
> add_cython_target(blosc2_ext blosc2_ext.pyx)
> add_library(blosc2_ext MODULE ${blosc2_ext})
20a14
> target_link_libraries(blosc2_ext blosc2_static)

Will keep looking into it, together with the requirements*.txt files.

DimitriPapadopoulos commented 1 year ago

Aligning the CMakeLists.txt files doesn't help.

I'm not even certain this is a problem of access rights to /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/, the error message is not helpful:

Installed /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/blosc-1.10.7.dev0-py3.11-macosx-11.0-x86_64.egg
Processing dependencies for blosc==1.10.7.dev0
Searching for blosc==1.10.7.dev0
Reading https://pypi.org/simple/blosc/
/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning:  is an invalid version and will not be supported in a future release
  warnings.warn(
No local packages or working download links found for blosc==1.10.7.dev0
error: Could not find suitable distribution for Requirement.parse('blosc==1.10.7.dev0')
Error: Process completed with exit code 1.
FrancescAlted commented 1 year ago

Hmm, strange. After a quick googling, I found this: https://stackoverflow.com/questions/55845606/no-local-packages-or-working-download-links-found-for-sip-in-the-appveyor-windo . There it is recommended an update of setuptools, but that could be extended to pip too. Worth a try?

DimitriPapadopoulos commented 1 year ago

Upgrading setuptools doesn't help.

The behaviour of python setup.py install really changes with macOS 11.0 and/or Python 3.11. Perhaps we should try pip install instead.

Or maybe the version of CMake is different. Let's check that as well.

DimitriPapadopoulos commented 1 year ago

Here are the dependencies we install: https://github.com/Blosc/python-blosc/blob/56c38f5a7c5ac58a7066fe3210e02b436540e77a/.github/workflows/build.yml#L25-L29

The result is almost identical between Python3.10 / macOS 10.16 and Python3.11 / macOS 11.0:

@@ -1,5 +1,6 @@
 Package      Version
------------- -------
+------------ ---------
+certifi      2022.9.24
 cmake        3.24.3
 distro       1.8.0
 numpy        1.23.4

Specifically pip and setuptools are up to date:

pip          22.3.1
setuptools   65.5.1
wheel        0.38.4
DimitriPapadopoulos commented 1 year ago

The build process fails after installing all dependencies as described above, inside the installation process: https://github.com/Blosc/python-blosc/blob/5a150b834f0d4832a9490b9e0ce10426cacdffe2/.github/workflows/build.yml#L29-L31

For some reason, python setup.py install behaves differently between Python 3.10 and 3.11 on macOS. What is interesting is the warning on the Python 3.11 / macOS 11.0 case:

running install
/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(

We don't get this warning in the Python 3.10 / macOS 10.16 case. Despite installing the same version of setuptools, we get different behaviour. Why?

DimitriPapadopoulos commented 1 year ago

Perhaps related to the scikit-build>=0.11.1 dependency?

This issue was closed only a few days ago: https://github.com/scikit-learn/scikit-learn/issues/21499

In both cases, we install:

scikit-build 0.16.2
DimitriPapadopoulos commented 1 year ago

Works much better with python -m pip install . instead of python setup.py install!

It might be time to start using current installation methods instead of the deprecated python setup.py install.

FrancescAlted commented 1 year ago

Great news @DimitriPapadopoulos ! Now, I think it would be good to drop support for Python 3.7 because producing wheels for 4 Python versions (3.8, 3.9, 3.10, 3.11) is already a lot. But that could be done in another PR.

DimitriPapadopoulos commented 1 year ago

Python 3.7 will be supported until June 2023. Can we wait until then?

By the way, do you build wheels twice, to test in .github/workflows/build.yml, and to publish in .github/workflows/cibuildwheels.yml? If so, what about testing a few Python versions only, including the oldest supported and the latest, such as 3.7, 3.9, 3.11?

FrancescAlted commented 1 year ago

Python 3.7 will be supported until June 2023. Can we wait until then?

As NumPy is a dependency here, I prefer to follow the specs for [NEP-29](https://numpy.org/neps/nep-0029-deprecation_policy.html#drop-schedule]. This document suggests deprecating support for Python 3.7 on Dec 26, 2021, so we are pretty safe in doing that now that more than 10 months have passed already.

By the way, do you build wheels twice, to test in .github/workflows/build.yml, and to publish in .github/workflows/cibuildwheels.yml? If so, what about testing a few Python versions only, including the oldest supported and the latest, such as 3.7, 3.9, 3.11?

As long as 3.8 is the new minimum, :+1: