Closed AlexIoannides closed 2 years ago
I can successfully install flax==0.5.2
, so I'm attributing this issue solely to the addition of tensorstore
in setup.py
requirements, which first appeared in flax==0.5.3
.
The dependency of tensorstore was added in flax 0.5.3 to support multi-process checkpointing. I suppose we could either make it optional or pin down its installation at a safe version that works in every environment.
Update: This seems to be an existed issue of Tensorstore: https://github.com/google/tensorstore/issues/33
My personal preference would be to have it as an optional/extra dependency, until the Tensorstore project can provide a binary that works for all platforms.
Thanks for doing this 🙏🏽
The original Tensorstore issue mentioned that arm64 should be supported now:
As of 0.1.22, there are now macOS arm64 binary packages available on PyPI.
Alex @AlexIoannides , could you help us by running pip install tensorstore
on Python 3.9 or greater and see if it works on your M1 now? Thank you!
That works perfectly - thank you all 🙏🏽
Sorry to necro but thought I'd mention that I am experiencing a similar issue trying to install tensorstore
in Python 3.11 on an M1 mac:
Building wheels for collected packages: jax, msgpack, tensorstore, dm-tree
Building wheel for jax (setup.py) ... done
Created wheel for jax: filename=jax-0.3.25-py3-none-any.whl size=1308496 sha256=a71e153b66de5e869aff8cc554ee6de05e3665bbe773721cb426306b4fa6d8b7
Stored in directory: /Users/harrisonwilde/Library/Caches/pip/wheels/5e/31/a1/9e64af5876ad5cea8ea582193835d3b21d7643be9f4bf4b6f5
Building wheel for msgpack (pyproject.toml) ... done
Created wheel for msgpack: filename=msgpack-1.0.4-cp311-cp311-macosx_11_0_arm64.whl size=68200 sha256=a05685c1e038ebaed224de176a3c0f63050e49961872e8aae9bc20532b925ce9
Stored in directory: /Users/harrisonwilde/Library/Caches/pip/wheels/be/38/62/bffc8d68ee5e3a6a3080b2f8a520e8302fe333528d93a488af
Building wheel for tensorstore (pyproject.toml) ... error
error: subprocess-exited-with-error
× Building wheel for tensorstore (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [181 lines of output]
running bdist_wheel
running build
running build_py
creating /var/folders/fy/skyvcp4s3tj6z_xhj2sv7jf00000gn/T/tmpna3qrqis/lib.macosx-11.0-arm64-cpython-311
creating /var/folders/fy/skyvcp4s3tj6z_xhj2sv7jf00000gn/T/tmpna3qrqis/lib.macosx-11.0-arm64-cpython-311/tensorstore
copying python/tensorstore/__init__.py -> /var/folders/fy/skyvcp4s3tj6z_xhj2sv7jf00000gn/T/tmpna3qrqis/lib.macosx-11.0-arm64-cpython-311/tensorstore
running build_ext
/Users/harrisonwilde/miniforge3/envs/flowmc/bin/python3.11 -u bazelisk.py build -c opt //python/tensorstore:_tensorstore__shared_objects --verbose_failures --macos_minimum_os=11.0 --copt=-fvisibility=hidden
Downloading https://releases.bazel.build/5.0.0/release/bazel-5.0.0-darwin-arm64...
Downloading https://releases.bazel.build/5.0.0/release/bazel-5.0.0-darwin-arm64.sha256...
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
Loading:
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
currently loading: python/tensorstore
Analyzing: target //python/tensorstore:_tensorstore__shared_objects (1 packages loaded, 0 targets configured)
Analyzing: target //python/tensorstore:_tensorstore__shared_objects (31 packages loaded, 10 targets configured)
Analyzing: target //python/tensorstore:_tensorstore__shared_objects (32 packages loaded, 10 targets configured)
Analyzing: target //python/tensorstore:_tensorstore__shared_objects (32 packages loaded, 10 targets configured)
Analyzing: target //python/tensorstore:_tensorstore__shared_objects (35 packages loaded, 166 targets configured)
Analyzing: target //python/tensorstore:_tensorstore__shared_objects (35 packages loaded, 166 targets configured)
Analyzing: target //python/tensorstore:_tensorstore__shared_objects (112 packages loaded, 1266 targets configured)
Analyzing: target //python/tensorstore:_tensorstore__shared_objects (128 packages loaded, 2556 targets configured)
Analyzing: target //python/tensorstore:_tensorstore__shared_objects (134 packages loaded, 3306 targets configured)
INFO: Analyzed target //python/tensorstore:_tensorstore__shared_objects (136 packages loaded, 5925 targets configured).
INFO: Found 1 target...
[0 / 315] [Prepa] BazelWorkspaceStatusAction stable-status.txt
INFO: From Compiling blosc/shuffle.c:
external/org_blosc_cblosc/blosc/shuffle.c:306:4: warning: Hardware-acceleration detection not implemented for the target architecture. Only the generic shuffle/unshuffle routines will be available. [-W#warnings]
#warning Hardware-acceleration detection not implemented for the target architecture. Only the generic shuffle/unshuffle routines will be available.
^
external/org_blosc_cblosc/blosc/shuffle.c:316:22: warning: unused variable 'cpu_features' [-Wunused-variable]
blosc_cpu_features cpu_features = blosc_get_cpu_features();
^
In file included from external/org_blosc_cblosc/blosc/shuffle.c:12:
external/org_blosc_cblosc/blosc/shuffle-generic.h:32:13: warning: unused function 'shuffle_generic_inline' [-Wunused-function]
static void shuffle_generic_inline(const size_t type_size,
^
external/org_blosc_cblosc/blosc/shuffle-generic.h:61:13: warning: unused function 'unshuffle_generic_inline' [-Wunused-function]
static void unshuffle_generic_inline(const size_t type_size,
^
4 warnings generated.
INFO: From Compiling lib/vtls/sectransp.c:
external/se_curl/lib/vtls/sectransp.c:1006:6: warning: address of function 'SecCertificateCopyLongDescription' will always evaluate to 'true' [-Wpointer-bool-conversion]
if(SecCertificateCopyLongDescription)
~~ ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
external/se_curl/lib/vtls/sectransp.c:1006:6: note: prefix with the address-of operator to silence this warning
if(SecCertificateCopyLongDescription)
^
&
external/se_curl/lib/vtls/sectransp.c:1013:6: warning: address of function 'SecCertificateCopySubjectSummary' will always evaluate to 'true' [-Wpointer-bool-conversion]
if(SecCertificateCopySubjectSummary)
~~ ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
external/se_curl/lib/vtls/sectransp.c:1013:6: note: prefix with the address-of operator to silence this warning
if(SecCertificateCopySubjectSummary)
^
&
external/se_curl/lib/vtls/sectransp.c:1127:6: warning: address of function 'SecItemCopyMatching' will always evaluate to 'true' [-Wpointer-bool-conversion]
if(SecItemCopyMatching && kSecClassIdentity) {
^~~~~~~~~~~~~~~~~~~ ~~
external/se_curl/lib/vtls/sectransp.c:1127:6: note: prefix with the address-of operator to silence this warning
if(SecItemCopyMatching && kSecClassIdentity) {
^
&
external/se_curl/lib/vtls/sectransp.c:1417:6: warning: address of function 'SSLSetProtocolVersionMax' will always evaluate to 'true' [-Wpointer-bool-conversion]
if(SSLSetProtocolVersionMax) {
~~ ^~~~~~~~~~~~~~~~~~~~~~~~
external/se_curl/lib/vtls/sectransp.c:1417:6: note: prefix with the address-of operator to silence this warning
if(SSLSetProtocolVersionMax) {
^
&
external/se_curl/lib/vtls/sectransp.c:1701:6: warning: address of function 'SSLCreateContext' will always evaluate to 'true' [-Wpointer-bool-conversion]
if(SSLCreateContext) { /* use the newer API if available */
~~ ^~~~~~~~~~~~~~~~
external/se_curl/lib/vtls/sectransp.c:1701:6: note: prefix with the address-of operator to silence this warning
if(SSLCreateContext) { /* use the newer API if available */
^
&
external/se_curl/lib/vtls/sectransp.c:1735:6: warning: address of function 'SSLSetProtocolVersionMax' will always evaluate to 'true' [-Wpointer-bool-conversion]
if(SSLSetProtocolVersionMax) {
~~ ^~~~~~~~~~~~~~~~~~~~~~~~
external/se_curl/lib/vtls/sectransp.c:1735:6: note: prefix with the address-of operator to silence this warning
if(SSLSetProtocolVersionMax) {
^
&
external/se_curl/lib/vtls/sectransp.c:1993:6: warning: address of function 'SSLSetSessionOption' will always evaluate to 'true' [-Wpointer-bool-conversion]
if(SSLSetSessionOption && darwinver_maj >= 13) {
^~~~~~~~~~~~~~~~~~~ ~~
external/se_curl/lib/vtls/sectransp.c:1993:6: note: prefix with the address-of operator to silence this warning
if(SSLSetSessionOption && darwinver_maj >= 13) {
^
&
external/se_curl/lib/vtls/sectransp.c:2084:6: warning: address of function 'SSLSetSessionOption' will always evaluate to 'true' [-Wpointer-bool-conversion]
if(SSLSetSessionOption) {
~~ ^~~~~~~~~~~~~~~~~~~
external/se_curl/lib/vtls/sectransp.c:2084:6: note: prefix with the address-of operator to silence this warning
if(SSLSetSessionOption) {
^
&
external/se_curl/lib/vtls/sectransp.c:2969:6: warning: address of function 'SecTrustCopyPublicKey' will always evaluate to 'true' [-Wpointer-bool-conversion]
if(SecTrustCopyPublicKey) {
~~ ^~~~~~~~~~~~~~~~~~~~~
external/se_curl/lib/vtls/sectransp.c:2969:6: note: prefix with the address-of operator to silence this warning
if(SecTrustCopyPublicKey) {
^
&
external/se_curl/lib/vtls/sectransp.c:3189:8: warning: address of function 'SSLCreateContext' will always evaluate to 'true' [-Wpointer-bool-conversion]
if(SSLCreateContext)
~~ ^~~~~~~~~~~~~~~~
external/se_curl/lib/vtls/sectransp.c:3189:8: note: prefix with the address-of operator to silence this warning
if(SSLCreateContext)
^
&
external/se_curl/lib/vtls/sectransp.c:3359:6: warning: address of function 'SSLSetSessionOption' will always evaluate to 'true' [-Wpointer-bool-conversion]
if(SSLSetSessionOption)
~~ ^~~~~~~~~~~~~~~~~~~
external/se_curl/lib/vtls/sectransp.c:3359:6: note: prefix with the address-of operator to silence this warning
if(SSLSetSessionOption)
^
&
11 warnings generated.
[577 / 2,647] Compiling tensorstore/index_space/transform_broadcastable_array.cc; 0s darwin-sandbox ... (10 actions, 9 running)
[769 / 2,647] Compiling src/crypto/fipsmodule/bcm.c; 1s darwin-sandbox ... (10 actions, 9 running)
INFO: From Compiling blosc/fastcopy.c:
external/org_blosc_cblosc/blosc/fastcopy.c:63:30: warning: unused function 'copy_5_bytes' [-Wunused-function]
static inline unsigned char *copy_5_bytes(unsigned char *out, const unsigned char *from) {
^
external/org_blosc_cblosc/blosc/fastcopy.c:68:30: warning: unused function 'copy_6_bytes' [-Wunused-function]
static inline unsigned char *copy_6_bytes(unsigned char *out, const unsigned char *from) {
^
external/org_blosc_cblosc/blosc/fastcopy.c:73:30: warning: unused function 'copy_7_bytes' [-Wunused-function]
static inline unsigned char *copy_7_bytes(unsigned char *out, const unsigned char *from) {
^
3 warnings generated.
INFO: From Compiling blosc/blosc.c:
external/org_blosc_cblosc/blosc/blosc.c:735:11: warning: variable 'ctbytes' set but not used [-Wunused-but-set-variable]
int32_t ctbytes = 0; /* number of compressed bytes in block */
^
1 warning generated.
[907 / 2,647] Compiling tensorstore/driver/json/driver.cc; 1s darwin-sandbox ... (10 actions, 9 running)
[1,022 / 2,647] Compiling tensorstore/data_type.cc; 1s darwin-sandbox ... (10 actions, 9 running)
[1,233 / 2,647] Compiling tensorstore/chunk_layout.cc; 1s darwin-sandbox ... (11 actions, 10 running)
INFO: From Linking external/com_google_absl/absl/types/libbad_variant_access.a:
warning: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: archive library: bazel-out/darwin_arm64-opt/bin/external/com_google_absl/absl/types/libbad_variant_access.a the table of contents is empty (no object file members in the library define global symbols)
[1,408 / 2,647] Compiling tensorstore/driver/zarr/metadata.cc; 2s darwin-sandbox ... (10 actions, 9 running)
[1,690 / 2,647] Compiling tensorstore/driver/neuroglancer_precomputed/chunk_encoding.cc; 0s darwin-sandbox ... (10 actions, 9 running)
INFO: From Compiling pngread.c:
external/png/pngread.c:3455:26: warning: variable 'row' set but not used [-Wunused-but-set-variable]
png_bytep row = png_voidcast(png_bytep, display->first_row);
^
1 warning generated.
[1,980 / 2,647] Compiling tensorstore/driver/image/png/driver.cc; 1s darwin-sandbox ... (10 actions, 9 running)
INFO: From Linking external/com_google_absl/absl/types/libbad_optional_access.a:
warning: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: archive library: bazel-out/darwin_arm64-opt/bin/external/com_google_absl/absl/types/libbad_optional_access.a the table of contents is empty (no object file members in the library define global symbols)
[2,300 / 2,647] Compiling aom_dsp/variance.c; 6s darwin-sandbox ... (10 actions, 9 running)
ERROR: /private/var/folders/fy/skyvcp4s3tj6z_xhj2sv7jf00000gn/T/pip-install-dpxxebuf/tensorstore_aee0c4b20e70448e85b67287bf270938/python/tensorstore/BUILD:381:20: Compiling python/tensorstore/bfloat16.cc failed: (Aborted): wrapped_clang_pp failed: error executing command
(cd /private/var/tmp/_bazel_harrisonwilde/1a7ae3d01b6af9d0d908731bfcb8de1c/sandbox/darwin-sandbox/1875/execroot/com_google_tensorstore && \
exec env - \
APPLE_SDK_PLATFORM=MacOSX \
APPLE_SDK_VERSION_OVERRIDE=13.0 \
PATH=/bin:/usr/bin:/usr/local/bin \
XCODE_VERSION_OVERRIDE=14.1.0.14B47b \
ZERO_AR_DATE=1 \
external/local_config_cc/wrapped_clang_pp '-D_FORTIFY_SOURCE=1' -fstack-protector -fcolor-diagnostics -Wall -Wthread-safety -Wself-assign -fno-omit-frame-pointer -g0 -O2 -DNDEBUG '-DNS_BLOCK_ASSERTIONS=1' '-std=c++11' 'DEBUG_PREFIX_MAP_PWD=.' -iquote . -iquote bazel-out/darwin_arm64-opt/bin -iquote external/com_github_nlohmann_json -iquote bazel-out/darwin_arm64-opt/bin/external/com_github_nlohmann_json -iquote external/com_github_pybind_pybind11 -iquote bazel-out/darwin_arm64-opt/bin/external/com_github_pybind_pybind11 -iquote external/local_config_python -iquote bazel-out/darwin_arm64-opt/bin/external/local_config_python -iquote external/pypa_numpy -iquote bazel-out/darwin_arm64-opt/bin/external/pypa_numpy -iquote external/com_google_absl -iquote bazel-out/darwin_arm64-opt/bin/external/com_google_absl -iquote external/com_google_boringssl -iquote bazel-out/darwin_arm64-opt/bin/external/com_google_boringssl -iquote external/com_google_riegeli -iquote bazel-out/darwin_arm64-opt/bin/external/com_google_riegeli -iquote external/net_sourceforge_half -iquote bazel-out/darwin_arm64-opt/bin/external/net_sourceforge_half -Ibazel-out/darwin_arm64-opt/bin/external/com_github_nlohmann_json/_virtual_includes/nlohmann_json -Ibazel-out/darwin_arm64-opt/bin/external/com_github_pybind_pybind11/_virtual_includes/pybind11 -Ibazel-out/darwin_arm64-opt/bin/external/local_config_python/_virtual_includes/python_headers -Ibazel-out/darwin_arm64-opt/bin/external/pypa_numpy/_virtual_includes/headers -Ibazel-out/darwin_arm64-opt/bin/external/net_sourceforge_half/_virtual_includes/half -isystem external/com_google_boringssl/src/include -isystem bazel-out/darwin_arm64-opt/bin/external/com_google_boringssl/src/include -MD -MF bazel-out/darwin_arm64-opt/bin/python/tensorstore/_objs/data_type/bfloat16.d '-DPYBIND11_NAMESPACE=pybind11' '-frandom-seed=bazel-out/darwin_arm64-opt/bin/python/tensorstore/_objs/data_type/bfloat16.o' -isysroot __BAZEL_XCODE_SDKROOT__ -F__BAZEL_XCODE_SDKROOT__/System/Library/Frameworks -F__BAZEL_XCODE_DEVELOPER_DIR__/Platforms/MacOSX.platform/Developer/Library/Frameworks '-mmacosx-version-min=11.0' -no-canonical-prefixes -pthread '-fdiagnostics-color=always' -Wno-deprecated-declarations -Wno-sign-compare '-fvisibility=hidden' '-std=c++17' -fsized-deallocation -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -target arm64-apple-macosx -c python/tensorstore/bfloat16.cc -o bazel-out/darwin_arm64-opt/bin/python/tensorstore/_objs/data_type/bfloat16.o)
# Configuration: 03a6b4f34d5dcec9521e97b85aedaac30bfd75fcf63639e077ffeb1b6e417cbc
# Execution platform: @local_config_platform//:host
Use --sandbox_debug to see verbose messages from the sandbox
python/tensorstore/bfloat16.cc:774:31: error: expression is not assignable
Py_TYPE(&NPyBfloat16_Descr) = &PyArrayDescr_Type;
~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^
1 error generated.
Error in child process '/usr/bin/xcrun'. 1
Target //python/tensorstore:_tensorstore__shared_objects failed to build
INFO: Elapsed time: 90.143s, Critical Path: 8.38s
INFO: 2605 processes: 732 internal, 1873 darwin-sandbox.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully
error: command '/Users/harrisonwilde/miniforge3/envs/flowmc/bin/python3.11' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for tensorstore
Building wheel for dm-tree (setup.py) ... error
Same as @HarrisonWilde -- tried specifying tensorstore==0.1.24
that worked for python (I am using Python 3.11) but this failed with a similar error.
tensorstore (and hence flax) fails to install on M1 macOS and linux/python3.11 environments. Submitted a PR #2692 to drop it again.
System information
>= 3.9
Problem you have encountered:
I cannot install Flax into a virtual environment using Python 3.9 or greater. This problem is not encountered when using Python 3.8.
What you expected to happen:
Flax to be successfully installed in a virtual environment
Steps to reproduce:
Logs, error messages, etc: