search

google / flax

Flax is a neural network library for JAX that is designed for flexibility.
https://flax.readthedocs.io
Apache License 2.0
6.06k stars 642 forks source link

Pip install fails on macOS with Apple M1 and Python >= 3.9 #2341

Closed AlexIoannides closed 2 years ago

AlexIoannides commented 2 years ago

System information

Problem you have encountered:

I cannot install Flax into a virtual environment using Python 3.9 or greater. This problem is not encountered when using Python 3.8.

What you expected to happen:

Flax to be successfully installed in a virtual environment

Steps to reproduce:

$ python3.9 -m venv .venv
$ source .venv/bin/activate
$ pip install -U pip
$ pip install flax

Logs, error messages, etc:

Building wheels for collected packages: jax, tensorstore
  Building wheel for jax (setup.py) ... done
  Created wheel for jax: filename=jax-0.3.15-py3-none-any.whl size=1201920 sha256=3b2e1ffc02c289747e698e9f1ad4b6d189e291a1498d4f41f4af16c80c9ba07f
  Stored in directory: /Users/alexioannides/Library/Caches/pip/wheels/59/9d/cb/21506ed546ffa6e339e75a0bb7500d363d68349a89f9c5a080
  Building wheel for tensorstore (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for tensorstore (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [173 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating /var/folders/xk/fqw466sd3l7fztkl08x8hnk80000gn/T/tmpt8sfan0e/lib.macosx-10.9-universal2-cpython-39
      creating /var/folders/xk/fqw466sd3l7fztkl08x8hnk80000gn/T/tmpt8sfan0e/lib.macosx-10.9-universal2-cpython-39/tensorstore
      copying python/tensorstore/cc_test_driver_main.py -> /var/folders/xk/fqw466sd3l7fztkl08x8hnk80000gn/T/tmpt8sfan0e/lib.macosx-10.9-universal2-cpython-39/tensorstore
      copying python/tensorstore/__init__.py -> /var/folders/xk/fqw466sd3l7fztkl08x8hnk80000gn/T/tmpt8sfan0e/lib.macosx-10.9-universal2-cpython-39/tensorstore
      running build_ext
      /Users/alexioannides/Dropbox/work/workspace/python/data-science-and-ml-notebook/.venv/bin/python3.9 -u bazelisk.py build -c opt //python/tensorstore:_tensorstore__shared_objects --verbose_failures --macos_minimum_os=10.14
      Starting local Bazel server and connecting to it...
      Loading:
      Loading: 0 packages loaded
      Analyzing: target //python/tensorstore:_tensorstore__shared_objects (1 packages loaded, 0 targets configured)
      Analyzing: target //python/tensorstore:_tensorstore__shared_objects (34 packages loaded, 133 targets configured)
      Analyzing: target //python/tensorstore:_tensorstore__shared_objects (34 packages loaded, 133 targets configured)
      Analyzing: target //python/tensorstore:_tensorstore__shared_objects (34 packages loaded, 133 targets configured)
      Analyzing: target //python/tensorstore:_tensorstore__shared_objects (107 packages loaded, 1623 targets configured)
      Analyzing: target //python/tensorstore:_tensorstore__shared_objects (122 packages loaded, 2872 targets configured)
      INFO: Analyzed target //python/tensorstore:_tensorstore__shared_objects (125 packages loaded, 5487 targets configured).
      INFO: Found 1 target...
      [0 / 385] [Prepa] Writing script python/tensorstore/_tensorstore_cc.cppmap ... (3 actions, 0 running)
      [327 / 803] Compiling jchuff.c; 1s darwin-sandbox ... (10 actions, 9 running)
      [550 / 859] Compiling tensorstore/driver/copy.cc; 1s darwin-sandbox ... (10 actions, 9 running)
      [606 / 902] Compiling tensorstore/driver/virtual_chunked/virtual_chunked.cc; 2s darwin-sandbox ... (10 actions, 9 running)
      [646 / 946] Compiling python/tensorstore/virtual_chunked.cc; 2s darwin-sandbox ... (10 actions, 9 running)
      [751 / 1,035] Compiling python/tensorstore/virtual_chunked.cc; 4s darwin-sandbox ... (10 actions, 9 running)
      [820 / 1,093] Compiling tensorstore/context.cc; 2s darwin-sandbox ... (10 actions, 9 running)
      [928 / 1,188] Compiling python/tensorstore/status.cc; 2s darwin-sandbox ... (9 actions, 8 running)
      [1,086 / 1,452] Compiling tensorstore/driver/kvs_backed_chunk_driver.cc; 3s darwin-sandbox ... (10 actions, 9 running)
      INFO: From Compiling lib/vtls/sectransp.c:
      external/se_curl/lib/vtls/sectransp.c:1004:6: warning: address of function 'SecCertificateCopyLongDescription' will always evaluate to 'true' [-Wpointer-bool-conversion]
        if(SecCertificateCopyLongDescription)
        ~~ ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      external/se_curl/lib/vtls/sectransp.c:1004:6: note: prefix with the address-of operator to silence this warning
        if(SecCertificateCopyLongDescription)
           ^
           &
      external/se_curl/lib/vtls/sectransp.c:1011:6: warning: address of function 'SecCertificateCopySubjectSummary' will always evaluate to 'true' [-Wpointer-bool-conversion]
        if(SecCertificateCopySubjectSummary)
        ~~ ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      external/se_curl/lib/vtls/sectransp.c:1011:6: note: prefix with the address-of operator to silence this warning
        if(SecCertificateCopySubjectSummary)
           ^
           &
      external/se_curl/lib/vtls/sectransp.c:1125:6: warning: address of function 'SecItemCopyMatching' will always evaluate to 'true' [-Wpointer-bool-conversion]
        if(SecItemCopyMatching && kSecClassIdentity) {
           ^~~~~~~~~~~~~~~~~~~ ~~
      external/se_curl/lib/vtls/sectransp.c:1125:6: note: prefix with the address-of operator to silence this warning
        if(SecItemCopyMatching && kSecClassIdentity) {
           ^
           &
      external/se_curl/lib/vtls/sectransp.c:1415:6: warning: address of function 'SSLSetProtocolVersionMax' will always evaluate to 'true' [-Wpointer-bool-conversion]
        if(SSLSetProtocolVersionMax) {
        ~~ ^~~~~~~~~~~~~~~~~~~~~~~~
      external/se_curl/lib/vtls/sectransp.c:1415:6: note: prefix with the address-of operator to silence this warning
        if(SSLSetProtocolVersionMax) {
           ^
           &
      external/se_curl/lib/vtls/sectransp.c:1699:6: warning: address of function 'SSLCreateContext' will always evaluate to 'true' [-Wpointer-bool-conversion]
        if(SSLCreateContext) {  /* use the newer API if available */
        ~~ ^~~~~~~~~~~~~~~~
      external/se_curl/lib/vtls/sectransp.c:1699:6: note: prefix with the address-of operator to silence this warning
        if(SSLCreateContext) {  /* use the newer API if available */
           ^
           &
      external/se_curl/lib/vtls/sectransp.c:1733:6: warning: address of function 'SSLSetProtocolVersionMax' will always evaluate to 'true' [-Wpointer-bool-conversion]
        if(SSLSetProtocolVersionMax) {
        ~~ ^~~~~~~~~~~~~~~~~~~~~~~~
      external/se_curl/lib/vtls/sectransp.c:1733:6: note: prefix with the address-of operator to silence this warning
        if(SSLSetProtocolVersionMax) {
           ^
           &
      external/se_curl/lib/vtls/sectransp.c:1991:6: warning: address of function 'SSLSetSessionOption' will always evaluate to 'true' [-Wpointer-bool-conversion]
        if(SSLSetSessionOption && darwinver_maj >= 13) {
           ^~~~~~~~~~~~~~~~~~~ ~~
      external/se_curl/lib/vtls/sectransp.c:1991:6: note: prefix with the address-of operator to silence this warning
        if(SSLSetSessionOption && darwinver_maj >= 13) {
           ^
           &
      external/se_curl/lib/vtls/sectransp.c:2082:6: warning: address of function 'SSLSetSessionOption' will always evaluate to 'true' [-Wpointer-bool-conversion]
        if(SSLSetSessionOption) {
        ~~ ^~~~~~~~~~~~~~~~~~~
      external/se_curl/lib/vtls/sectransp.c:2082:6: note: prefix with the address-of operator to silence this warning
        if(SSLSetSessionOption) {
           ^
           &
      external/se_curl/lib/vtls/sectransp.c:2967:6: warning: address of function 'SecTrustEvaluateAsync' will always evaluate to 'true' [-Wpointer-bool-conversion]
        if(SecTrustEvaluateAsync) {
        ~~ ^~~~~~~~~~~~~~~~~~~~~
      external/se_curl/lib/vtls/sectransp.c:2967:6: note: prefix with the address-of operator to silence this warning
        if(SecTrustEvaluateAsync) {
           ^
           &
      external/se_curl/lib/vtls/sectransp.c:3187:8: warning: address of function 'SSLCreateContext' will always evaluate to 'true' [-Wpointer-bool-conversion]
          if(SSLCreateContext)
          ~~ ^~~~~~~~~~~~~~~~
      external/se_curl/lib/vtls/sectransp.c:3187:8: note: prefix with the address-of operator to silence this warning
          if(SSLCreateContext)
             ^
             &
      external/se_curl/lib/vtls/sectransp.c:3357:6: warning: address of function 'SSLSetSessionOption' will always evaluate to 'true' [-Wpointer-bool-conversion]
        if(SSLSetSessionOption)
        ~~ ^~~~~~~~~~~~~~~~~~~
      external/se_curl/lib/vtls/sectransp.c:3357:6: note: prefix with the address-of operator to silence this warning
        if(SSLSetSessionOption)
           ^
           &
      11 warnings generated.
      [1,252 / 1,485] Compiling lib/curl_rtmp.c; 0s darwin-sandbox ... (10 actions, 9 running)
      INFO: From Compiling blosc/blosc.c:
      external/org_blosc_cblosc/blosc/blosc.c:735:11: warning: variable 'ctbytes' set but not used [-Wunused-but-set-variable]
        int32_t ctbytes = 0;           /* number of compressed bytes in block */
                ^
      1 warning generated.
      INFO: From Compiling blosc/fastcopy.c:
      external/org_blosc_cblosc/blosc/fastcopy.c:63:30: warning: unused function 'copy_5_bytes' [-Wunused-function]
      static inline unsigned char *copy_5_bytes(unsigned char *out, const unsigned char *from) {
                                   ^
      external/org_blosc_cblosc/blosc/fastcopy.c:68:30: warning: unused function 'copy_6_bytes' [-Wunused-function]
      static inline unsigned char *copy_6_bytes(unsigned char *out, const unsigned char *from) {
                                   ^
      external/org_blosc_cblosc/blosc/fastcopy.c:73:30: warning: unused function 'copy_7_bytes' [-Wunused-function]
      static inline unsigned char *copy_7_bytes(unsigned char *out, const unsigned char *from) {
                                   ^
      3 warnings generated.
      [1,380 / 1,719] Compiling tensorstore/driver/neuroglancer_precomputed/driver.cc; 3s darwin-sandbox ... (10 actions, 9 running)
      [1,495 / 1,719] Compiling av1/decoder/decodeframe.c; 0s darwin-sandbox ... (10 actions running)
      [1,719 / 1,901] Compiling python/tensorstore/kvstore.cc; 4s darwin-sandbox ... (10 actions running)
      ERROR: /private/var/tmp/_bazel_alexioannides/746f47a53dbc822deda7b954035221c4/external/org_aomedia_aom/BUILD.bazel:580:11: Compiling aom_dsp/arm/subpel_variance_neon.c failed: (Exit 1): cc_wrapper.sh failed: error executing command
        (cd /private/var/tmp/_bazel_alexioannides/746f47a53dbc822deda7b954035221c4/sandbox/darwin-sandbox/890/execroot/com_google_tensorstore && \
        exec env - \
          PATH=/bin:/usr/bin:/usr/local/bin \
          PWD=/proc/self/cwd \
        external/local_config_cc/cc_wrapper.sh -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -Wunused-but-set-parameter -Wno-free-nonheap-object -fcolor-diagnostics -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections -MD -MF bazel-out/darwin_arm64-opt/bin/external/org_aomedia_aom/_objs/dsp/subpel_variance_neon.pic.d '-frandom-seed=bazel-out/darwin_arm64-opt/bin/external/org_aomedia_aom/_objs/dsp/subpel_variance_neon.pic.o' -fPIC -iquote external/org_aomedia_aom -iquote bazel-out/darwin_arm64-opt/bin/external/org_aomedia_aom '-fdiagnostics-color=always' -Wno-deprecated-declarations -Wno-sign-compare '-std=c99' -DNEON_INTRINSICS -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c external/org_aomedia_aom/aom_dsp/arm/subpel_variance_neon.c -o bazel-out/darwin_arm64-opt/bin/external/org_aomedia_aom/_objs/dsp/subpel_variance_neon.pic.o)
      # Configuration: 84799bc94b0aa478ce98deadb38451bae5772174dc99beda441f5876af874716
      # Execution platform: @local_config_platform//:host

      Use --sandbox_debug to see verbose messages from the sandbox
      external/org_aomedia_aom/aom_dsp/arm/subpel_variance_neon.c:121:10: error: implicit declaration of function 'aom_variance8x8_neon' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        return aom_variance8x8_neon(temp2, 8, dst, dst_stride, sse);
               ^
      external/org_aomedia_aom/aom_dsp/arm/subpel_variance_neon.c:121:10: note: did you mean 'aom_variance8x8_c'?
      bazel-out/darwin_arm64-opt/bin/external/org_aomedia_aom/config/aom_dsp_rtcd.h:4729:14: note: 'aom_variance8x8_c' declared here
      unsigned int aom_variance8x8_c(const uint8_t *src_ptr, int source_stride, const uint8_t *ref_ptr, int ref_stride, unsigned int *sse);
                   ^
      external/org_aomedia_aom/aom_dsp/arm/subpel_variance_neon.c:136:10: error: implicit declaration of function 'aom_variance16x16_neon' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        return aom_variance16x16_neon(temp2, 16, dst, dst_stride, sse);
               ^
      external/org_aomedia_aom/aom_dsp/arm/subpel_variance_neon.c:136:10: note: did you mean 'aom_variance16x16_c'?
      bazel-out/darwin_arm64-opt/bin/external/org_aomedia_aom/config/aom_dsp_rtcd.h:4663:14: note: 'aom_variance16x16_c' declared here
      unsigned int aom_variance16x16_c(const uint8_t *src_ptr, int source_stride, const uint8_t *ref_ptr, int ref_stride, unsigned int *sse);
                   ^
      external/org_aomedia_aom/aom_dsp/arm/subpel_variance_neon.c:151:10: error: implicit declaration of function 'aom_variance32x32_neon' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        return aom_variance32x32_neon(temp2, 32, dst, dst_stride, sse);
               ^
      external/org_aomedia_aom/aom_dsp/arm/subpel_variance_neon.c:151:10: note: did you mean 'aom_variance32x32_c'?
      bazel-out/darwin_arm64-opt/bin/external/org_aomedia_aom/config/aom_dsp_rtcd.h:4687:14: note: 'aom_variance32x32_c' declared here
      unsigned int aom_variance32x32_c(const uint8_t *src_ptr, int source_stride, const uint8_t *ref_ptr, int ref_stride, unsigned int *sse);
                   ^
      external/org_aomedia_aom/aom_dsp/arm/subpel_variance_neon.c:166:10: error: implicit declaration of function 'aom_variance64x64_neon' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        return aom_variance64x64_neon(temp2, 64, dst, dst_stride, sse);
               ^
      external/org_aomedia_aom/aom_dsp/arm/subpel_variance_neon.c:166:10: note: did you mean 'aom_variance64x64_c'?
      bazel-out/darwin_arm64-opt/bin/external/org_aomedia_aom/config/aom_dsp_rtcd.h:4717:14: note: 'aom_variance64x64_c' declared here
      unsigned int aom_variance64x64_c(const uint8_t *src_ptr, int source_stride, const uint8_t *ref_ptr, int ref_stride, unsigned int *sse);
                   ^
      4 errors generated.
      Target //python/tensorstore:_tensorstore__shared_objects failed to build
      INFO: Elapsed time: 51.604s, Critical Path: 8.05s
      INFO: 1751 processes: 869 internal, 882 darwin-sandbox.
      FAILED: Build did NOT complete successfully
      FAILED: Build did NOT complete successfully
      error: command '/Users/alexioannides/Dropbox/work/workspace/python/data-science-and-ml-notebook/.venv/bin/python3.9' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tensorstore
Successfully built jax
Failed to build tensorstore
ERROR: Could not build wheels for tensorstore, which is required to install pyproject.toml-based projects
AlexIoannides commented 2 years ago

I can successfully install flax==0.5.2, so I'm attributing this issue solely to the addition of tensorstore in setup.py requirements, which first appeared in flax==0.5.3.

IvyZX commented 2 years ago

The dependency of tensorstore was added in flax 0.5.3 to support multi-process checkpointing. I suppose we could either make it optional or pin down its installation at a safe version that works in every environment.

Update: This seems to be an existed issue of Tensorstore: https://github.com/google/tensorstore/issues/33

AlexIoannides commented 2 years ago

My personal preference would be to have it as an optional/extra dependency, until the Tensorstore project can provide a binary that works for all platforms.

AlexIoannides commented 2 years ago

Thanks for doing this 🙏🏽

IvyZX commented 2 years ago

The original Tensorstore issue mentioned that arm64 should be supported now:

As of 0.1.22, there are now macOS arm64 binary packages available on PyPI.

Alex @AlexIoannides , could you help us by running pip install tensorstore on Python 3.9 or greater and see if it works on your M1 now? Thank you!

AlexIoannides commented 2 years ago

That works perfectly - thank you all 🙏🏽

image
HarrisonWilde commented 1 year ago

Sorry to necro but thought I'd mention that I am experiencing a similar issue trying to install tensorstore in Python 3.11 on an M1 mac:

Building wheels for collected packages: jax, msgpack, tensorstore, dm-tree
  Building wheel for jax (setup.py) ... done
  Created wheel for jax: filename=jax-0.3.25-py3-none-any.whl size=1308496 sha256=a71e153b66de5e869aff8cc554ee6de05e3665bbe773721cb426306b4fa6d8b7
  Stored in directory: /Users/harrisonwilde/Library/Caches/pip/wheels/5e/31/a1/9e64af5876ad5cea8ea582193835d3b21d7643be9f4bf4b6f5
  Building wheel for msgpack (pyproject.toml) ... done
  Created wheel for msgpack: filename=msgpack-1.0.4-cp311-cp311-macosx_11_0_arm64.whl size=68200 sha256=a05685c1e038ebaed224de176a3c0f63050e49961872e8aae9bc20532b925ce9
  Stored in directory: /Users/harrisonwilde/Library/Caches/pip/wheels/be/38/62/bffc8d68ee5e3a6a3080b2f8a520e8302fe333528d93a488af
  Building wheel for tensorstore (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for tensorstore (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [181 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating /var/folders/fy/skyvcp4s3tj6z_xhj2sv7jf00000gn/T/tmpna3qrqis/lib.macosx-11.0-arm64-cpython-311
      creating /var/folders/fy/skyvcp4s3tj6z_xhj2sv7jf00000gn/T/tmpna3qrqis/lib.macosx-11.0-arm64-cpython-311/tensorstore
      copying python/tensorstore/__init__.py -> /var/folders/fy/skyvcp4s3tj6z_xhj2sv7jf00000gn/T/tmpna3qrqis/lib.macosx-11.0-arm64-cpython-311/tensorstore
      running build_ext
      /Users/harrisonwilde/miniforge3/envs/flowmc/bin/python3.11 -u bazelisk.py build -c opt //python/tensorstore:_tensorstore__shared_objects --verbose_failures --macos_minimum_os=11.0 --copt=-fvisibility=hidden
      Downloading https://releases.bazel.build/5.0.0/release/bazel-5.0.0-darwin-arm64...
      Downloading https://releases.bazel.build/5.0.0/release/bazel-5.0.0-darwin-arm64.sha256...
      Extracting Bazel installation...
      Starting local Bazel server and connecting to it...
      Loading:
      Loading: 0 packages loaded
      Loading: 0 packages loaded
      Loading: 0 packages loaded
          currently loading: python/tensorstore
      Analyzing: target //python/tensorstore:_tensorstore__shared_objects (1 packages loaded, 0 targets configured)
      Analyzing: target //python/tensorstore:_tensorstore__shared_objects (31 packages loaded, 10 targets configured)
      Analyzing: target //python/tensorstore:_tensorstore__shared_objects (32 packages loaded, 10 targets configured)
      Analyzing: target //python/tensorstore:_tensorstore__shared_objects (32 packages loaded, 10 targets configured)
      Analyzing: target //python/tensorstore:_tensorstore__shared_objects (35 packages loaded, 166 targets configured)
      Analyzing: target //python/tensorstore:_tensorstore__shared_objects (35 packages loaded, 166 targets configured)
      Analyzing: target //python/tensorstore:_tensorstore__shared_objects (112 packages loaded, 1266 targets configured)
      Analyzing: target //python/tensorstore:_tensorstore__shared_objects (128 packages loaded, 2556 targets configured)
      Analyzing: target //python/tensorstore:_tensorstore__shared_objects (134 packages loaded, 3306 targets configured)
      INFO: Analyzed target //python/tensorstore:_tensorstore__shared_objects (136 packages loaded, 5925 targets configured).
      INFO: Found 1 target...
      [0 / 315] [Prepa] BazelWorkspaceStatusAction stable-status.txt
      INFO: From Compiling blosc/shuffle.c:
      external/org_blosc_cblosc/blosc/shuffle.c:306:4: warning: Hardware-acceleration detection not implemented for the target architecture. Only the generic shuffle/unshuffle routines will be available. [-W#warnings]
        #warning Hardware-acceleration detection not implemented for the target architecture. Only the generic shuffle/unshuffle routines will be available.
         ^
      external/org_blosc_cblosc/blosc/shuffle.c:316:22: warning: unused variable 'cpu_features' [-Wunused-variable]
        blosc_cpu_features cpu_features = blosc_get_cpu_features();
                           ^
      In file included from external/org_blosc_cblosc/blosc/shuffle.c:12:
      external/org_blosc_cblosc/blosc/shuffle-generic.h:32:13: warning: unused function 'shuffle_generic_inline' [-Wunused-function]
      static void shuffle_generic_inline(const size_t type_size,
                  ^
      external/org_blosc_cblosc/blosc/shuffle-generic.h:61:13: warning: unused function 'unshuffle_generic_inline' [-Wunused-function]
      static void unshuffle_generic_inline(const size_t type_size,
                  ^
      4 warnings generated.
      INFO: From Compiling lib/vtls/sectransp.c:
      external/se_curl/lib/vtls/sectransp.c:1006:6: warning: address of function 'SecCertificateCopyLongDescription' will always evaluate to 'true' [-Wpointer-bool-conversion]
        if(SecCertificateCopyLongDescription)
        ~~ ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      external/se_curl/lib/vtls/sectransp.c:1006:6: note: prefix with the address-of operator to silence this warning
        if(SecCertificateCopyLongDescription)
           ^
           &
      external/se_curl/lib/vtls/sectransp.c:1013:6: warning: address of function 'SecCertificateCopySubjectSummary' will always evaluate to 'true' [-Wpointer-bool-conversion]
        if(SecCertificateCopySubjectSummary)
        ~~ ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      external/se_curl/lib/vtls/sectransp.c:1013:6: note: prefix with the address-of operator to silence this warning
        if(SecCertificateCopySubjectSummary)
           ^
           &
      external/se_curl/lib/vtls/sectransp.c:1127:6: warning: address of function 'SecItemCopyMatching' will always evaluate to 'true' [-Wpointer-bool-conversion]
        if(SecItemCopyMatching && kSecClassIdentity) {
           ^~~~~~~~~~~~~~~~~~~ ~~
      external/se_curl/lib/vtls/sectransp.c:1127:6: note: prefix with the address-of operator to silence this warning
        if(SecItemCopyMatching && kSecClassIdentity) {
           ^
           &
      external/se_curl/lib/vtls/sectransp.c:1417:6: warning: address of function 'SSLSetProtocolVersionMax' will always evaluate to 'true' [-Wpointer-bool-conversion]
        if(SSLSetProtocolVersionMax) {
        ~~ ^~~~~~~~~~~~~~~~~~~~~~~~
      external/se_curl/lib/vtls/sectransp.c:1417:6: note: prefix with the address-of operator to silence this warning
        if(SSLSetProtocolVersionMax) {
           ^
           &
      external/se_curl/lib/vtls/sectransp.c:1701:6: warning: address of function 'SSLCreateContext' will always evaluate to 'true' [-Wpointer-bool-conversion]
        if(SSLCreateContext) {  /* use the newer API if available */
        ~~ ^~~~~~~~~~~~~~~~
      external/se_curl/lib/vtls/sectransp.c:1701:6: note: prefix with the address-of operator to silence this warning
        if(SSLCreateContext) {  /* use the newer API if available */
           ^
           &
      external/se_curl/lib/vtls/sectransp.c:1735:6: warning: address of function 'SSLSetProtocolVersionMax' will always evaluate to 'true' [-Wpointer-bool-conversion]
        if(SSLSetProtocolVersionMax) {
        ~~ ^~~~~~~~~~~~~~~~~~~~~~~~
      external/se_curl/lib/vtls/sectransp.c:1735:6: note: prefix with the address-of operator to silence this warning
        if(SSLSetProtocolVersionMax) {
           ^
           &
      external/se_curl/lib/vtls/sectransp.c:1993:6: warning: address of function 'SSLSetSessionOption' will always evaluate to 'true' [-Wpointer-bool-conversion]
        if(SSLSetSessionOption && darwinver_maj >= 13) {
           ^~~~~~~~~~~~~~~~~~~ ~~
      external/se_curl/lib/vtls/sectransp.c:1993:6: note: prefix with the address-of operator to silence this warning
        if(SSLSetSessionOption && darwinver_maj >= 13) {
           ^
           &
      external/se_curl/lib/vtls/sectransp.c:2084:6: warning: address of function 'SSLSetSessionOption' will always evaluate to 'true' [-Wpointer-bool-conversion]
        if(SSLSetSessionOption) {
        ~~ ^~~~~~~~~~~~~~~~~~~
      external/se_curl/lib/vtls/sectransp.c:2084:6: note: prefix with the address-of operator to silence this warning
        if(SSLSetSessionOption) {
           ^
           &
      external/se_curl/lib/vtls/sectransp.c:2969:6: warning: address of function 'SecTrustCopyPublicKey' will always evaluate to 'true' [-Wpointer-bool-conversion]
        if(SecTrustCopyPublicKey) {
        ~~ ^~~~~~~~~~~~~~~~~~~~~
      external/se_curl/lib/vtls/sectransp.c:2969:6: note: prefix with the address-of operator to silence this warning
        if(SecTrustCopyPublicKey) {
           ^
           &
      external/se_curl/lib/vtls/sectransp.c:3189:8: warning: address of function 'SSLCreateContext' will always evaluate to 'true' [-Wpointer-bool-conversion]
          if(SSLCreateContext)
          ~~ ^~~~~~~~~~~~~~~~
      external/se_curl/lib/vtls/sectransp.c:3189:8: note: prefix with the address-of operator to silence this warning
          if(SSLCreateContext)
             ^
             &
      external/se_curl/lib/vtls/sectransp.c:3359:6: warning: address of function 'SSLSetSessionOption' will always evaluate to 'true' [-Wpointer-bool-conversion]
        if(SSLSetSessionOption)
        ~~ ^~~~~~~~~~~~~~~~~~~
      external/se_curl/lib/vtls/sectransp.c:3359:6: note: prefix with the address-of operator to silence this warning
        if(SSLSetSessionOption)
           ^
           &
      11 warnings generated.
      [577 / 2,647] Compiling tensorstore/index_space/transform_broadcastable_array.cc; 0s darwin-sandbox ... (10 actions, 9 running)
      [769 / 2,647] Compiling src/crypto/fipsmodule/bcm.c; 1s darwin-sandbox ... (10 actions, 9 running)
      INFO: From Compiling blosc/fastcopy.c:
      external/org_blosc_cblosc/blosc/fastcopy.c:63:30: warning: unused function 'copy_5_bytes' [-Wunused-function]
      static inline unsigned char *copy_5_bytes(unsigned char *out, const unsigned char *from) {
                                   ^
      external/org_blosc_cblosc/blosc/fastcopy.c:68:30: warning: unused function 'copy_6_bytes' [-Wunused-function]
      static inline unsigned char *copy_6_bytes(unsigned char *out, const unsigned char *from) {
                                   ^
      external/org_blosc_cblosc/blosc/fastcopy.c:73:30: warning: unused function 'copy_7_bytes' [-Wunused-function]
      static inline unsigned char *copy_7_bytes(unsigned char *out, const unsigned char *from) {
                                   ^
      3 warnings generated.
      INFO: From Compiling blosc/blosc.c:
      external/org_blosc_cblosc/blosc/blosc.c:735:11: warning: variable 'ctbytes' set but not used [-Wunused-but-set-variable]
        int32_t ctbytes = 0;           /* number of compressed bytes in block */
                ^
      1 warning generated.
      [907 / 2,647] Compiling tensorstore/driver/json/driver.cc; 1s darwin-sandbox ... (10 actions, 9 running)
      [1,022 / 2,647] Compiling tensorstore/data_type.cc; 1s darwin-sandbox ... (10 actions, 9 running)
      [1,233 / 2,647] Compiling tensorstore/chunk_layout.cc; 1s darwin-sandbox ... (11 actions, 10 running)
      INFO: From Linking external/com_google_absl/absl/types/libbad_variant_access.a:
      warning: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: archive library: bazel-out/darwin_arm64-opt/bin/external/com_google_absl/absl/types/libbad_variant_access.a the table of contents is empty (no object file members in the library define global symbols)
      [1,408 / 2,647] Compiling tensorstore/driver/zarr/metadata.cc; 2s darwin-sandbox ... (10 actions, 9 running)
      [1,690 / 2,647] Compiling tensorstore/driver/neuroglancer_precomputed/chunk_encoding.cc; 0s darwin-sandbox ... (10 actions, 9 running)
      INFO: From Compiling pngread.c:
      external/png/pngread.c:3455:26: warning: variable 'row' set but not used [-Wunused-but-set-variable]
                     png_bytep row = png_voidcast(png_bytep, display->first_row);
                               ^
      1 warning generated.
      [1,980 / 2,647] Compiling tensorstore/driver/image/png/driver.cc; 1s darwin-sandbox ... (10 actions, 9 running)
      INFO: From Linking external/com_google_absl/absl/types/libbad_optional_access.a:
      warning: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: archive library: bazel-out/darwin_arm64-opt/bin/external/com_google_absl/absl/types/libbad_optional_access.a the table of contents is empty (no object file members in the library define global symbols)
      [2,300 / 2,647] Compiling aom_dsp/variance.c; 6s darwin-sandbox ... (10 actions, 9 running)
      ERROR: /private/var/folders/fy/skyvcp4s3tj6z_xhj2sv7jf00000gn/T/pip-install-dpxxebuf/tensorstore_aee0c4b20e70448e85b67287bf270938/python/tensorstore/BUILD:381:20: Compiling python/tensorstore/bfloat16.cc failed: (Aborted): wrapped_clang_pp failed: error executing command
        (cd /private/var/tmp/_bazel_harrisonwilde/1a7ae3d01b6af9d0d908731bfcb8de1c/sandbox/darwin-sandbox/1875/execroot/com_google_tensorstore && \
        exec env - \
          APPLE_SDK_PLATFORM=MacOSX \
          APPLE_SDK_VERSION_OVERRIDE=13.0 \
          PATH=/bin:/usr/bin:/usr/local/bin \
          XCODE_VERSION_OVERRIDE=14.1.0.14B47b \
          ZERO_AR_DATE=1 \
        external/local_config_cc/wrapped_clang_pp '-D_FORTIFY_SOURCE=1' -fstack-protector -fcolor-diagnostics -Wall -Wthread-safety -Wself-assign -fno-omit-frame-pointer -g0 -O2 -DNDEBUG '-DNS_BLOCK_ASSERTIONS=1' '-std=c++11' 'DEBUG_PREFIX_MAP_PWD=.' -iquote . -iquote bazel-out/darwin_arm64-opt/bin -iquote external/com_github_nlohmann_json -iquote bazel-out/darwin_arm64-opt/bin/external/com_github_nlohmann_json -iquote external/com_github_pybind_pybind11 -iquote bazel-out/darwin_arm64-opt/bin/external/com_github_pybind_pybind11 -iquote external/local_config_python -iquote bazel-out/darwin_arm64-opt/bin/external/local_config_python -iquote external/pypa_numpy -iquote bazel-out/darwin_arm64-opt/bin/external/pypa_numpy -iquote external/com_google_absl -iquote bazel-out/darwin_arm64-opt/bin/external/com_google_absl -iquote external/com_google_boringssl -iquote bazel-out/darwin_arm64-opt/bin/external/com_google_boringssl -iquote external/com_google_riegeli -iquote bazel-out/darwin_arm64-opt/bin/external/com_google_riegeli -iquote external/net_sourceforge_half -iquote bazel-out/darwin_arm64-opt/bin/external/net_sourceforge_half -Ibazel-out/darwin_arm64-opt/bin/external/com_github_nlohmann_json/_virtual_includes/nlohmann_json -Ibazel-out/darwin_arm64-opt/bin/external/com_github_pybind_pybind11/_virtual_includes/pybind11 -Ibazel-out/darwin_arm64-opt/bin/external/local_config_python/_virtual_includes/python_headers -Ibazel-out/darwin_arm64-opt/bin/external/pypa_numpy/_virtual_includes/headers -Ibazel-out/darwin_arm64-opt/bin/external/net_sourceforge_half/_virtual_includes/half -isystem external/com_google_boringssl/src/include -isystem bazel-out/darwin_arm64-opt/bin/external/com_google_boringssl/src/include -MD -MF bazel-out/darwin_arm64-opt/bin/python/tensorstore/_objs/data_type/bfloat16.d '-DPYBIND11_NAMESPACE=pybind11' '-frandom-seed=bazel-out/darwin_arm64-opt/bin/python/tensorstore/_objs/data_type/bfloat16.o' -isysroot __BAZEL_XCODE_SDKROOT__ -F__BAZEL_XCODE_SDKROOT__/System/Library/Frameworks -F__BAZEL_XCODE_DEVELOPER_DIR__/Platforms/MacOSX.platform/Developer/Library/Frameworks '-mmacosx-version-min=11.0' -no-canonical-prefixes -pthread '-fdiagnostics-color=always' -Wno-deprecated-declarations -Wno-sign-compare '-fvisibility=hidden' '-std=c++17' -fsized-deallocation -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -target arm64-apple-macosx -c python/tensorstore/bfloat16.cc -o bazel-out/darwin_arm64-opt/bin/python/tensorstore/_objs/data_type/bfloat16.o)
      # Configuration: 03a6b4f34d5dcec9521e97b85aedaac30bfd75fcf63639e077ffeb1b6e417cbc
      # Execution platform: @local_config_platform//:host

      Use --sandbox_debug to see verbose messages from the sandbox
      python/tensorstore/bfloat16.cc:774:31: error: expression is not assignable
        Py_TYPE(&NPyBfloat16_Descr) = &PyArrayDescr_Type;
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^
      1 error generated.
      Error in child process '/usr/bin/xcrun'. 1
      Target //python/tensorstore:_tensorstore__shared_objects failed to build
      INFO: Elapsed time: 90.143s, Critical Path: 8.38s
      INFO: 2605 processes: 732 internal, 1873 darwin-sandbox.
      FAILED: Build did NOT complete successfully
      FAILED: Build did NOT complete successfully
      error: command '/Users/harrisonwilde/miniforge3/envs/flowmc/bin/python3.11' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tensorstore
  Building wheel for dm-tree (setup.py) ... error
Noahyt commented 1 year ago

Same as @HarrisonWilde -- tried specifying tensorstore==0.1.24 that worked for python (I am using Python 3.11) but this failed with a similar error.

wookayin commented 1 year ago

tensorstore (and hence flax) fails to install on M1 macOS and linux/python3.11 environments. Submitted a PR #2692 to drop it again.