search

oracle / graalpython

A high-performance embeddable Python 3 runtime for Java
https://www.graalvm.org/python/
Other
1.22k stars 104 forks source link

`pip install torch` fails #394

Closed kalaracey closed 5 months ago

kalaracey commented 6 months ago

https://www.graalvm.org/python/compatibility/ lists torch 1.13.1 as having Test Level % 68.89, but I cannot install it.

Apple clang version 15.0.0 (clang-1500.1.0.2.5) Target: x86_64-apple-darwin22.6.0 GraalPy 3.10.8 (Oracle GraalVM Native 23.1.2) macOS 13.6.1

Full build log: torch graalpy install.txt

Failure snippet:

      [5249/5701] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/torch/csrc/jit/runtime/register_ops_utils.cpp.o
      FAILED: caffe2/CMakeFiles/torch_cpu.dir/__/torch/csrc/jit/runtime/register_ops_utils.cpp.o
      /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -DAT_PER_OPERATOR_HEADERS -DCPUINFO_SUPPORTED_PLATFORM=1 -DFMT_HEADER_ONLY=1 -DFXDIV_USE_INLINE_ASSEMBLY=0 -DHAVE_MMAP=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DNNP_CONVOLUTION_ONLY=0 -DNNP_INFERENCE_ONLY=0 -DONNXIFI_ENABLE_EXT=1 -DONNX_ML=1 -DONNX_NAMESPACE=onnx_torch -DUSE_EXTERNAL_MZCRC -D_FILE_OFFSET_BITS=64 -Dtorch_cpu_EXPORTS -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/build/aten/src -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/aten/src -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/build -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962 -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/third_party/onnx -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/build/third_party/onnx -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/third_party/foxi -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/build/third_party/foxi -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/torch/csrc/api -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/torch/csrc/api/include -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/caffe2/aten/src/TH -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/build/caffe2/aten/src/TH -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/build/caffe2/aten/src -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/build/caffe2/../aten/src -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/torch/csrc -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/third_party/miniz-2.1.0 -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/third_party/kineto/libkineto/include -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/third_party/kineto/libkineto/src -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/aten/../third_party/catch/single_include -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/aten/src/ATen/.. -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/third_party/FXdiv/include -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/c10/.. -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/third_party/pthreadpool/include -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/third_party/cpuinfo/include -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/third_party/QNNPACK/include -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/aten/src/ATen/native/quantized/cpu/qnnpack/include -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/aten/src/ATen/native/quantized/cpu/qnnpack/src -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/third_party/cpuinfo/deps/clog/include -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/third_party/NNPACK/include -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/third_party/fbgemm/include -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/third_party/fbgemm -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/third_party/fbgemm/third_party/asmjit/src -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/third_party/ittapi/src/ittnotify -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/third_party/FP16/include -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/third_party/fmt/include -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/build/third_party/ideep/mkl-dnn/third_party/oneDNN/include -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/third_party/ideep/mkl-dnn/third_party/oneDNN/src/../include -I/private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/third_party/flatbuffers/include -F/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk/System/Library/Frameworks -isystem /private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/third_party/protobuf/src -isystem /private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/third_party/gemmlowp -isystem /private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/third_party/neon2sse -isystem /private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/third_party/XNNPACK/include -isystem /private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/third_party/ittapi/include -isystem /private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/cmake/../third_party/eigen -isystem /private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/third_party/ideep/mkl-dnn/third_party/oneDNN/include -isystem /private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/third_party/ideep/include -isystem /private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/build/include -Wno-deprecated -fvisibility-inlines-hidden -Wno-deprecated-declarations -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wvla-extension -Wno-range-loop-analysis -Wno-pass-failed -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -Wconstant-conversion -Wno-invalid-partial-specialization -Wno-typedef-redefinition -Wno-unused-private-field -Wno-inconsistent-missing-override -Wno-c++14-extensions -Wno-constexpr-not-const -Wno-missing-braces -Wunused-lambda-capture -Wunused-local-typedef -Qunused-arguments -fcolor-diagnostics -fdiagnostics-color=always -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -DUSE_MPS -fno-objc-arc -Wno-unguarded-availability-new -Wno-unused-private-field -Wno-missing-braces -Wno-c++14-extensions -Wno-constexpr-not-const -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -std=gnu++14 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk -mmacosx-version-min=13.6 -fPIC -DTH_HAVE_THREAD -Wall -Wextra -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-missing-field-initializers -Wno-write-strings -Wno-unknown-pragmas -Wno-type-limits -Wno-array-bounds -Wno-sign-compare -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-missing-braces -Wno-range-loop-analysis -fvisibility=hidden -O2 -DCAFFE2_BUILD_MAIN_LIB -DASMJIT_STATIC -MD -MT caffe2/CMakeFiles/torch_cpu.dir/__/torch/csrc/jit/runtime/register_ops_utils.cpp.o -MF caffe2/CMakeFiles/torch_cpu.dir/__/torch/csrc/jit/runtime/register_ops_utils.cpp.o.d -o caffe2/CMakeFiles/torch_cpu.dir/__/torch/csrc/jit/runtime/register_ops_utils.cpp.o -c /private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/torch/csrc/jit/runtime/register_ops_utils.cpp
      In file included from /private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/torch/csrc/jit/runtime/register_ops_utils.cpp:1:
      In file included from /private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/torch/csrc/jit/runtime/register_ops_utils.h:3:
      In file included from /private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/aten/src/ATen/Context.h:3:
      In file included from /private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/aten/src/ATen/CPUGeneratorImpl.h:3:
      In file included from /private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/aten/src/ATen/core/Generator.h:4:
      In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk/usr/include/c++/v1/mutex:191:
      In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk/usr/include/c++/v1/__memory/shared_ptr.h:30:
      In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk/usr/include/c++/v1/__memory/uninitialized_algorithms.h:13:
      In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk/usr/include/c++/v1/__algorithm/copy.h:12:
      In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk/usr/include/c++/v1/__algorithm/copy_move_common.h:12:
      /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk/usr/include/c++/v1/__algorithm/iterator_operations.h:134:12: error: calling a private constructor of class 'c10::impl::ListElementReference<at::Tensor, std::__wrap_iter<c10::IValue *>>'
          return *std::forward<_Iter>(__i);
                 ^
      /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk/usr/include/c++/v1/__algorithm/sift_down.h:55:28: note: in instantiation of function template specialization 'std::_IterOps<std::_ClassicAlgPolicy>::__iter_move<c10::impl::ListIterator<at::Tensor, std::__wrap_iter<c10::IValue *>> &>' requested here
          value_type __top(_Ops::__iter_move(__start));
                                 ^
      /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk/usr/include/c++/v1/__algorithm/make_heap.h:36:14: note: in instantiation of function template specialization 'std::__sift_down<std::_ClassicAlgPolicy, (lambda at /private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/torch/csrc/jit/runtime/register_ops_utils.cpp:67:7) &, c10::impl::ListIterator<at::Tensor, std::__wrap_iter<c10::IValue *>>>' requested here
              std::__sift_down<_AlgPolicy>(__first, __comp_ref, __n, __first + __start);
                   ^
      /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk/usr/include/c++/v1/__algorithm/partial_sort.h:39:8: note: in instantiation of function template specialization 'std::__make_heap<std::_ClassicAlgPolicy, (lambda at /private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/torch/csrc/jit/runtime/register_ops_utils.cpp:67:7) &, c10::impl::ListIterator<at::Tensor, std::__wrap_iter<c10::IValue *>>>' requested here
        std::__make_heap<_AlgPolicy>(__first, __middle, __comp);
             ^
      /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk/usr/include/c++/v1/__algorithm/partial_sort.h:66:12: note: in instantiation of function template specialization 'std::__partial_sort_impl<std::_ClassicAlgPolicy, (lambda at /private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/torch/csrc/jit/runtime/register_ops_utils.cpp:67:7) &, c10::impl::ListIterator<at::Tensor, std::__wrap_iter<c10::IValue *>>, c10::impl::ListIterator<at::Tensor, std::__wrap_iter<c10::IValue *>>>' requested here
            std::__partial_sort_impl<_AlgPolicy>(__first, __middle, __last, static_cast<__comp_ref_type<_Compare> >(__comp));
                 ^
      /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk/usr/include/c++/v1/__algorithm/sort.h:693:10: note: in instantiation of function template specialization 'std::__partial_sort<std::_ClassicAlgPolicy, (lambda at /private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/torch/csrc/jit/runtime/register_ops_utils.cpp:67:7), c10::impl::ListIterator<at::Tensor, std::__wrap_iter<c10::IValue *>>, c10::impl::ListIterator<at::Tensor, std::__wrap_iter<c10::IValue *>>>' requested here
          std::__partial_sort<_AlgPolicy>(__first, __last, __last, __comp);
               ^
      /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk/usr/include/c++/v1/__algorithm/sort.h:706:8: note: in instantiation of function template specialization 'std::__sort_impl<std::_ClassicAlgPolicy, c10::impl::ListIterator<at::Tensor, std::__wrap_iter<c10::IValue *>>, (lambda at /private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/torch/csrc/jit/runtime/register_ops_utils.cpp:67:7)>' requested here
        std::__sort_impl<_ClassicAlgPolicy>(std::move(__first), std::move(__last), __comp);
             ^
      /private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/torch/csrc/jit/runtime/register_ops_utils.cpp:64:8: note: in instantiation of function template specialization 'std::sort<c10::impl::ListIterator<at::Tensor, std::__wrap_iter<c10::IValue *>>, (lambda at /private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/torch/csrc/jit/runtime/register_ops_utils.cpp:67:7)>' requested here
        std::sort(
             ^
      /private/var/folders/23/82ghc8m51fx3c774b2v_hcxw0000gn/T/pip-install-vq_5cv6_/torch_339865c9ebef4d0e983fb65b581da962/aten/src/ATen/core/List.h:99:3: note: declared private here
        ListElementReference(ListElementReference&&) noexcept = default;
        ^
      1 error generated.

It sounds like this issue may have been fixed in torch 2.x (https://github.com/pytorch/pytorch/issues/100991) but for some reason I can't install that version:

(.venv) $ pip install torch==2.0.1
ERROR: Could not find a version that satisfies the requirement torch==2.0.1 (from versions: 1.13.1)
ERROR: No matching distribution found for torch==2.0.1
msimacek commented 6 months ago

PyTorch upstream doesn't bother shipping source packages at all. We patch pip to build few specific versions from their github releases. In the master we support torch 1.13.1 and 2.2.1. Please try the latest snapshot with pip install torch==2.2.1.

Note the tests on compatibility page are all done on Linux x86_64.

kalaracey commented 5 months ago

Thanks, that did the trick.