How to build on M1 Mac? (or is it possible?)

[i10416]

(edited) using libtorch 1.10.2( https://download.pytorch.org/libtorch/cpu/libtorch-macos-1.10.2.zip) on arm64 darwin

[error] ld: symbol(s) not found for architecture arm64
[error] clang-11: error: linker command failed with exit code 1 (use -v to see invocation)

using libtorch 1.13.1(https://download.pytorch.org/libtorch/cpu/libtorch-macos-1.13.1.zip) on arm64 darwin

ATen/CUDAGeneratorImpl.h' file not found

hindsight

• libtorch 1.13.1, which I tried to use, and 1.10.2, which is used in CI, have different include paths.
• with libtorch 1.10.2, linker still raises error on arm64 arch.

---

environment

system_profiler SPSoftwareDataType SPHardwareDataType 
Software:

    System Software Overview:

      System Version: macOS 12.5 (21G72)
      Kernel Version: Darwin 21.6.0
      ...
    Hardware:

      Hardware Overview:
      Model Name: MacBook Pro
      Model Identifier: MacBookPro18,2
      Chip: Apple M1 Max

SWIG Version 4.0.2

Compiled with clang++ [aarch64-apple-darwin22.1.0]

Configured options: +pcre

python --version
Python 3.7.15
import platform
print(platform.platform())
Darwin-21.6.0-arm64-arm-64bit

java --version
openjdk 17.0.3 2022-04-19 LTS
OpenJDK Runtime Environment Zulu17.34+19-CA (build 17.0.3+7-LTS)
OpenJDK 64-Bit Server VM Zulu17.34+19-CA (build 17.0.3+7-LTS, mixed mode, sharing)

reproduction

What I did for now are the followings

1. prepare swig4, Python with required packages(pyyaml, typing-extensions, setuptools,etc.), Java(Azul Systems, Inc. Java 17.0.3), bazel
2. run git clone git@github.com:microsoft:scala_torch.git --recursive
3. download libtorch from https://download.pytorch.org/libtorch/cpu/libtorch-macos-1.13.1.zip and export TORCH_DIR path
4. run bazel build generated_cpp at pytorch dir to get Declarations.yaml for bindgen.py
5. run python3 -m tools.codegen.gen -s aten/src/ATen -d torch/share/ATen
6. put Declarations.yaml from scala_torch/pytorch/bazel-bin/aten/src/ATen/Declarations.yaml to pytorch/torch/share/ATen/Declarations.yaml
7. run sbt compile

and I got

[info] Building library with native build tool CMake
[info] Using CMake version 3.22.3
[info] -- Static Pytorch: OFF
[info] -- torch dir?: /path/to/libtorch
[info] -- final torch dir: /path/to/libtorch
[info] -- CMAKE_PREFIX_PATH: 
[info] -- CMAKE_MODULE_PATH: /path/to/libtorch/../cmake/Modules;/usr/local/cmake/Modules/share/cmake-3.14
[info] -- JNI include directories: /path/to/zulu17.34.19-ca-jdk-17.0.3/include;/path/to/zulu17.34.19-ca-jdk-17.0.3/include;/path/to/zulu17.34.19-ca-jdk-17.0.3/include
[info] -- Torch include directories: /path/to/libtorch/include;/path/to/libtorch/include/torch/csrc/api/include
[info] -- torch libs : torch;torch_library;/path/to/libtorch/lib/libc10.dylib;/path/to/libtorch/lib/libkineto.a
[info] -- Configuring done
[info] -- Generating done
[info] -- Build files have been written to: /path/to/scala_torch/swig/target/native/arm64-darwin/build
[info] [ 50%] Building CXX object CMakeFiles/torch_swig0.dir/path/to/scala_torch/swig/target/src_managed/native/torch_swig.cxx.o
[error] /path/to/scala_torch/swig/target/src_managed/native/torch_swig.cxx:706:10: fatal error: 'ATen/CUDAGeneratorImpl.h' file not found
[error] #include <ATen/CUDAGeneratorImpl.h>
[error]          ^~~~~~~~~~~~~~~~~~~~~~~~~~
[error] 1 error generated.
[error] make[2]: *** [CMakeFiles/torch_swig0.dir/build.make:76: CMakeFiles/torch_swig0.dir/path/to/scala_torch/swig/target/src_managed/native/torch_swig.cxx.o] Error 1
[error] make[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/torch_swig0.dir/all] Error 2
[error] make: *** [Makefile:136: all] Error 2

[i10416]

Ah, I overlooked local development guide. I'll try again.

[i10416]

Following https://github.com/microsoft/scala_torch#local-development-macos, I still got error(libtorch 1.13.1)

 /path/to/scala_torch/swig/target/src_managed/native/torch_swig.cxx:706:10: fatal error: 'ATen/CUDAGeneratorImpl.h' file not found
[error] #include <ATen/CUDAGeneratorImpl.h>
[error]          ^~~~~~~~~~~~~~~~~~~~~~~~~~

sbt runs the following cmd

swig -DSWIGWORDSIZE64 -v -c++ -java -package com.microsoft.scalatorch.torch.internal -Ipath/to/scala_torch/libtorch/include -I/path/to/scala_torch/libtorch/include/torch/csrc/api/include -o /path/to/scala_torch/swig/target/src_managed/native/torch_swig.cxx -outdir /path/to/scala_torch/swig/target/src_managed/com/microsoft/scalatorch/torch/internal /path/to/scala_torch/swig/src/main/swig/torch_swig.i

and there exists CUDAGeneratorImpl.h

find pytorch  -name CUDAGeneratorImpl.h
pytorch/aten/src/ATen/CUDAGeneratorImpl.h

find libtorch -name CUDAGeneratorImpl.h
libtorch/include/ATen/cuda/CUDAGeneratorImpl.h

[i10416]

libtorch/include/ATen/cuda/CUDAGeneratorImpl.h

CUDAGeneratorImpl.h is in ATen/cuda/CUDAGeneratorImpl.h🤔 Perhaps, libtorch has different package structure depending on torch version?

[i10416]

downgrading libtorch to 1.10.2( https://download.pytorch.org/libtorch/cpu/libtorch-macos-1.10.2.zip) solved 'ATen/CUDAGeneratorImpl.h' file not found error, but then I got linker error.

[error] ld: symbol(s) not found for architecture arm64
[error] clang-11: error: linker command failed with exit code 1 (use -v to see invocation)

[i10416]

It seems arm64 is not supported?