Can you help me setting environment variables?

[luxiant]

This is all I've done in terminal

wget https://download.pytorch.org/libtorch/cpu/libtorch-shared-with-deps-2.3.0%2Bcpu.zip
zip libtorch-shared-with-deps-2.3.0+cpu.zip
export LIBTORCH=/home/test/libtorch
export LIBTORCH_INCLUDE=$LIBTORCH/include:$LIBTORCH/include/torch/csrc/api/include
export LIBTORCH_LIB=$LIBTORCH/lib
export LIBTORCH_STATIC=1
export LIBTORCH_BYPASS_VERSION_CHECK=1

Then I made a rust-bert project that requires tch-rs. Here is the error when I do cargo run of this project.

...

   Compiling torch-sys v0.14.0
The following warnings were emitted during compilation:

warning: libtch/torch_api.cpp:1:9: fatal error: torch/csrc/autograd/engine.h: No such file or directory
warning:     1 | #include<torch/csrc/autograd/engine.h>
warning:       |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
warning: compilation terminated.
warning: ToolExecError: Command "c++" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-gdwarf-4" "-fno-omit-frame-pointer" "-m64" "-I" "/home/test/libtorch/include:/home/test/libtorch/include/torch/csrc/api/include/include" "-I" "/home/test/libtorch/include:/home/test/libtorch/include/torch/csrc/api/include/include/torch/csrc/api/include" "-Wl,-rpath=/home/test/libtorch/lib/lib" "-std=c++17" "-D_GLIBCXX_USE_CXX11_ABI=1" "-o" "/home/test/abstractive_summary/target/debug/build/torch-sys-ae666f3c5f62c0d3/out/19072f24a82f85ae-torch_api.o" "-c" "libtch/torch_api.cpp" with args c++ did not execute successfully (status code exit status: 1).cargo:warning=In file included from libtch/torch_api_generated.h:2,
warning:                  from libtch/torch_api_generated.cpp:2:
warning: libtch/torch_api.h:6:9: fatal error: torch/torch.h: No such file or directory
warning:     6 | #include<torch/torch.h>
warning:       |         ^~~~~~~~~~~~~~~
warning: compilation terminated.
warning: ToolExecError: Command "c++" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-gdwarf-4" "-fno-omit-frame-pointer" "-m64" "-I" "/home/test/libtorch/include:/home/test/libtorch/include/torch/csrc/api/include/include" "-I" "/home/test/libtorch/include:/home/test/libtorch/include/torch/csrc/api/include/include/torch/csrc/api/include" "-Wl,-rpath=/home/test/libtorch/lib/lib" "-std=c++17" "-D_GLIBCXX_USE_CXX11_ABI=1" "-o" "/home/test/abstractive_summary/target/debug/build/torch-sys-ae666f3c5f62c0d3/out/19072f24a82f85ae-torch_api_generated.o" "-c" "libtch/torch_api_generated.cpp" with args c++ did not execute successfully (status code exit status: 1).

error: failed to run custom build command for `torch-sys v0.14.0`

Caused by:
  process didn't exit successfully: `/home/test/abstractive_summary/target/debug/build/torch-sys-68f46a5126ca625c/build-script-build` (exit status: 1)
  --- stdout
  cargo:rerun-if-env-changed=LIBTORCH_USE_PYTORCH
  cargo:rerun-if-env-changed=LIBTORCH
  cargo:rerun-if-env-changed=LIBTORCH_INCLUDE
  cargo:rerun-if-env-changed=LIBTORCH_LIB
  cargo:rerun-if-env-changed=LIBTORCH_BYPASS_VERSION_CHECK
  cargo:rerun-if-env-changed=LIBTORCH_CXX11_ABI
  cargo:rerun-if-env-changed=LIBTORCH_STATIC
  cargo:rustc-link-search=native=/home/test/libtorch/lib/lib
  cargo:rerun-if-changed=libtch/fake_cuda_dependency.cpp
  cargo:rerun-if-changed=libtch/torch_python.cpp
  cargo:rerun-if-changed=libtch/torch_python.h
  cargo:rerun-if-changed=libtch/torch_api_generated.cpp
  cargo:rerun-if-changed=libtch/torch_api_generated.h
  cargo:rerun-if-changed=libtch/torch_api.cpp
  cargo:rerun-if-changed=libtch/torch_api.h
  cargo:rerun-if-changed=libtch/stb_image_write.h
  cargo:rerun-if-changed=libtch/stb_image_resize.h
  cargo:rerun-if-changed=libtch/stb_image.h
  cargo:libtorch_lib=/home/test/libtorch/lib/lib
  TARGET = Some("x86_64-unknown-linux-gnu")
  OPT_LEVEL = Some("0")
  HOST = Some("x86_64-unknown-linux-gnu")
  cargo:rerun-if-env-changed=CXX_x86_64-unknown-linux-gnu
  CXX_x86_64-unknown-linux-gnu = None
  cargo:rerun-if-env-changed=CXX_x86_64_unknown_linux_gnu
  CXX_x86_64_unknown_linux_gnu = None
  cargo:rerun-if-env-changed=HOST_CXX
  HOST_CXX = None
  cargo:rerun-if-env-changed=CXX
  CXX = None
  cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("true")
  CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
  cargo:rerun-if-env-changed=CXXFLAGS_x86_64-unknown-linux-gnu
  CXXFLAGS_x86_64-unknown-linux-gnu = None
  cargo:rerun-if-env-changed=CXXFLAGS_x86_64_unknown_linux_gnu
  CXXFLAGS_x86_64_unknown_linux_gnu = None
  cargo:rerun-if-env-changed=HOST_CXXFLAGS
  HOST_CXXFLAGS = None
  cargo:rerun-if-env-changed=CXXFLAGS
  CXXFLAGS = None
  cargo:warning=libtch/torch_api.cpp:1:9: fatal error: torch/csrc/autograd/engine.h: No such file or directory
  cargo:warning=    1 | #include<torch/csrc/autograd/engine.h>
  cargo:warning=      |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  cargo:warning=compilation terminated.
  exit status: 1
  cargo:warning=ToolExecError: Command "c++" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-gdwarf-4" "-fno-omit-frame-pointer" "-m64" "-I" "/home/test/libtorch/include:/home/test/libtorch/include/torch/csrc/api/include/include" "-I" "/home/test/libtorch/include:/home/test/libtorch/include/torch/csrc/api/include/include/torch/csrc/api/include" "-Wl,-rpath=/home/test/libtorch/lib/lib" "-std=c++17" "-D_GLIBCXX_USE_CXX11_ABI=1" "-o" "/home/test/abstractive_summary/target/debug/build/torch-sys-ae666f3c5f62c0d3/out/19072f24a82f85ae-torch_api.o" "-c" "libtch/torch_api.cpp" with args c++ did not execute successfully (status code exit status: 1).cargo:warning=In file included from libtch/torch_api_generated.h:2,
  cargo:warning=                 from libtch/torch_api_generated.cpp:2:
  cargo:warning=libtch/torch_api.h:6:9: fatal error: torch/torch.h: No such file or directory
  cargo:warning=    6 | #include<torch/torch.h>
  cargo:warning=      |         ^~~~~~~~~~~~~~~
  cargo:warning=compilation terminated.

  exit status: 1
  cargo:warning=ToolExecError: Command "c++" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-gdwarf-4" "-fno-omit-frame-pointer" "-m64" "-I" "/home/test/libtorch/include:/home/test/libtorch/include/torch/csrc/api/include/include" "-I" "/home/test/libtorch/include:/home/test/libtorch/include/torch/csrc/api/include/include/torch/csrc/api/include" "-Wl,-rpath=/home/test/libtorch/lib/lib" "-std=c++17" "-D_GLIBCXX_USE_CXX11_ABI=1" "-o" "/home/test/abstractive_summary/target/debug/build/torch-sys-ae666f3c5f62c0d3/out/19072f24a82f85ae-torch_api_generated.o" "-c" "libtch/torch_api_generated.cpp" with args c++ did not execute successfully (status code exit status: 1).
  exit status: 0

  --- stderr

  error occurred: Command "c++" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-gdwarf-4" "-fno-omit-frame-pointer" "-m64" "-I" "/home/test/libtorch/include:/home/test/libtorch/include/torch/csrc/api/include/include" "-I" "/home/test/libtorch/include:/home/test/libtorch/include/torch/csrc/api/include/include/torch/csrc/api/include" "-Wl,-rpath=/home/test/libtorch/lib/lib" "-std=c++17" "-D_GLIBCXX_USE_CXX11_ABI=1" "-o" "/home/test/abstractive_summary/target/debug/build/torch-sys-ae666f3c5f62c0d3/out/19072f24a82f85ae-torch_api_generated.o" "-c" "libtch/torch_api_generated.cpp" with args c++ did not execute successfully (status code exit status: 1).

warning: build failed, waiting for other jobs to finish...

There must be something wrong when setting environment variables, but I can't figure out. Can you help with this?

[bionicles]

for me it was an extra "include" and "torch" if you have a trailing one export LIBTORCH_INCLUDE=/home/bion/hax/externals/pytorch/torch/csrc/api no trailing /include because the build.rs might add that path part again non-idempotently

for autograd stuff, the issue could be they are in separate areas of the torch directory tree

(py310) [2024-05-24T08:29:36-0400] ~/hax/externals/pytorch/torch (main) $

fd engine.h csrc/autograd/engine.h csrc/autograd/python_engine.h csrc/distributed/autograd/engine/dist_engine.h (py310) [2024-05-24T08:29:46-0400] ~/hax/externals/pytorch/torch (main) $ fd torch.h csrc/api/include/torch/torch.h

if we only "include" /api/include/ then we're missing the /autograd/

https://github.com/LaurentMazare/tch-rs/blob/a90854b2135b91546ca6082418d478126e1f6a26/torch-sys/build.rs#L248C1-L249C89

sadly the build script isn't working right now with latest pytorch i guess, this line linked above might need to also include the

/pytorch/torch/csrc/autograd/

ugh i could try it locally

[ramintoosi]

I only set two variables like this

export LIBTORCH=/home/test/libtorch
export LD_LIBRARY_PATH=/home/test/libtorch/lib/:$LD_LIBRARY_PATH

and it works.