Build issues with rust-bert and PyO3

[diptanu]

Hi @guillaume-be

I am using rust_bert from master because I wanted the tokenizer API to be exposed from pipelines. After the latest commit on master the build started failing. I am depending on rust-bert like this -

rust-bert = {git = "https://github.com/guillaume-be/rust-bert.git", branch= "master", features=["download-libtorch"]}

Getting the error -

error: failed to run custom build command for `torch-sys v0.13.0`

Caused by:
  process didn't exit successfully: `/home/diptanuc/Projects/indexify/target/debug/build/torch-sys-74e0cb8dc55c224f/build-script-build` (exit status: 1)
  --- stdout
  cargo:rerun-if-env-changed=LIBTORCH_USE_PYTORCH
  cargo:rerun-if-env-changed=LIBTORCH

  --- stderr
  Error:
  Cannot find a libtorch install, you can either:
  - Install libtorch manually and set the LIBTORCH environment variable to appropriate path.
  - Use a system wide install in /usr/lib/libtorch.so.
  - Use a Python environment with PyTorch installed by setting LIBTORCH_USE_PYTORCH=1

  See the readme for more details:
  https://github.com/LaurentMazare/tch-rs/blob/main/README.md

warning: build failed, waiting for other jobs to finish...

I tried setting LIBTORCH to a downloaded 2.0.1 version but the build is failing on that one after the latest commit as well.

[guillaume-be]

It looks like the feature is not enabled for torch-sys since you are seeing this message: https://github.com/LaurentMazare/tch-rs/blob/d8b6ad6ad7fefa384fa2685f3190d122bf90f7b3/torch-sys/build.rs#L264. The CI pipeline seems to be working when passing the feature.

For troubleshooting purposes, does the issue still occur if you add to your toml the following:

torch-sys= { version = "0.13.0" features = ["download-libtorch"] }

[diptanu]

@guillaume-be Looks like just upgrading rust-bert to master doesn't fail but when I introduce pyo3 libtorch is not getting compiled.

So basically adding fails the build -

pyo3 = {version = "^0", features = ["auto-initialize"]}

The error is -

       Compiling indexify v0.1.0 (/home/diptanuc/Projects/indexify)
       Compiling pyo3-ffi v0.18.3
       Compiling pyo3 v0.18.3
    error: linking with `cc` failed: exit status: 1
      |
      = note: "cc" "-m64" "/tmp/rustcVNhMXf/symbols.o" "/home/diptanuc/Projects/indexify/target/debug/deps/indexify-148f38010fc4be06.11pv5um63o7zvpk5.rcgu.o" "/home/diptanuc/Projects/indexify/target/debug/deps/indexify-148f38010fc4be06.15q19iny
      = note: /usr/bin/ld: /home/diptanuc/Projects/indexify/target/debug/deps/libtorch_sys-7567d92f0dab79f1.rlib(torch_api.o): in function `c10::Device::validate()':

  .... 
      = note: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
     = note: use the `-l` flag to specify native libraries to link
     = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#cargorustc-link-libkindname)

[diptanu]

Indeed this was a pyo3 issue - https://pyo3.rs/main/faq#i-cant-run-cargo-test-or-i-cant-build-in-a-cargo-workspace-im-having-linker-issues-like-symbol-not-found-or-undefined-reference-to-_pyexc_systemerror

It surfaces as libtorch compilation issue. I am renaming this issue in case someone comes here facing the same problem.