Building rust-bert on docker

[brunohanss]

Hi, I have been trying to build my app on ubuntu using docker but i keep having major issues with libtorch

running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "/usr/libtorch/libtorch/include" "-I" "/usr/libtorch/libtorch/include/torch/csrc/api/include" "-Wl,-rpath=/usr/libtorch/libtorch/lib" "-std=c++14" "-D_GLIBCXX_USE_CXX11_ABI=1" "-o" "/nlp-utils/target/release/build/torch-sys-8060600507191f67/out/libtch/torch_api.o" "-c" "libtch/torch_api.cpp"
#0 115.5   running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "/usr/libtorch/libtorch/include" "-I" "/usr/libtorch/libtorch/include/torch/csrc/api/include" "-Wl,-rpath=/usr/libtorch/libtorch/lib" "-std=c++14" "-D_GLIBCXX_USE_CXX11_ABI=1" "-o" "/nlp-utils/target/release/build/torch-sys-8060600507191f67/out/libtch/fake_cuda_dependency.o" "-c" "libtch/fake_cuda_dependency.cpp"
#0 115.5   exit status: 0
#0 115.5   cargo:warning=In file included from libtch/torch_api.cpp:13:
#0 115.5   cargo:warning=libtch/torch_api_generated.cpp.h: In function 'void atg__flash_scaled_dot_product_attention(at::Tensor**, tensor, tensor, tensor, tensor, tensor, int64_t, int64_t, double, int)':     
#0 115.5   cargo:warning=libtch/torch_api_generated.cpp.h:1049:29: error: '_flash_scaled_dot_product_attention' is not a member of 'torch'
#0 115.5   cargo:warning= 1049 |     auto outputs__ = torch::_flash_scaled_dot_product_attention(*query, *key, *value, *cum_seq_q, *cum_seq_k, max_q, max_k, dropout_p, (bool)is_causal);
#0 115.5   cargo:warning=      |                             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#0 115.5   cargo:warning=libtch/torch_api.h:16:5: note: in definition of macro 'PROTECT'
#0 115.5   cargo:warning=   16 |     x \
#0 115.5   cargo:warning=      |     ^
#0 115.5   cargo:warning=libtch/torch_api_generated.cpp.h: In function 'void atg__lstm_mps_out(at::Tensor**, tensor, tensor, tensor, tensor, tensor, tensor, at::Tensor**, int, at::Tensor**, int, int, int64_t, double, int, int, int)':
#0 115.5   cargo:warning=libtch/torch_api_generated.cpp.h:1425:102: error: invalid initialization of reference of type 'const at::Tensor&' from expression of type 'std::vector<at::Tensor>'
#0 115.5   cargo:warning= 1425 |     auto outputs__ = torch::_lstm_mps_out(*out0, *out1, *out2, *out3, *out4, *input, of_carray_tensor(hx_data, hx_len), of_carray_tensor(params_data, params_len), (bool)has_biases, num_layers, dropout, (bool)train, (bool)bidirectional, (bool)batch_first);
#0 115.5   cargo:warning=      |                                                                                      ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~
#0 115.5   cargo:warning=libtch/torch_api.h:16:5: note: in definition of macro 'PROTECT'

Here is my actual Dockerfile

FROM rust:1.69.0

RUN USER=root cargo new --bin nlp-utils
WORKDIR /nlp-utils
RUN apt update && apt install -y gcc libc6-dev git build-essential zlib1g-dev libbz2-dev pkg-config cmake python3 wget curl unzip gnupg 

# Download and extract libtorch
RUN wget https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-2.0.0%2Bcpu.zip -P /tmp/libtorch
RUN unzip /tmp/libtorch/libtorch-cxx11-abi-shared-with-deps-2.0.0+cpu.zip -d /usr/libtorch
ENV LIBTORCH /usr/libtorch/libtorch
ENV LD_LIBRARY_PATH ${LIBTORCH}/lib:$LD_LIBRARY_PATH
ENV LIBTORCH_INCLUDE /usr/libtorch/libtorch
ENV LIBTORCH_LIB /usr/libtorch/libtorch

# Copy the manifests
COPY ./Cargo.lock ./Cargo.lock
COPY ./Cargo.toml ./Cargo.toml

# Build only the dependencies to cache them
RUN cargo build --release
RUN rm src/*.rs

# Copy the source code
COPY ./src ./src

# Set the appropriate include path for libtorch
# ENV CPLUS_INCLUDE_PATH ${LIBTORCH}/include/torch/csrc/api/include

# Build for release
RUN cargo install --path .

CMD ["nlp-utils"]

# 4. Now that the dependency is built, copy your source code
COPY ./src ./src

# 5. Build for release.

RUN rm .\/target/release/nlp-utils
RUN cargo install --path .

CMD ["nlp-utils"]

Does someone have a valid Dockerfile ready to use or any advice to make my file works ? That would be greatly appreciated

[brunohanss]

Just found another dockerfile using python here : https://github.com/guillaume-be/rust-bert/issues/261

But i get an error running this command error occurred: Command "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-I" "/opt/conda/envs/dev/lib/python3.8/site-packages/torch/include" "-I" "/opt/conda/envs/dev/lib/python3.8/site-packages/torch/include/torch/csrc/api/include" "-Wl,-rpath=/opt/conda/envs/dev/lib/python3.8/site-packages/torch/lib" "-std=c++14" "-D_GLIBCXX_USE_CXX11_ABI=0" "-o" "/target/release/build/torch-sys-15cc74e69ce914a1/out/libtch/torch_api.o" "-c" "libtch/torch_api.cpp" with args "c++" did not execute successfully

[samsafay]

FROM rust:latest as builder

# Install CMake
RUN apt update && apt full-upgrade -y && apt autoremove -y

# Install libtorch
RUN wget https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-2.0.1%2Bcpu.zip && \
  unzip libtorch-cxx11-abi-shared-with-deps-2.0.1+cpu.zip && \
  rm libtorch-cxx11-abi-shared-with-deps-2.0.1+cpu.zip 

ENV LIBTORCH=/libtorch
ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$LIBTORCH/lib

RUN mkdir -p /app

# Copy the Warp Rust project into the container
WORKDIR /app
COPY . .
RUN cargo build --release

FROM rust:slim
RUN apt update && apt full-upgrade -y && apt autoremove -y

COPY --from=builder /libtorch .

RUN mkdir /app
WORKDIR /app
COPY --from=builder /app/target/release/dl-webserver .

ENV LIBTORCH=/libtorch
ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$LIBTORCH/lib

EXPOSE 3030
CMD ["/app/dl-webserver"]

[samsafay]

This is the Dockerfile that it works with the recommended example rust-dl-webserver.

Just clone the example.

git clone https://github.com/epwalsh/rust-dl-webserver.git
cd rust-dl-webserver

Then update the cargo.toml file.

[package]
name = "dl-webserver"
version = "0.1.0"
authors = ["epwalsh <epwalsh10@gmail.com>"]
edition = "2018"

[dependencies]
batched-fn = { version = "0.2.4" }
tokio = { version = "1.14.0", features = ["full"] }
warp = "0.3.5"
tch = "0.13.0"
rust-bert = "0.21.0"
env_logger = "0.10.0"
log = "0.4"
serde = "1.0"
dirs = "5.0.1"
anyhow="1.0.72"

put the Dockerfile in the root directory of the cloned project.

Then execute the following:

docker build -t your/imagename . && docker push your/imagename

docker run -p 3030:3030 your/imagename

open another terminal and execute this query

curl -d '{"text":"Hello, World!"}' \
    -H "Content-Type: application/json" \
    http://localhost:3030/generate &
curl -d '{"text":"Stay at home"}' \
    -H "Content-Type: application/json" \
    http://localhost:3030/generate &
curl -d '{"text":"Wash your hands"}' \
    -H "Content-Type: application/json" \
    http://localhost:3030/generate &
curl -d '{"text":"Do not touch your face"}' \
    -H "Content-Type: application/json" \
    http://localhost:3030/generate &

Note: at this point you should wait a few seconds depending on your internet speed till the entire model is downloaded. And then you will get a respond back...

[brunohanss]

Great !! Thank you @samsafay