docker compose fails while building attestation-service - Too many open files (os error 24)

[niteeshkd]

When I run docker compose up -d to start trustee containers on Ubuntu 22.04, it fails while building attestation service container with the following messages. But, i don't notice this problem on Ubuntu 20.04.

$ docker compose up -d
[+] Running 4/4
 # keyprovider 3 layers [###]      0B/0B      Pulled                                                                                                                        2.6s 
   # 99803d4b97f3 Pull complete                                                                                                                                             0.5s 
   # 465683745186 Pull complete                                                                                                                                             2.1s 
   # 410b84ec302a Pull complete                                                                                                                                             0.5s 
[+] Building 97.2s (27/31)                                                                                                                                        docker:default
 => [rvps internal] load build definition from Dockerfile                                                                                                                   0.0s
 => => transferring dockerfile: 663B                                                                                                                                        0.0s
 => [rvps internal] load metadata for docker.io/library/debian:latest                                                                                                       0.2s
 => [as internal] load metadata for docker.io/library/rust:latest                                                                                                           0.4s
 => [rvps auth] library/rust:pull token for registry-1.docker.io                                                                                                            0.0s
 => [rvps auth] library/debian:pull token for registry-1.docker.io                                                                                                          0.0s
 => [rvps internal] load .dockerignore                                                                                                                                      0.0s
 => => transferring context: 74B                                                                  
...
 => => writing image sha256:52861869cae6044f194cb8fd8cb8b8925990742ae4804259cb23f9bffa03cc56                                                                                0.0s 
 => => naming to docker.io/library/trustee-rvps                                                                                                                             0.0s 
 => [as internal] load build definition from Dockerfile.as-grpc                                                                                                             0.0s 
 => => transferring dockerfile: 2.12kB                         
 ...
 => [as builder 5/7] RUN apt-get update && apt-get install -y protobuf-compiler clang libtss2-dev                                                                          16.3s
 => [as builder 6/7] RUN curl -L https://download.01.org/intel-sgx/sgx_repo/ubuntu/intel-sgx-deb.key | tee intel-sgx-deb.key | apt-key add - &&     echo 'deb [arch=amd64]  9.5s
 => ERROR [as builder 7/7] RUN cargo install --path attestation-service/attestation-service --bin grpc-as --features grpc-bin --locked                                     26.9s
------
 > [as builder 7/7] RUN cargo install --path attestation-service/attestation-service --bin grpc-as --features grpc-bin --locked:
0.545 warning: virtual workspace defaulting to `resolver = "1"` despite one or more workspace members being on edition 2021 which implies `resolver = "2"`
0.545 note: to keep the current resolver, specify `workspace.resolver = "1"` in the workspace root's manifest
0.545 note: to use the edition 2021 resolver, specify `workspace.resolver = "2"` in the workspace root's manifest
0.545 note: for more details see https://doc.rust-lang.org/cargo/reference/resolver.html#resolver-versions
0.562 warning: virtual workspace defaulting to `resolver = "1"` despite one or more workspace members being on edition 2021 which implies `resolver = "2"`
0.562 note: to keep the current resolver, specify `workspace.resolver = "1"` in the workspace root's manifest
0.562 note: to use the edition 2021 resolver, specify `workspace.resolver = "2"` in the workspace root's manifest
0.562 note: for more details see https://doc.rust-lang.org/cargo/reference/resolver.html#resolver-versions
0.569   Installing attestation-service v0.1.0 (/usr/src/attestation-service/attestation-service/attestation-service)
0.583     Updating crates.io index
...
23.67   Downloaded cipher v0.4.4
23.67   Downloaded bindgen v0.60.1
23.68   Downloaded proc-macro2 v1.0.78
23.68   Downloaded prettyplease v0.1.25
23.75    Compiling proc-macro2 v1.0.78
...
23.82    Compiling data-encoding v2.5.0
23.82    Compiling language-tags v0.3.2
23.82    Compiling pathdiff v0.2.1
23.89    Compiling tracing-core v0.1.32
23.94    Compiling generic-array v0.14.7
23.94 error: could not compile `clap_lex` (lib)
23.94 
23.94 Caused by:
23.94   could not execute process `/usr/local/rustup/toolchains/1.76.0-x86_64-unknown-linux-gnu/bin/rustc --crate-name clap_lex --edition=2021 /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/clap_lex-0.7.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=bd6601e90003c752 -C extra-filename=-bd6601e90003c752 --out-dir /usr/src/attestation-service/target/release/deps -L dependency=/usr/src/attestation-service/target/release/deps --cap-lints allow` (never executed)
23.95 
23.95 Caused by:
23.95   Too many open files (os error 24)
23.95 warning: build failed, waiting for other jobs to finish...
23.95 error: could not compile `linux-raw-sys` (lib)
23.95 
23.95 Caused by:
23.95   could not execute process `/usr/local/rustup/toolchains/1.76.0-x86_64-unknown-linux-gnu/bin/rustc --crate-name linux_raw_sys --edition=2021 /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/linux-raw-sys-0.4.13/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg 'feature="elf"' --cfg 'feature="errno"' --cfg 'feature="general"' --cfg 'feature="ioctl"' --cfg 'feature="no_std"' -C metadata=18ce2bdc37be49e4 -C extra-filename=-18ce2bdc37be49e4 --out-dir /usr/src/attestation-service/target/release/deps -L dependency=/usr/src/attestation-service/target/release/deps --cap-lints allow` (never executed)
23.95 
23.95 Caused by:
23.96   Too many open files (os error 24)
...
26.69 error: failed to compile `attestation-service v0.1.0 (/usr/src/attestation-service/attestation-service/attestation-service)`, intermediate artifacts can be found at `/usr/src/attestation-service/target`.
26.69 To reuse those artifacts with a future compilation, set the environment variable `CARGO_TARGET_DIR` to that path.
------
failed to solve: process "/bin/sh -c cargo install --path attestation-service/attestation-service --bin grpc-as --features grpc-bin --locked" did not complete successfully: exit code: 101

[niteeshkd]

When I add the argument CARGO_TARGET_DIR in docker-compose.yml to reuse the compiled dependencies as follows, it is working fine.

$ git diff
diff --git a/docker-compose.yml b/docker-compose.yml
index 8e14c38..ba75201 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -3,6 +3,8 @@ services:
   kbs:
     build:
       context: .
+      args:
+      - CARGO_TARGET_DIR=/tmp
       dockerfile: ./kbs/docker/Dockerfile.coco-as-grpc
     #image: ghcr.io/confidential-containers/key-broker-service:latest
     command: [
@@ -23,6 +25,8 @@ services:
   as:
     build:
       context: .
+      args:
+      - CARGO_TARGET_DIR=/tmp
       dockerfile: ./attestation-service/Dockerfile.as-grpc
     #image: ghcr.io/confidential-containers/attestation-service:latest
     ports:
@@ -46,6 +50,8 @@ services:
     #image: ghcr.io/confidential-containers/reference-value-provider-service:latest
     build:
       context: .
+      args:
+      - CARGO_TARGET_DIR=/tmp
       dockerfile: ./attestation-service/rvps/Dockerfile
     restart: always # keep the server running
     ports:

[niteeshkd]

It seems use of CARGO_TARGET_DIR could save some time in runningdocker compose up -d. On Ubuntu 20.04, i noticed ~25 sec saving of time.

Without using CARGO_TARGET_DIR:

$ time docker compose up -d
...
real    4m41.491s
user    0m3.891s
sys     0m2.740s

After killing the trustee containers, removing their images and then starting the trustee containers with CARG_TARGET_DIR:

$ time docker compose up -d
...
real    4m16.658s
user    0m3.764s
sys     0m2.430s

[Xynnn007]

Hi @niteeshkd , I am not sure if it is cause by the OS number limit of opened files. Could you try ulimit -S and ulimit -H to check the limited number of opened file handlers on Ubuntu 22.04 and try to make it bigger by ulimit -n <a-bigger-number>

[niteeshkd]

Hi @Xynnn007 , increasing the number of file handlers for the user on the host does not help. It shows the same error. I tested it by increasing number of file handlers from default (i.e. 1024) to 4096, 524288, 1048576 (i.e. default maximum).

$ ulimit -S
unlimited

$ ulimit -H
unlimited

$ ulimit -n
1024

$ ulimit -Sn
1024

$ ulimit -Hn
1048576

$ ulimit -n 4096
$ ulimit -n
4096
$ docker compose up -d
...
23.99 Caused by:
23.99   Too many open files (os error 24)
27.29 error: failed to compile `attestation-service v0.1.0 (/usr/src/attestation-service/attestation-service/attestation-service)`, intermediate artifacts can be found at `/usr/src/attestation-service/target`.
27.29 To reuse those artifacts with a future compilation, set the environment variable `CARGO_TARGET_DIR` to that path.
------
failed to solve: process "/bin/sh -c cargo install --path attestation-service/attestation-service --bin grpc-as --features grpc-bin --locked" did not complete successfully: exit code: 101

$ ulimit -n 524288
$ ulimit -n
524288
$ docker compose up -d
...
22.35 Caused by:
22.35   Too many open files (os error 24)
25.57 error: failed to compile `attestation-service v0.1.0 (/usr/src/attestation-service/attestation-service/attestation-service)`, intermediate artifacts can be found at `/usr/src/attestation-service/target`.
25.57 To reuse those artifacts with a future compilation, set the environment variable `CARGO_TARGET_DIR` to that path.
------
failed to solve: process "/bin/sh -c cargo install --path attestation-service/attestation-service --bin grpc-as --features grpc-bin --locked" did not complete successfully: exit code: 101

$ ulimit -n 1048576
$ ulimit -n
1048576
$ docker compose up -d
...
21.14 Caused by:
21.14   Too many open files (os error 24)
24.06 error: failed to compile `attestation-service v0.1.0 (/usr/src/attestation-service/attestation-service/attestation-service)`, intermediate artifacts can be found at `/usr/src/attestation-service/target`.
24.06 To reuse those artifacts with a future compilation, set the environment variable `CARGO_TARGET_DIR` to that path.
------
failed to solve: process "/bin/sh -c cargo install --path attestation-service/attestation-service --bin grpc-as --features grpc-bin --locked" did not complete successfully: exit code: 101

[niteeshkd]

I tried specifying the number of open file handlers in the docker compose file. It does not seem helping either.

$ git diff
diff --git a/docker-compose.yml b/docker-compose.yml
index 8e14c38..c573d2a 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -11,6 +11,8 @@ services:
         "/etc/kbs-config.toml",
       ]
     restart: always # keep the server running
+    ulimits:
+        nofile: 524288
     ports:
       - "8080:8080"
     volumes:
@@ -28,6 +30,8 @@ services:
     ports:
     - "50004:50004"
     restart: always
+    ulimits:
+        nofile: 524288
     volumes:
     - ./kbs/data/attestation-service:/opt/confidential-containers/attestation-service:rw
     - ./kbs/config/as-config.json:/etc/as-config.json:rw
@@ -48,6 +52,8 @@ services:
       context: .
       dockerfile: ./attestation-service/rvps/Dockerfile
     restart: always # keep the server running
+    ulimits:
+        nofile: 524288
     ports:
       - "50003:50003"
     volumes:

$ docker compose up -d
...
23.96 Caused by:
23.96   Too many open files (os error 24)
27.30 error: failed to compile `attestation-service v0.1.0 (/usr/src/attestation-service/attestation-service/attestation-service)`, intermediate artifacts can be found at `/usr/src/attestation-service/target`.
27.30 To reuse those artifacts with a future compilation, set the environment variable `CARGO_TARGET_DIR` to that path.
------
failed to solve: process "/bin/sh -c cargo install --path attestation-service/attestation-service --bin grpc-as --features grpc-bin --locked" did not complete successfully: exit code: 101

@fitzthum

[Xynnn007]

Hi @niteeshkd

I test it again in a new VM on alibaba cloud with 8 vCPU & 16 GB RAM. Ubuntu 22.04.

Only installed the newest docker following https://docs.docker.com/engine/install/ubuntu/ and rust

Then clone trustee repo and run docker compose up -d

And the building process succeeded.

I am afraid that something wrong with your environment. Try to remove image rust:latest which AS' dockerfile uses and let it download the real latest.

Another way is to run the commands of Dockerfile.as-grpc inside a new docker container to check if it works step by step to locate the error.

[niteeshkd]

Another way is to run the commands of Dockerfile.as-grpc inside a new docker container to check if it works step by step to locate the error.

I have tested this way. It works this way.

[Xynnn007]

@niteeshkd Oh, well. That seems more like an execution environment issue than the specific build code in the repo

[fitzthum]

@niteeshkd can we close this?

[niteeshkd]

@niteeshkd can we close this?

I think we can close it for now.