Rust/cargo dependencies are adding significant build time to existing pipelines

[jakevose]

@jakevose We can look into ways in which we can decrease the build time. Can you open a new issue and provide some details about how your builds work? E.g. spinning up new hosts in an autoscaling group and building efs-utils via the user data script? We can help come up with a temporary solution that mitigates the longer build time problem.

We operate in a multi-tenant infrastructure where project teams are provided a hardened base OS AMI by a DevSecOps team and subsequently run Packer via CI to layer additional updates, dependencies, frameworks, and code. Rust tooling is not provided in the base, but we are allowed to include it as part of our application builds.

Once we've completed our build phase, the AMI is saved and reused in CloudFormation across pre-production and eventually production environments following validation and approval.

We don't use EFS as a dependency on all of our downstream projects (and other tenants are mixed as well) so it's currently not possible for us to get Rust and EFS dependencies baked into the hardened base OS.

For the EFS portion of the build, we are installing dependencies and checking out the repository, followed by:

./build-deb.sh
DEBIAN_FRONTEND=noninteractive apt-get -y install ./build/amazon-efs-utils*deb

Rust is a relatively quick install, but the cargo build is taking around 6 additional minutes to compile the ~135 packages on the build instances with the EC2 classes we're allocated by the infrastructure team. For now, we've locked the dependency to v1.36.0 to avoid the additional framework and build time.

An additional concern raised by my team is that the Rust dependency effectively adds another maintenance vector with respect to security scans, version updates, and CVE remediations on systems which have not historically dealt with Rust tooling.

Thank you for the consideration and assistance.

[bmlong137]

I can confirm this issue. We did the same thing, locking down to v1.36.0. The increase in our case goes from less than a minute to a solid 30 minutes.

[fdo-imit]

The excess in our build times is causing our pipeline to repeatedly fail. At this point we will be looking at alternatives to efs-utils if we cannot even reasonably build our own code anymore.

[RyanStan]

Thank you @jakevose for the details. I think a solution here is for us to attach the release artifacts to this Github page, instead of having users clone the repository and build the package from source.

[justin-octo]

Perhaps we could just NOT use Rust for efs-proxy and write it in another language. :)

[brettburch]

I can confirm this issue. We did the same thing, locking down to v1.36.0. The increase in our case goes from less than a minute to a solid 30 minutes.

We have done the same as of Friday.

RUN apt-get ...
    git clone --depth 1 --branch v1.36.0 https://github.com/aws/efs-utils && \
    ...
    ./build-deb.sh && ... 
       ... install ./build/amazon-efs-utils*deb

Calling out --depth 1 as a way to also decrease clone time.

[norrisjeremy]

Is there some reason Amazon simply can't provide pre-built binaries via Yum & Apt repos? Seems like it would solve all of the headaches.