Bug: Cannot build projects with multiple Rust-based Lambdas and using Cargo workspaces

lmammino commented 3 months ago

Description:

When creating multiple lambda functions in Rust, it is a common practice to organise functions in independent cargo subprojects and then use a cargo workspace at the top level.

This is convenient because it allows to have shared crates that can be imported by multiple lambdas in the project.

Something that might look like this in terms of files structure:

- function_a/src/main.rs
- function_a/Cargo.toml
- function_b/src/main.rs
- function_b/Cargo.toml
- shared/src/lib.rs
- shared/Cargo.toml
- Cargo.toml <- root manifest (defines the workspace)
- template.yml <- sam template

When trying to build this project with sam build --beta-features it produces the following error:

[...]

RustCargoLambdaBuilder:RustCopyAndRename raised unhandled exception
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/aws-sam-cli/1.113.0/libexec/lib/python3.12/site-packages/aws_lambda_builders/workflow.py", line 372, in run
    action.execute()
  File "/opt/homebrew/Cellar/aws-sam-cli/1.113.0/libexec/lib/python3.12/site-packages/aws_lambda_builders/workflows/rust_cargo/actions.py", line 143,
in execute
    binary_path = self.binary_path()
                  ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/aws-sam-cli/1.113.0/libexec/lib/python3.12/site-packages/aws_lambda_builders/workflows/rust_cargo/actions.py", line 130,
in binary_path
    output = os.listdir(base)
             ^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/Users/luciano/Desktop/cargo-lambda-multi-test/function_a/target/lambda'

My understanding of the problem is the following:

When using workspaces all the targets are produced in the root folder of the project: so in the example above:

./target/lambda/function_a
./target/lambda/function_b

But the SAM integration with cargo-lambda is not aware of this and keeps looking for the binaries in:

./function_a/target/lambda/function_a
./function_b/target/lambda/function_b

And therefore the error mentioned above is raised.

Also, note that the current documentation doesn't seem to cover this particular case, so if this is fixed, it's probably worth adding a new example there.

Finally, as far as I know, cargo-lambda-cdk correctly support this particular use case.

Steps to reproduce:

I have created a sample repo here: https://github.com/lmammino/broken-sam-cargo-lambda-workspaces-example

Clone the repo: git clone git@github.com:lmammino/broken-sam-cargo-lambda-workspaces-example.git
cd into the cloned project: cd broken-sam-cargo-lambda-workspaces-example.git
Build with: sam build --beta-features --debug

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

OS: mac Os
sam --version: SAM CLI, version 1.113.0
AWS region: eu-west-1

{
  "version": "1.113.0",
  "system": {
    "python": "3.12.2",
    "os": "macOS-14.3.1-arm64-arm-64bit"
  },
  "additional_dependencies": {
    "docker_engine": "25.0.3",
    "aws_cdk": "2.122.0 (build 7e77e02)",
    "terraform": "1.3.9"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}

lmammino commented 3 months ago

Current workaround:

Build the lambdas manually and force them to be built in their own folders:

# build function_a
cd function_a
CARGO_TARGET_DIR=target cargo lambda build --release

cd ..

# build function_b
cd function_b
CARGO_TARGET_DIR=target cargo lambda build --release

# build with SAM (it will rebuild and place your lambdas in `./target/lambda` but then it will copy the files into the final build folder from the previous manual builds
sam build --beta-features --debug

lmammino commented 3 months ago

Even simpler workaround

I just realised that this works as well:

CARGO_TARGET_DIR=target sam build --beta-features

hawflau commented 3 months ago

Hi @lmammino Thanks for raising the issue and providing the workarounds. I believe the 2nd workaround is indeed what we should in the fix.

I'll move the issue to Lambda Builders since the fix will be there.

hawflau commented 3 months ago

Here's a brief description of what the fix will look like - in https://github.com/aws/aws-lambda-builders/blob/develop/aws_lambda_builders/workflows/rust_cargo/cargo_lambda.py#L72, add the env var CARGO_TARGET_DIR=target so the underlying cargo lambda build run will create the binary at the specified directory.

github-actions[bot] commented 2 months ago

Patch is released in v1.50.0. If you are AWS SAM CLI user, please wait for next AWS SAM CLI release. Closing

aws / aws-lambda-builders