Bug: sam local start-api not working in a docker-out-of-docker setup, due to an empty /var/task directory.

git-noise commented 2 months ago

Description:

When running sam local start-api in a docker-out-of-docker setup, it seems that the /var/task folder mounted into the lambda-executing container is not populated with the zip containing the lambda binary.

Steps to reproduce:

Create a simple JS hello-world, zip it and drop to a common path - here /tmp/lambda-bins

Create a basic setup with an API Gateway and one lambda endpoint:

Resources:
HelloWorld:
Type: AWS::Serverless::Function
Properties:
  FunctionName: HelloWorld
  Handler: index.handler
  Runtime: nodejs16.x
  CodeUri: ./lambda-hello-world.zip

ApiGateway:
Type: AWS::Serverless::Api
Properties:
  Name: api-gateway
  StageName: prod
  DefinitionBody:
    swagger: '2.0'
    paths:
      /hello:
        get:
          produces:
            - application/json
          responses:
            200:
              description: '200 response'
          x-amazon-apigateway-integration:
            httpMethod: GET
            type: aws_proxy
            uri:
              Fn::Sub: arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${HelloWorld.Arn}/invocations

Create a simple docker container with the aws-sam-cli.
```
FROM python:3.10-alpine
```

RUN apk add --no-cache --virtual build-deps build-base libffi-dev gcc \ && pip install --no-cache-dir aws-sam-cli==1.116.0 \ && pip3 uninstall --yes pip \ && apk del build-deps

4. Create a compose file to spin everything up:

networks: sam_subnet: driver: bridge name: test-sub

services: sam-lambda: build: . working_dir: /opt/data ports:

3000:3000 volumes:
Docker
/var/run/docker.sock:/var/run/docker.sock
SAM template
./template.yaml:/opt/data/template.yaml
Directory on the host with the lambda binaries
/tmp/lambda-bins:/tmp/lambda-bins networks:
sam_subnet command: [ "sam", "local", "start-api", "--debug", "--template", "template.yaml", "--host", "0.0.0.0", "--docker-network", "test-sub", "--docker-volume-basedir", "/tmp/lambda-bins", "--container-host-interface", "0.0.0.0", "--container-host", "host.docker.internal", ] extra_hosts:

"host.docker.internal:host-gateway"


A few points of interest:
- the working directory for the aws-sam-cli container is `/opt/data`. It will only contain the sam template.
- the lambda binaries are shared via another mounted folder `/tmp/lambda-bins`

Call the lambda function using curl:
```
curl -v http://localhost:3000/hello
```

Observed result:

The lambda function fails to execute with the following:

sam-lambda-1  | 2024-05-06 17:41:09,702 | Found one Lambda function with name 'HelloWorld'
sam-lambda-1  | 2024-05-06 17:41:09,702 | Invoking index.handler (nodejs16.x)
sam-lambda-1  | 2024-05-06 17:41:09,702 | No environment variables found for function 'HelloWorld'
sam-lambda-1  | 2024-05-06 17:41:09,702 | Loading AWS credentials from session with profile 'None'
sam-lambda-1  | 2024-05-06 17:41:11,714 | Resolving code path. Cwd=/tmp/lambda-bins, CodeUri=/tmp/lambda-bins/lambda-hello-world.zip
sam-lambda-1  | 2024-05-06 17:41:11,714 | Resolved absolute path to code is /tmp/lambda-bins/lambda-hello-world.zip
sam-lambda-1  | 2024-05-06 17:41:11,714 | Resolving code path. Cwd=/opt/data, CodeUri=/tmp/lambda-bins/lambda-hello-world.zip
sam-lambda-1  | 2024-05-06 17:41:11,714 | Resolved real code path to /tmp/lambda-bins/lambda-hello-world.zip
sam-lambda-1  | 2024-05-06 17:41:11,715 | Decompressing /tmp/lambda-bins/lambda-hello-world.zip
sam-lambda-1  | 2024-05-06 17:41:12,404 | Local image is up-to-date
sam-lambda-1  | 2024-05-06 17:41:12,409 | Checking free port on 0.0.0.0:5345
sam-lambda-1  | 2024-05-06 17:41:12,411 | Using local image: public.ecr.aws/lambda/nodejs:16-rapid-x86_64.
sam-lambda-1  | 
sam-lambda-1  | 2024-05-06 17:41:12,411 | Mounting /tmp/tmpqn5dh6cm as /var/task:ro,delegated, inside runtime container
sam-lambda-1  | 2024-05-06 17:41:12,833 | Starting a timer for 3 seconds for function 'HelloWorld'
sam-lambda-1  | 2024-05-06 17:41:12,833 | Getting lock for the key host.docker.internal-5345
sam-lambda-1  | 2024-05-06 17:41:12,833 | Waiting to retrieve the lock (host.docker.internal-5345) to start invocation
sam-lambda-1  | START RequestId: 37f0705f-d4e9-4a29-922f-60307d74dc08 Version: $LATEST
sam-lambda-1  | 2024-05-06T17:41:12.941Z    undefined   ERROR   Uncaught Exception  {"errorType":"Runtime.ImportModuleError","errorMessage":"Error: Cannot find module 'index'\nRequire stack:\n- /var/runtime/index.mjs","stack":["Runtime.ImportModuleError: Error: Cannot find module 'index'","Require stack:","- /var/runtime/index.mjs","    at _loadUserApp (file:///var/runtime/index.mjs:1087:17)","    at async Object.UserFunction.js.module.exports.load (file:///var/runtime/index.mjs:1119:21)","    at async start (file:///var/runtime/index.mjs:1282:23)","    at async file:///var/runtime/index.mjs:1288:1"]}

As one can see, although the binary seems properly located as per this line: sam-lambda-1 | 2024-05-06 17:41:11,715 | Decompressing /tmp/lambda-bins/lambda-hello-world.zip

It seems that /var/task ends up being empty in the lambda-executing container.

Expected result:

We would expect the binary to be mounted into the lambda-executing container, and then the lambda function to properly execute.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

OS: Linux Ubuntu 24.04
sam --version: 1.116.0
AWS region: None

# Paste the output of `sam --info` here
sam-lambda-1  | {
sam-lambda-1  |   "version": "1.116.0",
sam-lambda-1  |   "system": {
sam-lambda-1  |     "python": "3.10.14",
sam-lambda-1  |     "os": "Linux-6.8.0-31-generic-x86_64-with"
sam-lambda-1  |   },
sam-lambda-1  |   "additional_dependencies": {
sam-lambda-1  |     "docker_engine": "26.1.1",
sam-lambda-1  |     "aws_cdk": "Not available",
sam-lambda-1  |     "terraform": "Not available"
sam-lambda-1  |   },
sam-lambda-1  |   "available_beta_feature_env_vars": [
sam-lambda-1  |     "SAM_CLI_BETA_FEATURES",
sam-lambda-1  |     "SAM_CLI_BETA_BUILD_PERFORMANCE",
sam-lambda-1  |     "SAM_CLI_BETA_TERRAFORM_SUPPORT",
sam-lambda-1  |     "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
sam-lambda-1  |   ]
sam-lambda-1  | }

hawflau commented 2 months ago

Hey @git-noise thanks for raising the issue. After some digging, it seems the limitation is that, in the docker-in-docker scenario, the path (/tmp/tmpqn5dh6cm in your example) mounted to /var/task is relative to the host but not to the container. /tmp/tmpqn5dh6cm does not exist in the host but only in the container. That's why /var/task is empty. (reference: https://forums.docker.com/t/mounting-a-volume-not-working-with-running-docker-in-docker/25775/3)

SAM CLI does not support "Docker-in-Docker" right now. I'll mark this issue as a feature request for support for "Docker-in-Docker" and bring that up to the team.

git-noise commented 2 months ago

Hello, thanks a lot for looking into it @hawflau.

Based on your insight, I was able to confirm that it is indeed an issue relating to volume mounting. A quick test check that consists in mounting the full /tmp in the docker container, ensures that the temporary directory - here /tmp/tmpqn5dh6cm - is available within both the host and the container, and then /var/task properly populated.

Maybe an "easy" fix would be to adjust the function that creates these temporary directory, to take an optional base directory parameter? It would then allow for these temporary directory to be still created on the host, but in a specific directory that could be mounted on the sam-cli container?

Thanks for your help,

mattparrilla commented 1 month ago

@hawflau did this break recently? There are lots of posts and references for how to do docker-in-docker, including issues in this repo discussing how to get it working. I'm very surprised to see that it is not supported.

FWIW, I have the same exact issue (empty /var/task)

aws / aws-sam-cli