Bug: local commands crash when trying to symlink files from a non-existent host directory

[ffMathy]

Description:

Before v104, I was able to use the SAM CLI normally. Now I can't.

Steps to reproduce:

DOCKER_HOST_MACHINE_IP_ADDRESS="172.17.0.1"

sam local start-api \
    --port ${PORT} \
    --debug \
    --container-env-vars ./.aws-sam/scripts/container-environment.start.json \
    --template ./.aws-sam/build/template.yaml \
    --docker-volume-basedir ${LOCAL_WORKSPACE_FOLDER}/src/apps/redacted/src/.aws-sam/build \
    --container-host "${DOCKER_HOST_MACHINE_IP_ADDRESS}" \
    --host "0.0.0.0" \
    --container-host-interface "0.0.0.0" \
    --env-vars $ENV_VARS_FILE \
    --profile $AWS_PROFILE \
    --region eu-west-1

Observed result:

2024-01-04 13:20:49,271 | Constructed Event Version 2.0 to invoke Lambda. Event: {'version': '2.0', 'routeKey': 'POST /events', 'rawPath': '/events', 'rawQueryString': '', 'cookies': [], 'headers': {'Accept': 'application/json, text/plain, */*', 'Content-Type': 'application/json', 'User-Agent': 'axios/1.6.4', 'Content-Length': '68', 'Accept-Encoding': 'gzip, compress, deflate, br', 'Host': 'localhost:46109', 'Connection': 'keep-alive', 'X-Forwarded-Proto': 'http', 'X-Forwarded-Port': '46109'}, 'requestContext': {'accountId': '123456789012', 'apiId': '1234567890', 'http': {'method': 'POST', 'path': '/events', 'protocol': 'HTTP/1.1', 'sourceIp': '127.0.0.1', 'userAgent': 'Custom User Agent String'}, 'requestId': '1281b6bd-fd66-441f-aea5-8afb8898a582', 'routeKey': 'POST /events', 'stage': '$default', 'time': '04/Jan/2024:13:19:48 +0000', 'timeEpoch': 1704374388, 'domainName': 'localhost', 'domainPrefix': 'localhost'}, 'body': '{"topic":"redacted"}', 'pathParameters...
  2024-01-04 13:20:49,271 | Found one Lambda function with name 'Lambda'
  2024-01-04 13:20:49,271 | Invoking app.handler (nodejs20.x)
  2024-01-04 13:20:49,272 | Environment variables data found for specific function in standard format
  2024-01-04 13:20:49,272 | Resolving code path. Cwd=/home/runner/work/redacted/redacted/src/apps/redacted/src/.aws-sam/build, CodeUri=Lambda
  2024-01-04 13:20:49,272 | Resolved absolute path to code is /home/runner/work/redacted/redacted/src/apps/redacted/src/.aws-sam/build/Lambda
  2024-01-04 13:20:49,272 | Code /home/runner/work/redacted/redacted/src/apps/redacted/src/.aws-sam/build/Lambda is not a zip/jar file
  2024-01-04 13:20:49,280 | Checking free port on 0.0.0.0:6283
  2024-01-04 13:20:49,283 | Using local image: public.ecr.aws/lambda/nodejs:20-rapid-x86_64.
  2024-01-04 13:20:49,283 | Mounting /home/runner/work/redacted/redacted/src/apps/redacted/src/.aws-sam/build/Lambda as /var/task:ro,delegated, inside runtime container
  2024-01-04 13:20:49,283 | Cleaning all decompressed code dirs
  2024-01-04 13:20:49,283 | Exception on /events [POST]
  Traceback (most recent call last):
    File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1455, in wsgi_app
      response = self.full_dispatch_request()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 869, in full_dispatch_request
      rv = self.handle_user_exception(e)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 867, in full_dispatch_request
      rv = self.dispatch_request()
           ^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 852, in dispatch_request
      return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/dist-packages/samcli/local/apigw/local_apigw_service.py", line 723, in _request_handler
      lambda_response = self._invoke_lambda_function(route.function_name, route_lambda_event)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/dist-packages/samcli/local/apigw/local_apigw_service.py", line 616, in _invoke_lambda_function
      self.lambda_runner.invoke(lambda_function_name, event_str, stdout=stdout_writer, stderr=self.stderr)
    File "/usr/local/lib/python3.11/dist-packages/samcli/commands/local/lib/local_lambda.py", line 144, in invoke
      self.local_runtime.invoke(
    File "/usr/local/lib/python3.11/dist-packages/samcli/lib/telemetry/metric.py", line 324, in wrapped_func
      return_value = func(*args, **kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/dist-packages/samcli/local/lambdafn/runtime.py", line 189, in invoke
      container = self.create(function_config, debug_context, container_host, container_host_interface)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/dist-packages/samcli/local/lambdafn/runtime.py", line 104, in create
      self._container_manager.create(container)
    File "/usr/local/lib/python3.11/dist-packages/samcli/local/docker/manager.py", line 95, in create
      container.create()
    File "/usr/local/lib/python3.11/dist-packages/samcli/local/docker/container.py", line 167, in create
      **self._create_mapped_symlink_files(),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/dist-packages/samcli/local/docker/container.py", line 250, in _create_mapped_symlink_files
      with os.scandir(self._host_dir) as directory_iterator:
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  FileNotFoundError: [Errno 2] No such file or directory: '/home/runner/work/redacted/redacted/src/apps/redacted/src/.aws-sam/build/Lambda'
  2024-01-04 13:20:49,284 | Lambda execution failed (<InternalServerError '500: Internal Server Error'>,)

Expected result:

I expect it to work like it did in 403.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: DevContainer (so Linux)
2. sam --version: 104, 105 and 106 are affected
3. AWS region: eu-west-1

# Paste the output of `sam --info` here
{
  "version": "1.104.0",
  "system": {
    "python": "3.11.2",
    "os": "Linux-6.4.16-linuxkit-aarch64-with-glibc2.36"
  },
  "additional_dependencies": {
    "docker_engine": "24.0.6",
    "aws_cdk": "Not available",
    "terraform": "Not available"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}

Add --debug flag to command you are running

[mildaniel]

Thanks @ffMathy for reporting. I'm looking into this now but having some trouble reproducing the issue. Does this only occur when using dev container?

[ffMathy]

Not sure. It's probably hard to make a repro. We're just going to stay on v103. Just wanted to let you know of an issue being introduced in 104. Sorry I can't be of much more help.

[mildaniel]

I'm sorry for the trouble you've had using SAM CLI. For this issue, it seems like that the path detected by SAM CLI might be on the host instead of inside the container. It depends on the container configuration and the which socket file the container is pointing to.

In any case, SAM CLI shouldn't fail here and we will prioritize a working on a fix.

[ffMathy]

Alright. Is there an ETA on a solution?

[github-actions[bot]]

Patch is released in v1.108.0. Closing