search

aws / aws-sam-cli

CLI tool to build, test, debug, and deploy Serverless applications using AWS SAM
https://aws.amazon.com/serverless/sam/
Apache License 2.0
6.5k stars 1.17k forks source link

AWS SAM running local won't resolve FindInMap Intrinsic #2694

Open ricardoinacio opened 3 years ago

ricardoinacio commented 3 years ago

Description:

I can't make a template.yaml with Mappings work when testing locally, although they will deploy and work as expected.

Steps to reproduce:

  1. sam init --runtime python3.8
    1. Insert Mappings such as the following in template.yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  sam-app

  Sample SAM Template for sam-app

Parameters:
  Environment:
    Description: 'Required. Environment'
    Type: String
    AllowedValues:
      - test
      - prod
    Default: test

# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 3
    MemorySize: !FindInMap [Environments, !Ref Environment, MemorySize]

Mappings:
  Environments:
    test:
      MemorySize: 512
    prod:
      MemorySize: 2048

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: hello_world/
      Handler: app.lambda_handler
      Runtime: python3.8
      Events:
        HelloWorld:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /hello
            Method: get

Outputs:
  # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
  # Find out more about other implicit resources you can reference within SAM
  # https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
  HelloWorldApi:
    Description: "API Gateway endpoint URL for Prod stage for Hello World function"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"
  HelloWorldFunction:
    Description: "Hello World Lambda Function ARN"
    Value: !GetAtt HelloWorldFunction.Arn
  HelloWorldFunctionIamRole:
    Description: "Implicit IAM Role created for Hello World function"
    Value: !GetAtt HelloWorldFunctionRole.Arn
  1. sam build --parameter-overrides ParamaterKey=Environment,ParameterValue=test --use-container
  2. sam local start-api --debug, and it will show unresolved references.
2021-03-06 00:30:42,394 | local start-api command is called
2021-03-06 00:30:42,399 | Collected default values for parameters: {'Environment': 'test'}
2021-03-06 00:30:42,417 | Unable to resolve property MemorySize: OrderedDict([('Fn::FindInMap', ['Environments', OrderedDict([('Ref', 'Environment')]), 'MemorySize'])]). Leaving as is.
2021-03-06 00:30:42,417 | 2 resources found in the stack 
2021-03-06 00:30:42,417 | Collected default values for parameters: {'Environment': 'test'}
2021-03-06 00:30:42,433 | Unable to resolve property MemorySize: OrderedDict([('Fn::FindInMap', ['Environments', OrderedDict([('Ref', 'Environment')]), 'MemorySize'])]). Leaving as is.
2021-03-06 00:30:42,434 | Collected default values for parameters: {'Environment': 'test'}
2021-03-06 00:30:42,449 | Unable to resolve property MemorySize: OrderedDict([('Fn::FindInMap', ['Environments', OrderedDict([('Ref', 'Environment')]), 'MemorySize'])]). Leaving as is.
2021-03-06 00:30:42,449 | Found Serverless function with name='HelloWorldFunction' and CodeUri='HelloWorldFunction'
2021-03-06 00:30:42,452 | Collected default values for parameters: {'Environment': 'test'}
2021-03-06 00:30:42,467 | Unable to resolve property MemorySize: OrderedDict([('Fn::FindInMap', ['Environments', OrderedDict([('Ref', 'Environment')]), 'MemorySize'])]). Leaving as is.
2021-03-06 00:30:42,467 | 2 resources found in the template
2021-03-06 00:30:42,468 | Found '1' API Events in Serverless function with name 'HelloWorldFunction'
2021-03-06 00:30:42,468 | Detected Inline Swagger definition
2021-03-06 00:30:42,468 | Lambda function integration not found in Swagger document at path='/hello' method='get'
2021-03-06 00:30:42,468 | Found '0' APIs in resource 'ServerlessRestApi'
2021-03-06 00:30:42,468 | Removed duplicates from '0' Explicit APIs and '1' Implicit APIs to produce '1' APIs
2021-03-06 00:30:42,468 | 1 APIs found in the template
2021-03-06 00:30:42,471 | Mounting HelloWorldFunction at http://127.0.0.1:3000/hello [GET]
2021-03-06 00:30:42,471 | You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
2021-03-06 00:30:42,471 | Localhost server is starting up. Multi-threading = True
2021-03-06 00:30:42  * Running on http://127.0.0.1:3000/ (Press CTRL+C to quit)

Observed result:

  1. Insist with curl http://127.0.0.1/hello and sam will show an error:
2021-03-06 00:33:25,099 | Found one Lambda function with name 'HelloWorldFunction'
2021-03-06 00:33:25,099 | Invoking app.lambda_handler (python3.8)
2021-03-06 00:33:25,099 | No environment variables found for function 'HelloWorldFunction'
2021-03-06 00:33:25,099 | Environment variables overrides data is standard format
2021-03-06 00:33:25,099 | Loading AWS credentials from session with profile 'None'
2021-03-06 00:33:25,113 | Resolving code path. Cwd=/home/ricardo/Sandbox/learning/aws/sam-app/.aws-sam/build, CodeUri=/home/ricardo/Sandbox/learning/aws/sam-app/.aws-sam/build/HelloWorldFunction
2021-03-06 00:33:25,113 | Resolved absolute path to code is /home/ricardo/Sandbox/learning/aws/sam-app/.aws-sam/build/HelloWorldFunction
2021-03-06 00:33:25,113 | Code /home/ricardo/Sandbox/learning/aws/sam-app/.aws-sam/build/HelloWorldFunction is not a zip/jar file
2021-03-06 00:33:25,127 | Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-python3.8:rapid-1.19.1.

2021-03-06 00:33:25,127 | Mounting /home/ricardo/Sandbox/learning/aws/sam-app/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated inside runtime container
2021-03-06 00:33:25,127 | Cleaning all decompressed code dirs
2021-03-06 00:33:25,128 | Exception on /hello [GET]
Traceback (most recent call last):
  File "/home/ricardo/.local/lib/python3.8/site-packages/docker/utils/utils.py", line 415, in parse_bytes
    digits = int(digits_part)
ValueError: invalid literal for int() with base 10: "OrderedDict([('Fn::FindInMap', ['Environments', OrderedDict([('Ref', 'Environment')]), 'MemorySize'])])"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ricardo/.local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/ricardo/.local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/ricardo/.local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/ricardo/.local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/ricardo/.local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/ricardo/.local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/ricardo/.local/lib/python3.8/site-packages/samcli/local/apigw/local_apigw_service.py", line 297, in _request_handler
    self.lambda_runner.invoke(route.function_name, event, stdout=stdout_stream_writer, stderr=self.stderr)
  File "/home/ricardo/.local/lib/python3.8/site-packages/samcli/commands/local/lib/local_lambda.py", line 111, in invoke
    self.local_runtime.invoke(config, event, debug_context=self.debug_context, stdout=stdout, stderr=stderr)
  File "/home/ricardo/.local/lib/python3.8/site-packages/samcli/lib/telemetry/metric.py", line 216, in wrapped_func
    return_value = func(*args, **kwargs)
  File "/home/ricardo/.local/lib/python3.8/site-packages/samcli/local/lambdafn/runtime.py", line 148, in invoke
    container = self.create(function_config, debug_context)
  File "/home/ricardo/.local/lib/python3.8/site-packages/samcli/local/lambdafn/runtime.py", line 82, in create
    self._container_manager.create(container)
  File "/home/ricardo/.local/lib/python3.8/site-packages/samcli/local/docker/manager.py", line 95, in create
    container.create()
  File "/home/ricardo/.local/lib/python3.8/site-packages/samcli/local/docker/container.py", line 167, in create
    real_container = self.docker_client.containers.create(self._image, **kwargs)
  File "/home/ricardo/.local/lib/python3.8/site-packages/docker/models/containers.py", line 860, in create
    create_kwargs = _create_container_args(kwargs)
  File "/home/ricardo/.local/lib/python3.8/site-packages/docker/models/containers.py", line 1077, in _create_container_args
    create_kwargs['host_config'] = HostConfig(**host_config_kwargs)
  File "/home/ricardo/.local/lib/python3.8/site-packages/docker/types/containers.py", line 182, in __init__
    self['Memory'] = parse_bytes(mem_limit)
  File "/home/ricardo/.local/lib/python3.8/site-packages/docker/utils/utils.py", line 417, in parse_bytes
    raise errors.DockerException(
docker.errors.DockerException: Failed converting the string value for memory (OrderedDict([('Fn::FindInMap', ['Environments', OrderedDict([('Ref', 'Environment')]), 'MemorySize'])])) to an integer.

Expected result:

There shouldn't be unresolved FindInMap causing exceptions, just as it doesn't happen with CloudFormation itself.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

  1. OS: Pop_OS! version 20.10, based on Ubuntu.
  2. sam --version: 1.19.1 and 1.20.0
  3. AWS region: not sure if relevant, but I deploy to us-west-2.
CoshUS commented 2 years ago

Thanks for the feedback. We will be adding this to our intake for prioritization.

In the meantime, we highly recommend you to use SAM Accelerate to test this in your development environment in the cloud. Here is a helpful video to learn more about it, and here is the documentation for the beta feature. Please let us know if this helps with your testing needs.

Rabadash8820 commented 2 years ago

I am also experiencing this issue. It's not too bad for environment variables, as I can just overwrite them in the file passed to --env-vars, but this isn't possible if I use Mappings in an actual function property like Timeout. SAM Accelerate doesn't help either when I need this template to run entirely locally (for a demo where I won't have internet).

wouterlemcke commented 2 years ago

Any update on this?

dgard1981 commented 2 years ago

I came across an issue (reported here) where layers were not being included, and it was because I was using a mapping to provide the layer version. A SAM template is supposed to be an extension of a CloudFormation template, so it somewhat beggars believe that not all of the intrinsic functions that are available in a CF template can be used.

Will this issue be fixed? If so, hopefully it will be soon.

dgard1981 commented 2 years ago

I've been stung by this issue again today, although with the !Sub intrinsic. I can find previous issues that have been closed by AWS (including #528 and #1038), but these issues have been closed without resolution.

For some it is not appropriate sam sync, so that cannot be the solution. The sam local commands have been provided and should be expected to work.

Please provide an update on the overarching issue of intrinsic functions not working with sam local.

cscetbon commented 1 year ago

I've just noticed something weird. When I run sam local invoke with an environment that has an integer value in the map, the corresponding environment variable is empty, but not if it's a string

Mappings:
  Environments:
    local:
      collectionPort: 9200
...
Globals:
  Function:
    Environment:
      Variables:
        COLLECTION_PORT: !FindInMap [ Environments, !Ref StageName, collectionPort ]
...
$ sam local invoke ...
...
[INFO]  2023-08-03T17:34:55.361Z    ace04caa-7c77-4244-b7ad-db85e23dee42    
environ=environ({'AWS_LAMBDA_FUNCTION_MEMORY_SIZE': '1024', '_HANDLER': 
'handler.handler', .... 'COLLECTION_PORT': '', ...
Mappings:
  Environments:
    local:
      collectionHost: aos
      collectionPort: "9200"
...
Globals:
  Function:
    Environment:
      Variables:
        COLLECTION_PORT: !FindInMap [ Environments, !Ref StageName, collectionPort ]
...
$ sam local invoke ...
...
[INFO]  2023-08-03T17:34:55.361Z    ace04caa-7c77-4244-b7ad-db85e23dee42    
environ=environ({'AWS_LAMBDA_FUNCTION_MEMORY_SIZE': '1024', 
'_HANDLER': 'handler.handler', .... 'COLLECTION_PORT': '9200', ...