search

aws / aws-sam-cli

CLI tool to build, test, debug, and deploy Serverless applications using AWS SAM
https://aws.amazon.com/serverless/sam/
Apache License 2.0
6.51k stars 1.17k forks source link

Unable to run/debug functions locally if they depend on api gateway #2177

Closed deanthemachine2 closed 4 years ago

deanthemachine2 commented 4 years ago

Description

After upgrading to the new version of the SAM cli and the new version of the AWS toolkit for VS Code, we can no longer run and debug lambda functions locally if they depend on an API Gateway (also created and referenced in the same template.yaml file). This seems like a regression because this yaml file builds and deploys perfectly in our environments and until the latest version of SAM, also worked perfectly fine locally.

Steps to reproduce

Here's an example of what's in our yaml file:

  ApiGatewayBackend:
    Type: AWS::Serverless::Api
    Properties:
      StageName: !Ref environmentParam
      Domain:
        DomainName: !Ref domainNameParam
        CertificateArn: !Ref certificateARNParam
        EndpointConfiguration: EDGE
      MinimumCompressionSize: 0
      Auth:
        AddDefaultAuthorizerToCorsPreflight: false
        Authorizers:
          CognitoAuthorizer:
            UserPoolArn: !Ref cognitoUserPoolARNParam

  PostgresSelect:
    Type: AWS::Serverless::Function 
    DependsOn: 
    - ApiGatewayBackend
    Properties:
      FunctionName: !Sub "${environmentParam}-app-select"
      Description: stuff
      Runtime: python3.8
      CodeUri: src/functions/
      Handler: app_select.select_handler
      MemorySize: 896
      Policies: 
      - AmazonRDSDataFullAccess
      Environment:
        Variables:
          DATABASE_NAME: !Ref databaseNameParam
          CLUSTER_ARN: !Ref clusterARNParam
          SECRET_ARN: !Ref secretARNParam
      Events:
        ApiEvent:
          Type: Api
          Properties:
            RestApiId:
              Ref: ApiGatewayBackend
            Path: /select
            Method: get
            Auth:
              AddDefaultAuthorizerToCorsPreflight: false
              Authorizer: CognitoAuthorizer

Observed result

It starts running ("Preparing to debug 'app_selectvsctkdebug.lambda_handler' locally... Building SAM Application...") but then I get an error from VS Code that says this:

Error with child process: Traceback (most recent call last):
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\samlib\wrapper.py", line 76, in run_plugins
    parser.parse(template_copy, all_plugins)  # parse() will run all configured plugins
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\samlib\wrapper.py", line 131, in parse
    sam_plugins.act(LifeCycleEvents.before_transform_template, sam_template)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samtranslator\plugins\__init__.py", line 136, in act
    raise ex
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samtranslator\plugins\__init__.py", line 133, in act
    getattr(plugin, method_name)(*args, **kwargs)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samtranslator\plugins\api\implicit_api_plugin.py", line 92, in on_before_transform_template
    raise InvalidDocumentException(errors)
**samtranslator.model.exceptions.InvalidDocumentException: [InvalidResourceException('PostgresSelect', "Event with id [ApiEvent] is invalid. RestApiId must be a valid reference to an 'AWS::Serverless::Api' resource in same template.")]**

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\obj\windows-release\37amd64_Release\msi_python\zip_amd64\runpy.py", line 193, in _run_module_as_main
  File "D:\obj\windows-release\37amd64_Release\msi_python\zip_amd64\runpy.py", line 85, in _run_code
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\__main__.py", line 12, in <module>
    cli(prog_name="sam")
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\telemetry\metrics.py", line 96, in wrapped
    raise exception  # pylint: disable=raising-bad-type
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\telemetry\metrics.py", line 62, in wrapped
    return_value = func(*args, **kwargs)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\commands\build\command.py", line 129, in cli
    mode,
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\commands\build\command.py", line 179, in do_cli
    mode=mode,
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\commands\build\build_context.py", line 62, in __enter__
    self._function_provider = SamFunctionProvider(self._template_dict, self._parameter_overrides)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\providers\sam_function_provider.py", line 40, in __init__
    self.template_dict = SamFunctionProvider.get_template(template_dict, parameter_overrides)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\providers\sam_base_provider.py", line 126, in get_template
    template_dict = SamTranslatorWrapper(template_dict, parameter_values=parameters_values).run_plugins()
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\samlib\wrapper.py", line 79, in run_plugins
    functools.reduce(lambda message, error: message + " " + str(error), e.causes, str(e))
**samcli.commands.validate.lib.exceptions.InvalidSamDocumentException: [InvalidResourceException('PostgresSelect', "Event with id [ApiEvent] is invalid. RestApiId must be a valid reference to an 'AWS::Serverless::Api' resource in same template.")] ('PostgresSelect', "Event with id [ApiEvent] is invalid. RestApiId must be a valid reference to an 'AWS::Serverless::Api' resource in same template.")**

Expected result

I expected the function to run locally, like it did 1000x times before with this exact configuration. I added the 'depends on' part of the yaml to try to solve this but it does nothing. Also if I comment out these 2 lines:

"RestApiId: Ref: ApiGatewayBackend"

Then it works, however none of the gateway features are being tested locally, such as authorization. I should not we are not using a custom authorizer function we are just using cognito. This should be supported and it worked fine in the past.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

  1. OS: Windows 10 Pro
  2. sam --version: 1.1

Thank you for making such an awesome toolset - please let me know if I'm suddenly doing something wrong or if the docs changed, but I don't think I am.

wlsmark commented 4 years ago

I have the exact same issue!

deanthemachine2 commented 4 years ago

Any updates on this? This is preventing us from testing any of our functions locally.

qingchm commented 4 years ago

Any updates on this? This is preventing us from testing any of our functions locally.

@deanthemachine2 Hey so I wasn't able to reproduce issue, didn't have the same issue as both build and local invoke succeeded (on windows and macOS too). Also updated the indentation for your yaml code file part mentioned in this issue, can you take a look if you have the same indentation for your yaml file? If yes then would you mind sending me the entire template file so that I can learn more about why you would have this problem and try to reproduce and resolve your issue? Thanks!

deanthemachine2 commented 4 years ago

Hi @qingchm thanks for taking a look! Here's a full version with fake ARNs in our parameters. We really appreciate any help! 

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Globals:
  Function:
    Timeout: 60
  Api:
    Cors: 
      AllowMethods: "'GET,POST,PUT,PATCH,DELETE,OPTIONS'"
      AllowHeaders: "'Accept-Encoding,Content-Encoding,Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token,Client-ID'"
      AllowOrigin: "'*'"

Parameters:
  environmentParam:
    Type: String
    Default: "dev"
  databaseNameParam:
    Type: String
    Default: "tpadb"
  clusterARNParam:
    Type: String
    Default: "arn:aws:rds:us-east-1:xxxxxxxx:cluster:xxxxxx"
  secretARNParam:
    Type: String
    Default: "arn:aws:secretsmanager:us-east-1:xxxxxxxxxx:secret:rds-db-credentials/dev/xxxxxxxxxxxx"
  domainNameParam:
    Type: String
    Default: "d.newapi.xxxxxxxxx.io" 
  certificateARNParam:
    Type: String
    Default: "arn:aws:acm:us-east-1:xxxxxxxxx:certificate/xxxxxxx" 
  cognitoUserPoolARNParam:
    Type: String
    Default: "arn:aws:cognito-idp:us-east-1:xxxxxx:userpool/us-east-1_xxxxxxxx" 

Resources:
  ApiGatewayBackend:
    Type: AWS::Serverless::Api
    Properties:
      StageName: !Ref environmentParam
      Domain:
        DomainName: !Ref domainNameParam
        CertificateArn: !Ref certificateARNParam
        EndpointConfiguration: EDGE
      MinimumCompressionSize: 0
      Auth:
        AddDefaultAuthorizerToCorsPreflight: false
        Authorizers:
          CognitoAuthorizer:
            UserPoolArn: !Ref cognitoUserPoolARNParam
  PostgresSelect:
    Type: AWS::Serverless::Function 
    Properties:
      FunctionName: !Sub "${environmentParam}-app-select"
      Description: Select anything out of postgres by performing sql injection
      Runtime: python3.8
      CodeUri: src/functions/
      Handler: app_select.select_handler
      MemorySize: 896
      Policies: 
      - AmazonRDSDataFullAccess
      Environment:
        Variables:
          DATABASE_NAME: !Ref databaseNameParam
          CLUSTER_ARN: !Ref clusterARNParam
          SECRET_ARN: !Ref secretARNParam
      Events:
        ApiEvent:
          Type: Api
          Properties:
            RestApiId:
              Ref: ApiGatewayBackend
            Path: /select
            Method: get
            Auth:
              AddDefaultAuthorizerToCorsPreflight: false
              Authorizer: CognitoAuthorizer
qingchm commented 4 years ago

Hi @qingchm thanks for taking a look! Here's a full version with fake ARNs in our parameters. We really appreciate any help!

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Globals:
  Function:
    Timeout: 60
  Api:
    Cors: 
      AllowMethods: "'GET,POST,PUT,PATCH,DELETE,OPTIONS'"
      AllowHeaders: "'Accept-Encoding,Content-Encoding,Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token,Client-ID'"
      AllowOrigin: "'*'"

Parameters:
  environmentParam:
    Type: String
    Default: "dev"
  databaseNameParam:
    Type: String
    Default: "tpadb"
  clusterARNParam:
    Type: String
    Default: "arn:aws:rds:us-east-1:xxxxxxxx:cluster:xxxxxx"
  secretARNParam:
    Type: String
    Default: "arn:aws:secretsmanager:us-east-1:xxxxxxxxxx:secret:rds-db-credentials/dev/xxxxxxxxxxxx"
  domainNameParam:
    Type: String
    Default: "d.newapi.xxxxxxxxx.io" 
  certificateARNParam:
    Type: String
    Default: "arn:aws:acm:us-east-1:xxxxxxxxx:certificate/xxxxxxx" 
  cognitoUserPoolARNParam:
    Type: String
    Default: "arn:aws:cognito-idp:us-east-1:xxxxxx:userpool/us-east-1_xxxxxxxx" 

Resources:
  ApiGatewayBackend:
    Type: AWS::Serverless::Api
    Properties:
      StageName: !Ref environmentParam
      Domain:
        DomainName: !Ref domainNameParam
        CertificateArn: !Ref certificateARNParam
        EndpointConfiguration: EDGE
      MinimumCompressionSize: 0
      Auth:
        AddDefaultAuthorizerToCorsPreflight: false
        Authorizers:
          CognitoAuthorizer:
            UserPoolArn: !Ref cognitoUserPoolARNParam
  PostgresSelect:
    Type: AWS::Serverless::Function 
    Properties:
      FunctionName: !Sub "${environmentParam}-app-select"
      Description: Select anything out of postgres by performing sql injection
      Runtime: python3.8
      CodeUri: src/functions/
      Handler: app_select.select_handler
      MemorySize: 896
      Policies: 
      - AmazonRDSDataFullAccess
      Environment:
        Variables:
          DATABASE_NAME: !Ref databaseNameParam
          CLUSTER_ARN: !Ref clusterARNParam
          SECRET_ARN: !Ref secretARNParam
      Events:
        ApiEvent:
          Type: Api
          Properties:
            RestApiId:
              Ref: ApiGatewayBackend
            Path: /select
            Method: get
            Auth:
              AddDefaultAuthorizerToCorsPreflight: false
              Authorizer: CognitoAuthorizer

@deanthemachine2 Interestingly I couldn't reproduce the issue using your template. Did you try uninstalling the samcli completely and reinstall the newest version? A reasonable guess is that something was corrupted when you updated the samcli, that I would not suspect a newly installed samcli to have the same issue. But please let me know if this doesn't fix your issue, then we can discuss further on why this is happening! Thanks for your patience!

deanthemachine2 commented 4 years ago

@qingchm Unfortunately this didn't solve the issue. I just tried it, plus I've had this issue on 2 brand new installs on different machines. I am trying to think of all the other dependencies that could be at issue here - I've ruled out docker. Do you think it's potentially something wrong with the template we use to launch events with the vs code aws toolkit? I guess that would mean I am in the wrong repo but it makes me happy at least that you've been able to confirm that we are not doing something totally wrong with our template. Thank you for the sanity check - let me know if you have any additional ideas I'm open to trying anything.

qingchm commented 4 years ago

@qingchm Unfortunately this didn't solve the issue. I just tried it, plus I've had this issue on 2 brand new installs on different machines. I am trying to think of all the other dependencies that could be at issue here - I've ruled out docker. Do you think it's potentially something wrong with the template we use to launch events with the vs code aws toolkit? I guess that would mean I am in the wrong repo but it makes me happy at least that you've been able to confirm that we are not doing something totally wrong with our template. Thank you for the sanity check - let me know if you have any additional ideas I'm open to trying anything.

@deanthemachine2 Unfortunately still could not reproduce the issue that you had, have you tried to reproduce the issue without the use of vscode aws toolkit? If this is a problem only when you use vscode aws toolkit then it would be an issue for another repo I guess

deanthemachine2 commented 4 years ago

@qingchm thanks for all your help! I was finally able to identify it as an issue with the aws vs code toolkit (https://github.com/aws/aws-toolkit-vscode/issues/1352). Closing this here, leaving the link for anyone else that runs into this in the future.