Custom pipeline resolver breaks mocking

[ben-elsen]

Describe the bug I have added/created a custom resolver pipeline with 2 functions. I have added the config in the CustomResources.json file and placed the vtl templates in the resolvers folder.

When I try amplify push, everything works and when I test the custom pipeline (mutation) in the console, everything works.

When I try amplify mock, I get the following error: Failed to start API Mock endpoint Error: Invalid config for PIPELINE_RESOLVER {"typeName":"Mutation","functions":["SaveKnowledgeNicheFunction","SendKnowledgeNicheSignedUpEventFunction"],"fieldName":"signUpKnowledgeNiche","requestMappingTemplateLocation":"resolvers/Mutation.signUpKnowledgeNiche.Pipeline.req.vtl","responseMappingTemplateLocation":"resolvers/Mutation.signUpKnowledgeNiche.Pipeline.res.vtl","kind":"PIPELINE"}

Amplify CLI Version 4.2.0

To Reproduce Add the following to the CustomResources.json file of your api:

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "An auto-generated nested stack.",
    "Metadata": {},
    "Parameters": {
        "AppSyncApiId": {
            "Type": "String",
            "Description": "The id of the AppSync API associated with this project."
        },
        "AppSyncApiName": {
            "Type": "String",
            "Description": "The name of the AppSync API",
            "Default": "AppSyncSimpleTransform"
        },
        "env": {
            "Type": "String",
            "Description": "The environment name. e.g. Dev, Test, or Production",
            "Default": "NONE"
        },
        "S3DeploymentBucket": {
            "Type": "String",
            "Description": "The S3 bucket containing all deployment assets for the project."
        },
        "S3DeploymentRootKey": {
            "Type": "String",
            "Description": "An S3 key relative to the S3DeploymentBucket that points to the root\nof the deployment directory."
        }
    },
    "Resources": {
        "AppSyncEventBridgeRole": {
            "Type": "AWS::IAM::Role",
            "Properties": {
                "AssumeRolePolicyDocument": {
                    "Statement": [
                        {
                            "Action": "sts:AssumeRole",
                            "Effect": "Allow",
                            "Principal": {
                                "Service": "appsync.amazonaws.com"
                            }
                        }
                    ],
                    "Version": "2012-10-17"
                }
            }
        },
        "AppSyncEventBridgeRoleDefaultPolicy": {
            "Type": "AWS::IAM::Policy",
            "Properties": {
                "PolicyDocument": {
                    "Statement": [
                        {
                            "Action": "events:Put*",
                            "Effect": "Allow",
                            "Resource": "*"
                        }
                    ],
                    "Version": "2012-10-17"
                },
                "PolicyName": "AppSyncEventBridgeRoleDefaultPolicy",
                "Roles": [
                    {
                        "Ref": "AppSyncEventBridgeRole"
                    }
                ]
            }
        },
        "EventBridgeDataSource": {
            "Type": "AWS::AppSync::DataSource",
            "DependsOn": [
                "AppSyncEventBridgeRole"
            ],
            "Properties": {
                "ApiId": {
                    "Ref": "AppSyncApiId"
                },
                "Name": "EventBridgeDataSource",
                "Type": "HTTP",
                "HttpConfig": {
                    "AuthorizationConfig": {
                        "AuthorizationType": "AWS_IAM",
                        "AwsIamConfig": {
                            "SigningRegion": {
                                "Ref": "AWS::Region"
                            },
                            "SigningServiceName": "events"
                        }
                    },
                    "Endpoint": {
                        "Fn::Join": [
                            "",[
                                "https://events.",
                                {
                                    "Ref": "AWS::Region"
                                },
                                ".amazonaws.com/"
                            ]
                        ]
                    }
                },
                "ServiceRoleArn": {
                    "Fn::GetAtt": [
                        "AppSyncEventBridgeRole",
                        "Arn"
                    ]
                }
            }
        },
        "SignUpKnowledgeNicheResolver": {
            "Type": "AWS::AppSync::Resolver",
            "DependsOn": [
                "SaveKnowledgeNicheFunction",
                "SendKnowledgeNicheSignedUpEventFunction"
            ],
            "Properties": {
                "ApiId": {
                    "Ref": "AppSyncApiId"
                },
                "FieldName": "signUpKnowledgeNiche",
                "TypeName": "Mutation",
                "Kind": "PIPELINE",
                "PipelineConfig": {
                    "Functions": [
                        {
                            "Fn::GetAtt": [
                                "SaveKnowledgeNicheFunction",
                                "FunctionId"
                            ]
                        },
                        {
                            "Fn::GetAtt": [
                                "SendKnowledgeNicheSignedUpEventFunction",
                                "FunctionId"
                            ]
                        }
                    ]
                },
                "RequestMappingTemplateS3Location": {
                    "Fn::Sub": [
                        "s3://${S3DeploymentBucket}/${S3DeploymentRootKey}/resolvers/${ResolverFileName}",
                        {
                            "S3DeploymentBucket": {
                                "Ref": "S3DeploymentBucket"
                            },
                            "S3DeploymentRootKey": {
                                "Ref": "S3DeploymentRootKey"
                            },
                            "ResolverFileName": {
                                "Fn::Join": [
                                    ".",
                                    [
                                        "Mutation",
                                        "signUpKnowledgeNiche",
                                        "Pipeline",
                                        "req",
                                        "vtl"
                                    ]
                                ]
                            }
                        }
                    ]
                },
                "ResponseMappingTemplateS3Location": {
                    "Fn::Sub": [
                        "s3://${S3DeploymentBucket}/${S3DeploymentRootKey}/resolvers/${ResolverFileName}",
                        {
                            "S3DeploymentBucket": {
                                "Ref": "S3DeploymentBucket"
                            },
                            "S3DeploymentRootKey": {
                                "Ref": "S3DeploymentRootKey"
                            },
                            "ResolverFileName": {
                                "Fn::Join": [
                                    ".",
                                    [
                                        "Mutation",
                                        "signUpKnowledgeNiche",
                                        "Pipeline",
                                        "res",
                                        "vtl"
                                    ]
                                ]
                            }
                        }
                    ]
                }
            }
        },
        "SaveKnowledgeNicheFunction": {
            "Type": "AWS::AppSync::FunctionConfiguration",
            "Properties": {
                "ApiId": {
                    "Ref": "AppSyncApiId"
                },
                "Name": "SaveKnowledgeNicheFunction",
                "DataSourceName": "KnowledgeNicheTable",
                "FunctionVersion": "2018-05-29",
                "RequestMappingTemplateS3Location": {
                    "Fn::Sub": [
                        "s3://${S3DeploymentBucket}/${S3DeploymentRootKey}/resolvers/${ResolverFileName}",
                        {
                            "S3DeploymentBucket": {
                                "Ref": "S3DeploymentBucket"
                            },
                            "S3DeploymentRootKey": {
                                "Ref": "S3DeploymentRootKey"
                            },
                            "ResolverFileName": {
                                "Fn::Join": [
                                    ".",
                                    [
                                        "Mutation",
                                        "signUpKnowledgeNiche",
                                        "Function",
                                        "createEntity",
                                        "req",
                                        "vtl"
                                    ]
                                ]
                            }
                        }
                    ]
                },
                "ResponseMappingTemplateS3Location": {
                    "Fn::Sub": [
                        "s3://${S3DeploymentBucket}/${S3DeploymentRootKey}/resolvers/${ResolverFileName}",
                        {
                            "S3DeploymentBucket": {
                                "Ref": "S3DeploymentBucket"
                            },
                            "S3DeploymentRootKey": {
                                "Ref": "S3DeploymentRootKey"
                            },
                            "ResolverFileName": {
                                "Fn::Join": [
                                    ".",
                                    [
                                        "Mutation",
                                        "signUpKnowledgeNiche",
                                        "Function",
                                        "createEntity",
                                        "res",
                                        "vtl"
                                    ]
                                ]
                            }
                        }
                    ]
                }
            }
        },
        "SendKnowledgeNicheSignedUpEventFunction": {
            "Type": "AWS::AppSync::FunctionConfiguration",
            "DependsOn": [
                "EventBridgeDataSource"
            ],
            "Properties": {
                "ApiId": {
                    "Ref": "AppSyncApiId"
                },
                "Name": "SendKnowledgeNicheSignedUpFunction",
                "DataSourceName": "EventBridgeDataSource",
                "FunctionVersion": "2018-05-29",
                "RequestMappingTemplateS3Location": {
                    "Fn::Sub": [
                        "s3://${S3DeploymentBucket}/${S3DeploymentRootKey}/resolvers/${ResolverFileName}",
                        {
                            "S3DeploymentBucket": {
                                "Ref": "S3DeploymentBucket"
                            },
                            "S3DeploymentRootKey": {
                                "Ref": "S3DeploymentRootKey"
                            },
                            "ResolverFileName": {
                                "Fn::Join": [
                                    ".",
                                    [
                                        "Mutation",
                                        "signUpKnowledgeNiche",
                                        "Function",
                                        "sendEvent",
                                        "req",
                                        "vtl"
                                    ]
                                ]
                            }
                        }
                    ]
                },
                "ResponseMappingTemplateS3Location": {
                    "Fn::Sub": [
                        "s3://${S3DeploymentBucket}/${S3DeploymentRootKey}/resolvers/${ResolverFileName}",
                        {
                            "S3DeploymentBucket": {
                                "Ref": "S3DeploymentBucket"
                            },
                            "S3DeploymentRootKey": {
                                "Ref": "S3DeploymentRootKey"
                            },
                            "ResolverFileName": {
                                "Fn::Join": [
                                    ".",
                                    [
                                        "Mutation",
                                        "signUpKnowledgeNiche",
                                        "Function",
                                        "sendEvent",
                                        "res",
                                        "vtl"
                                    ]
                                ]
                            }
                        }
                    ]
                }
            }
        }
 },
    "Outputs": {
        "AppSyncApiId": {
            "Value": {
                "Ref": "AppSyncApiId"
            },
            "Export": {
                "Name": "AppSyncApiId"
            }
        }
    }
}

Expected behavior The mocking endpoint and interface working.

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: Mac
• Node Version. v9.7.1

Additional context Add any other context about the problem here.

[ben-elsen]

I found the reason why it is broken, I have a function in my pipeline that has a HTTP datasource. When I remove that one, it works.

I found this by diving in the code. Line 52 in the amplify-appsync-simulator/lib/pipeline-resolver.js gives an error: config.functions.map(function (fn) { return simulatorContext.getFunction(fn); });

If I isolate this line in a new try/catch block, I get:

Error: Missing function SendKnowledgeNicheSignedUpEventFunction at AmplifyAppSyncSimulator.getFunction (/usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-appsync-simulator/src/index.ts:144:13) at /usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-appsync-simulator/src/resolvers/pipeline-resolver.ts:11:6 at Array.map (<anonymous>) at new AppSyncPipelineResolver (/usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-appsync-simulator/src/resolvers/pipeline-resolver.ts:11:6) at /usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-appsync-simulator/src/index.ts:101:15 at Array.reduce (<anonymous>) at AmplifyAppSyncSimulator.init (/usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-appsync-simulator/src/index.ts:94:41) at APITest.<anonymous> (/usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-util-mock/lib/api/src/api/api.ts:48:29) at Generator.next (<anonymous>) at fulfilled (/usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-util-mock/lib/api/api.js:5:58)

[gviger]

@yuth I assume you guys are aware of this but just to point out that while @ben-elsen fixed his problem the bigger problem still exists. Even with a very simple structure resources:

• Datasource
• Resolver
• Pipeline Resolver I get the the Pipeline_Resolver error identified. For testing purposes the Resolver req and res are just {} and empty respectively. The resolver datasource references the datasource defined of course.

I'm not sure if I'm doing something wrong or if it is another bug, but under Amplify Push if a Datasource Resource is defined (as required by Mock) and that Datasource already exists in AppSync say for an existing DynamoDB table, then the Amplify Push fails. Remove the Datasource and Amplify Push works as expected. However without the Datasource, Mock no longer works.

It would be great if Amplify Push could accept an existing datasource or if Mock could work without a datasource defined. Here is my basic CustomResources.json to test.

 "Resources": {
        "Accounts": {
            "Type": "AWS::AppSync::DataSource",
            "Properties": {
                "Type": "AMAZON_DYNAMODB",
              "Name": "Accounts",
              "ApiId": {"Ref": "AppSyncApiId"},
              "ServiceRoleArn": "arn:aws:iam::407886809666:role/service-role/appsync-ds-ddb-xurmft-Accounts-devb",
              "DynamoDBConfig": {
                "TableName": "Accounts-devb",
                "AwsRegion": "us-west-2"
                }
            }
        },
        "GetAccountRecordsFunction": {
          "Type": "AWS::AppSync::FunctionConfiguration",
          "Properties": {"ApiId": {"Ref": "AppSyncApiId"},
            "Name": "getAccountRecords",
            "DataSourceName": "Accounts","FunctionVersion": "2018-05-29",
            "RequestMappingTemplateS3Location": {"Fn::Sub": ["s3://${S3DeploymentBucket}/${S3DeploymentRootKey}/resolvers/Query.getAccountRecords.req.vtl",{"S3DeploymentBucket": {"Ref": "S3DeploymentBucket"},"S3DeploymentRootKey": {"Ref": "S3DeploymentRootKey"}}]},
            "ResponseMappingTemplateS3Location": {"Fn::Sub": ["s3://${S3DeploymentBucket}/${S3DeploymentRootKey}/resolvers/Query.getAccountRecords.res.vtl",{"S3DeploymentBucket": {"Ref": "S3DeploymentBucket"},"S3DeploymentRootKey": {"Ref": "S3DeploymentRootKey"}}]}
            }
        },      
        "GetFunctionPipelineResolver": {
          "Type": "AWS::AppSync::Resolver",
          "Properties": {"ApiId": {"Ref": "AppSyncApiId"},"TypeName": "Query","Kind": "PIPELINE",
            "FieldName": "getFunction",
            "PipelineConfig": {
              "Functions": [
                {"Fn::GetAtt": ["GetAccountRecordsFunction", "FunctionId"]}
              ]
            },
            "RequestMappingTemplateS3Location": {"Fn::Sub": ["s3://${S3DeploymentBucket}/${S3DeploymentRootKey}/resolvers/Query.getFunction.req.vtl",{"S3DeploymentBucket": {"Ref": "S3DeploymentBucket"},"S3DeploymentRootKey": {"Ref": "S3DeploymentRootKey"}}]},
            "ResponseMappingTemplateS3Location": {"Fn::Sub": ["s3://${S3DeploymentBucket}/${S3DeploymentRootKey}/resolvers/Query.getFunction.res.vtl",{"S3DeploymentBucket": {"Ref": "S3DeploymentBucket"},"S3DeploymentRootKey": {"Ref": "S3DeploymentRootKey"}}]}
          },
          "DependsOn": ["GetAccountRecordsFunction"]
        },
           "EmptyResource": {
            "Type": "Custom::EmptyResource",
            "Condition": "AlwaysFalse"
        }
    },

[yuth]

@gviger Is the data source you are using in the pipeline a already existing table? Mock operates on tables locally and never connects to cloud. Could you share a sample schema where I can reproduce this.

Also we recently updated how Cloudformation files are parsed by mock and it might error out earlier if the table is referenced in cloudformation. Could you give it a try and see if this is still an issue.

[gviger]

@yuth I can't even run mock now. Getting the 'Cannont read property 'line' of undefined' error referenced in issue 2620. At first I assumed it was because I added my datasources into my stack/CustomResources.json. But even when I remove them to a stage that is working with push I the error remains.

To answer your questions:

1. Having the datasources in CustomResolver when using Mock worked as expected... they were more or less ignored because I was off line (opposite of your point 1)

2. It was during an amplify push that an error came up when the datasources existed in AppSync/GraphQL but were not created/defined during cloudformation (ie pre-existing tables and datasources)... at least I'm assuming it was because they were not created through cloudformation. In general it tried to create the datasource and errored/failed/rolled back if it already existed. Really if a datasource type exists during cloudformation it should ignore it (and comment) but otherwise continue.

3. Interestingly I just had to rebuild my amplify structure yesterday. I tried using a CustomResources with all my desired datasources and resolvers (off all 3 types normal, pipeline and pipeline functions). This failed as even though the datasources definitions were first in the resources. It gave error that they did not exist when it got to building the resolvers. I however now realized that I did not have a dependency in my resolver for the the datasource resource. My solution however was to just run it twice. First a CustomResolver with just the datasources. Pushed, then ran again with just the Resolvers which runs each time I push now. This worked. Again have not tried mock and it has its own issues now.

CLI: 4.13.4, Windows 10 WSL Ubuntu 18.04 LTS

[yuth]

@gviger Would you mind sharing a schema where can reproduce this

[kaustavghosh06]

@ben-elsen @gviger It would be great if you could hsare your schema along with any custom stacks/resolvers so that we can reproduce this and provide a fix. You can email it to us at amplify-cli@amazon.com

[ben-elsen]

Hi @kaustavghosh06 ,

I don't know if I can help in this. Our api has evolved and is now managed as a seperate microservice with its own (SAM) cloudformation template (stack) and many pipeline resolvers that trigger events to Eventbridge..

Best, Ben

[gviger]

@kaustavghosh06 I will get this to you. Mock still breaks for me with my full CustomResources.json so trying to reproduce the problem with a smaller number of entries. Haven't had time to focus on that part in the past bit but will in the next couple of weeks. Please leave this open for the moment and I'll update with more findings and send as requested. Thx.

[jake-barry]

I was having a similar problem. The following fixed mine, in my custom resources stack:

1.) My "AWS::AppSync::Resolver" pipeline resource had an inline mapping template (such as: "RequestMappingTemplate": "{}",). However, it seems like the CFNParser looks for the s3 location (@aws-amplify/cli/node_modules/amplify-util-mock/lib/CFNParser/appsync-resource-processor.js graphqlFunctionHandler)

2.) My "AWS::AppSync::FunctionConfiguration" resources had different names for the CFN resources than in the "name" property. Looks like @aws-amplify/cli/node_modules/amplify-util-mock/lib/CFNParser/appsync-resource-processor.js getAppSyncFunctionName is looking at the stack resource name while @aws-amplify/cli/node_modules/amplify-appsync-simulator/lib/index.js AmplifyAppSyncSimulator.prototype.init is building a map with keys that should be the same as this but instead they are from the name property of the resource. Then, @aws-amplify/cli/node_modules/amplify-appsync-simulator/lib/resolvers/pipeline-resolver.js AppSyncPipelineResolver tries to find the values of simulatorContext using config.functions and the names don't match.

Annoyingly, amplify push worked just fine as, it seems, actual Cloudformation is fine with both of these...

[gviger]

Thanks for sharing @jake-barry. That is good to know for future.

For anyone else landing here, I moved to using serverless with apollo-server-lambda for query and mutations fronted by ALB in production (lower cost and better handling of Cognito than API Gateway). The complexity of our app and resolvers was just too challenging and limited under Amplify/AppSync. The ability to use Node JS with merge-graphql-schemas package in our Lambdas is working well for us in general. Initially we will still use AppSync for Subscriptions because we use a specific subscription pipeline from DynamoDB triggers we can attach to. Hopefully that will also move to serverless but apollo-lambda's ability to handle them is a bit trickier and expensive with the suggested API Gateway websockets approach (as is Appsync Subscriptions at scale). Likely we will land on our own apollo EC2 instance behind the ALB for subscriptions for cost benefit and minimal server clusters to manage.

If you have other ideas or suggestions please share. Still trying to find the perfect scalable serverless solution to GraphQL backend.

[yuth]

1.) My "AWS::AppSync::Resolver" pipeline resource had an inline mapping template (such as: "RequestMappingTemplate": "{}",). However, it seems like the CFNParser looks for the s3 location (@aws-amplify/cli/node_modules/amplify-util-mock/lib/CFNParser/appsync-resource-processor.js graphqlFunctionHandler)

@jake-barry PR https://github.com/aws-amplify/amplify-cli/pull/4225 should fix the above issue

[SwaySway]

Closing this issue as this has been addressed in PR #4225.

[github-actions[bot]]

This issue has been automatically locked since there hasn't been any recent activity after it was closed. Please open a new issue for related bugs.

Looking for a help forum? We recommend joining the Amplify Community Discord server *-help channels for those types of questions.