Allow APIGatewayResolver to handle Custom Domain with Mapping and still work

walmsles commented 3 years ago

Is your feature request related to a problem? Please describe. Have opened as a feature request as I believe this is not exactly a bug, perhaps an oversight in the ApiGatewayResolver design (happy to discuss more).
A couple of months ago I developed an API for a project whose API gateway was associated with a custom domain. This week I built a new API extension (new gateway) using AWS Lambda Powertools for Python and have applied several routes into the one lambda using the API Gateway Resolver with the intention of adding to the same custom domain since want the ApiGateway to be hosted on the one common DNS domain. When associating a second gateway to a custom domain you must associate a mapping for the additional gateways so the API paths do not collide and everything works.

Within my lambda resolver setup for this second gateway if I have a resolver route of "/status" setup this works fine if it is mounted as-is on the root of the domain. If I add to a custom domain as a second gateway I need to add in a "mapping", for example, sake let's say I choose "unique". The AWS API Gateway event "path" for the API when I call it as "https://mycustomdomain.com/unique/status" is set to "/unique/status" which means the power tools Resolver will respond with a "404, NOT FOUND" since that path is not setup within the Resolver routes.

I have noticed that the "resource" path in the event correctly holds the route as "/status" but the path holds the route as "/unique/status"

This is a complicated one - the ApiGatewayResolver does not allow me to mount the gateways developed with this component on any custom domain with a mapping and have the lambda API actually work - I actually kind of think this is a bug but not raising as such since the implementation seems perfectly reasonable.

Describe the solution you'd like What I would ideally like is the freedom to be able to mount my Python API to a custom domain using any mapping I choose and still have the API resolver find my routes within the lambda code correctly.

The current implementation uses the "path" of the ApiGateway Lambda event which houses the complete API path including the mapping which breaks the resolver.

Describe alternatives you've considered As a work around I can simply change my routes to include the proposed mapping but then this stops me from being able to use Api Gateway configuration to remap an API in the future and actually have it work without changing my code which is not ideal given this is a feature of using Services like the AWS Api gateway.

Additional context This kind of also brings into question how the Event content is generated and passed to lambda by the ApiGateway since one could argue it makes no sense that the "path" also includes the logical "mapping" from the API gateway Custom Domain configuration (not an argument I want to start but a consideration given the logical config nature of this scenario).

I have taken a look at the event structure from this configuration and notice the "resource" contains a correct path that matches the route I have in my Lambda Resolver routes in python code but is possibly not ideal.

michaelbrewer commented 3 years ago

True, i never really thought about this use case, i have come across something similar for my old java lambdas. So might be useful is a way to either set a prefix to allow for this custom mappings.

Would you be able to post some test cases / failing examples. So it is easier to validate we have a good fix for this.

walmsles commented 3 years ago

Hi @michaelbrewer here is an event which is from a custom domain mapping from my actual use case (modified to remove project specific data and sensitive data).

The existing implementation uses the path to match to routes which will always include the Custom Domain Mapping value. If instead the resource is used it will always represent what I would naturally put into my route definition using the ApiGatewayResolver (kind of). This seems to represent the resource path for the actual APIGW itself.

Not sure if the the event pathParameters** array is useful here - ApiGateway actually lists out the path Parameters for Api calls which pulls out all the parameter values in the path for me to process.

Actual Example event for an API route of "/status/"

{
    "level": "INFO",
    "location": "decorate:345",
    "message":
    {
        "resource": "/status/{id}",
        "path": "/unique/status/xxyyzz",
        "httpMethod": "GET",
        "headers":
        {
            "Accept": "*/*",
            "Accept-Encoding": "gzip, deflate, br",
            "Host": "mydomain.com",
            "Postman-Token": "42ffaf84-16c0-405f-a696-fea861f0fa01",
            "User-Agent": "PostmanRuntime/7.28.1",
            "X-Amzn-Trace-Id": "Root=1-6103d40e-378940ba4213ff4453d029ab",
            "x-api-key": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
            "X-Forwarded-For": "124.170.115.3",
            "X-Forwarded-Port": "443",
            "X-Forwarded-Proto": "https"
        },
        "multiValueHeaders":
        {
            "Accept":
            [
                "*/*"
            ],
            "Accept-Encoding":
            [
                "gzip, deflate, br"
            ],
            "Host":
            [
                "mydomain.com"
            ],
            "Postman-Token":
            [
                "42ffaf84-16c0-405f-a696-fea861f0fa01"
            ],
            "User-Agent":
            [
                "PostmanRuntime/7.28.1"
            ],
            "X-Amzn-Trace-Id":
            [
                "Root=1-6103d40e-378940ba4213ff4453d029ab"
            ],
            "x-api-key":
            [
                "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
            ],
            "X-Forwarded-For":
            [
                "124.170.115.3"
            ],
            "X-Forwarded-Port":
            [
                "443"
            ],
            "X-Forwarded-Proto":
            [
                "https"
            ]
        },
        "queryStringParameters": null,
        "multiValueQueryStringParameters": null,
        "pathParameters":
        {
            "id": "xxyyzz"
        },
        "stageVariables": null,
        "requestContext":
        {
            "resourceId": "32pmz6",
            "resourcePath": "/status/{id}",
            "httpMethod": "GET",
            "extendedRequestId": "DR4SOES-SwMFu7w=",
            "requestTime": "30/Jul/2021:10:27:26 +0000",
            "path": "/unique/statusi/xxyyzz",
            "accountId": "161945688208",
            "protocol": "HTTP/1.1",
            "stage": "prod",
            "domainPrefix": "myapi",
            "requestTimeEpoch": 1627640846021,
            "requestId": "154d31f3-6620-445e-b606-b67e9c084bd9",
            "identity":
            {
                "cognitoIdentityPoolId": null,
                "cognitoIdentityId": null,
                "apiKey": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
                "principalOrgId": null,
                "cognitoAuthenticationType": null,
                "userArn": null,
                "apiKeyId": "xxxxxxxxxx",
                "userAgent": "PostmanRuntime/7.28.1",
                "accountId": null,
                "caller": null,
                "sourceIp": "124.170.115.3",
                "accessKey": null,
                "cognitoAuthenticationProvider": null,
                "user": null
            },
            "domainName": "mydomain.com",
            "apiId": "xxxxxxpxx"
        },
        "body": null,
        "isBase64Encoded": false
    },
    "timestamp": "2021-07-30 10:27:27,323+0000",
    "service": "service_undefined",
    "cold_start": true,
    "function_name": "my-prod-api",
    "function_memory_size": "1024",
    "function_arn": "arn:aws:lambda:ap-southeast-2:xxxxxxxxxxx:function:my-prod-api",
    "function_request_id": "452e2852-e010-4e9c-b9ed-74d591c1884c",
    "correlation_id": "154d31f3-6620-445e-b606-b67e9c084bd9",
    "xray_trace_id": "1-6103d40e-378940ba4213ff4453d029ab"
}

walmsles commented 3 years ago

Hi @michaelbrewer - have looked at the other Event Type in test folder to see formats. My statements are very particular to APi GW so thanks for highlighting ALL the use cases.

I feel the Path Mapping idea you have prototyped is useful - is there a way to use the resource if it exists to work out the mapping based on string difference? This would allow full use of the API GW custom domain mapping,

Also maybe this is something you don't want or need to support - it is probably a very unlikely use-case so was just raising as something I felt would be useful to more completely support APIGW events.

michaelbrewer commented 3 years ago

If you switch to http api gateways v2 the path is same regardless of how you remap it. So this might be a better option when you can do this.

michaelbrewer commented 3 years ago

@walmsles @heitorlessa So for api gateway there are 3 variations (and ALB is similar to the Http API V1), so it is a little complicated but possible.

Summary (TLDR)

It might be possible for {proxy+} mappings for Rest api and Http api v1 to use the resource part of the event to help autodetect the custom mappings, and then have a flag to auto strip it? But there are exceptions.

Otherwise an alternative way of doing the routing using the proxied path for the routing?

For Http api v2, nothing needs to be done.

Background

Rest api (the old rest api gateway)
Http api v1 (similar event as Rest api)
Http api v2 (a more concise version)

For the APIGatewayResolver we use path for Rest api and Http api v1 and rawPath for Http api v2 to determine how the routing works.

Examples

So here are some examples:

foo.foo.com/custom/foo/*- mapped via proxy+ to a lambda via a custom domain
foo.foo.com/foo/* - mapped via proxy+ to a lambda directly using route 53

Here is what that events look like stripping out parts:

Rest api

In the below examples path is different, but we could use the resource to determine the starting point and automatically strip off the /custom part?

GET https://foo.foo.com/custom/foo/example1

{
    "resource": "/foo/{proxy+}",
    "path": "/custom/foo/example1",
    "httpMethod": "GET",
    "headers": {},
    "multiValueHeaders": {},
    "queryStringParameters": {},
    "multiValueQueryStringParameters": {},
    "requestContext": {
        "resourcePath": "/foo/{proxy+}",
        "httpMethod": "GET",
        "path": "/custom/foo/example1",
        "stage": "v1",
        "domainPrefix": "foo",
        "identity": {},
        "domainName": "foo.gift-dev.solutions"
    },
    "pathParameters": {
        "proxy": "example1"
    }
}

GET https://foo.foo.com/foo/example1

{
    "resource": "/foo/{proxy+}",
    "path": "/foo/example1",
    "httpMethod": "GET",
    "headers": {},
    "multiValueHeaders": {},
    "queryStringParameters": {},
    "multiValueQueryStringParameters": {},
    "requestContext": {
        "resourcePath": "/foo/{proxy+}",
        "httpMethod": "GET",
        "path": "/foo/example1",
        "stage": "v1",
        "domainPrefix": "foo",
        "identity": {},
        "domainName": "foo.gift-dev.solutions"
    },
    "pathParameters": {
        "proxy": "example1"
    }
}

However for /{proxy+} again it differs, so only pathParameters.proxy can be used

GET https://foo.foo.com/custom/status

{
    "resource": "/{proxy+}",
    "path": "/custom/status",
    "httpMethod": "GET",
    "headers": {},
    "multiValueHeaders": {},
    "queryStringParameters": {},
    "multiValueQueryStringParameters": {},
    "requestContext": {
        "resourcePath": "/{proxy+}",
        "httpMethod": "GET",
        "path": "/custom/status",
        "stage": "v1",
        "domainPrefix": "foo",
        "identity": {},
        "domainName": "foo.gift-dev.solutions"
    },
    "pathParameters": {
        "proxy": "status"
    }
}

Http API V1:

Could use the same logic as Rest api and use resource for striping. Or use requestContext.path for mapping instead?

GET https://foo.foo.com/custom/foo/example1

{
    "version": "1.0",
    "resource": "/foo/{proxy+}",
    "path": "/custom/foo/example1",
    "httpMethod": "GET",
    "headers": {},
    "multiValueHeaders": {},
    "queryStringParameters": {},
    "multiValueQueryStringParameters": {},
    "requestContext": {
        "domainName": "foo.foo.com",
        "domainPrefix": "foo",
        "httpMethod": "GET",
        "identity": {},
        "path": "/foo/example1",
        "resourceId": "ANY /foo/{proxy+}",
        "resourcePath": "/foo/{proxy+}"
    },
    "pathParameters": {
        "proxy": "example1"
    }
}

GET https://foo.foo.com/foo/example1

{
    "version": "1.0",
    "resource": "/foo/{proxy+}",
    "path": "/foo/example1",
    "httpMethod": "GET",
    "headers": {},
    "multiValueHeaders": {},
    "queryStringParameters": {},
    "multiValueQueryStringParameters": {},
    "requestContext": {
        "domainName": "aaaa.execute-api.us-east-1.amazonaws.com",
        "domainPrefix": "aaaa",
        "httpMethod": "GET",
        "identity": {},
        "path": "/foo/example1",
        "resourceId": "ANY /foo/{proxy+}",
        "resourcePath": "/foo/{proxy+}"
    },
    "pathParameters": {
        "proxy": "example1"
    }
}

However for /{proxy+} to can't use /foo/ as the starting point

GET https://foo.foo.com/custom/status

{
    "version": "1.0",
    "resource": "/{proxy+}",
    "path": "/custom/status",
    "httpMethod": "GET",
    "headers": {},
    "multiValueHeaders": {},
    "queryStringParameters": {},
    "multiValueQueryStringParameters": {},
    "requestContext": {
        "domainName": "foo.foo.com",
        "domainPrefix": "foo",
        "httpMethod": "GET",
        "identity": {},
        "path": "/status",
        "resourceId": "ANY /{proxy+}",
        "resourcePath": "/{proxy+}"
    },
    "pathParameters": {
        "proxy": "status"

Http V2

Here rawPath is the same for both

GET "https://foo.foo.com/custom/foo/example1"

{
    "version": "2.0",
    "routeKey": "ANY /foo/{proxy+}",
    "rawPath": "/foo/example1",
    "headers": {},
    "queryStringParameters": {},
    "requestContext": {
        "domainName": "foo.foo.com",
        "domainPrefix": "foo",
        "http": {
            "method": "GET",
            "path": "/foo/example1"
        },
        "routeKey": "ANY /foo/{proxy+}"
    },
    "pathParameters": {
        "proxy": "example1"
    },
    "isBase64Encoded": false
}

GET https://aaaa.execute-api.us-east-1.amazonaws.com/foo/example1

{
    "version": "2.0",
    "routeKey": "ANY /foo/{proxy+}",
    "rawPath": "/foo/example1",
    "headers": {},
    "queryStringParameters": {},
    "requestContext": {
        "domainName": "aaaa.execute-api.us-east-1.amazonaws.com",
        "domainPrefix": "aaaa",
        "http": {
            "method": "GET",
            "path": "/foo/example1"
        },
        "routeKey": "ANY /foo/{proxy+}"
    },
    "pathParameters": {
        "proxy": "example1"
    },
    "isBase64Encoded": false
}

michaelbrewer commented 3 years ago

@heitorlessa @walmsles - what do you think is a good solution for this? (and i guess there can be more than one solutions):

Docs - Recommending the API Gateway Http API V2 integration, which does not run into this kind of issues?
Code - Add an option parameter to to strip by a prefix, but does not support more than one mapping at a time unless the lambda used environment variables. (#579)
Code - Add an jmespath option which allows you to select pathParameters.proxy (Rest API fix), or requestContext. path (Http API V1 fix)

I think maybe 1 & 3 might work well enough for the most cases, as 2 only solves for a single mapping conbination.

heitorlessa commented 3 years ago

Thanks a lot for raising this @walmsles -- this is a known problem in API Gateway that only got fixed in HTTP API as @michaelbrewer pointed out.

As far as I remember this also impacted validation in other frameworks.

I'll think this through with @michaelbrewer this week

Thanks again

michaelbrewer commented 3 years ago

@heitorlessa - based on our last discussion. One solution was to rebuild against the “resource” path, vs just using the “path” (which will include any custom mappings).

To be able to rebuild the resource path we need to replace the “resource” with “pathParameters”. As this might be a breaking change, we can add this as a flag which is turned of by default (like use_resource_path)

NOTE: And this would only apply to the Rest API events, as this does not apple to HTTP API V2.

heitorlessa commented 3 years ago

Transferring this to Roadmap to improve visibility on what's being worked on.

I'm pondering on whether we should be treating this as a bug for REST API v1 (sub-optimal event), since that will only break unit tests for folks not the actual experience - please correct me if I'm wrong.

heitorlessa commented 3 years ago

This got even more complex as API GW support an arbitrary level of mapping paths and customers could have multiple of these. Added details in the PR: https://github.com/awslabs/aws-lambda-powertools-python/pull/579#issuecomment-900307854

Single custom domain mapping path - v1

{
    "path": "/v1/payment",
    "resource": "/payment",
    "requestContext": {
        "resource": "/payment",
        "path": "/v1/payment",
        "httpMethod": "GET",
        "requestContext": {
            "resourceId": "j9knhf",
            "resourcePath": "/payment",
            "httpMethod": "GET",
            "path": "/v1/payment",
            "stage": "default",
            "domainPrefix": "api",
            "domainName": "api.serverlessa.dev",
        },
    }
}

Nested custom domain mapping path - v1/nested

Custom domain mapping - proxy resource

{
    "path": "/v1/nested/payment/123456789-afekl-13456/",
    "resource": "/payment/{invoice+}",
    "requestContext": {
        "resource": "/payment/{invoice+}",
        "path": "/v1/nested/payment/123456789-afekl-13456/",
        "httpMethod": "GET",
        "requestContext": {
            "resourceId": "8em8dt",
            "resourcePath": "/payment/{invoice+}",
            "httpMethod": "GET",
            "path": "/v1/nested/payment/123456789-afekl-13456/",
            "stage": "default",
            "domainPrefix": "api",
            "domainName": "api.serverlessa.dev",
        }
    }
}

heitorlessa commented 3 years ago

Now available in 1.20.0 and we support all discrepancies found in REST API, HTTP API v1 and v2 payloads

https://awslabs.github.io/aws-lambda-powertools-python/latest/core/event_handler/api_gateway/#custom-domain-api-mappings

aws-powertools / powertools-lambda