Feature/issue 186 Implement API keys

nikki-t commented 3 weeks ago

Github Issue: #186

Description

Implement API keys to control usage.

Enable API keys for the Hydrocron API and create two keys, one for default users and another for more specialized or heavy use.
Create usage plans for each API key starting with expected access patterns and cost modelling for defaults.
Create a Lambda authorizer which will facilitate the use of API keys and usage plans to throttle requests and set quotas.

NOTE - We cannot implement the trusted key usage plan quite yet as we are waiting on platform to implement a solution that allows the passing of the trusted API key to our API endpoint.

Overview of work done

Created a Lambda authorizer function
Defined API Keys and Authorizer definition in OpenAPI specification
Defined API gateway API keys
Defined API gateway usage plans and associated with gateway stage
Defined SSM parameters to store API key values

Overview of verification done

Created three new unit tests

1) Test the connection to SSM client 2) Test default user API key IAM policy response in Lambda Authorizer 3) Test trusted user API key IAM policy response in Lambda Authorizer

New and existing unit tests pass.

Overview of integration done

Deployed to SIT environment, reviewed API Gateway architecture, and ran the following commands to test.

Reach CSV (application/json)

curl --location 'https://soto.podaac.sit.earthdatacloud.nasa.gov/hydrocron/v1/timeseries?feature=Reach&feature_id=63470800171&start_time=2024-02-01T00:00:00%2b00:00&end_time=2024-10-30T00:00:00%2b00:00&output=csv&fields=reach_id,time_str,wse'

{
    "status": "200 OK",
    "time": 706.182,
    "hits": 2,
    "results": {
        "csv": "reach_id,time_str,wse,wse_units\n63470800171,2024-02-01T02:26:50Z,3386.9332,m\n63470800171,2024-02-08T13:48:41Z,1453.4136,m\n",
        "geojson": {}
    }
}

Node GeoJSON (application/geo+json)

curl --header "Accept: application/geo+json" --location 'https://soto.podaac.sit.earthdatacloud.nasa.gov/hydrocron/v1/timeseries?feature=Node&feature_id=12228200110861&start_time=2023-11-29T00:00:00Z&end_time=2024-10-30T00:00:00Z&output=geojson&fields=reach_id,node_id,time_str,wse'

{
    "type": "FeatureCollection",
    "features": [
        {
            "id": "0",
            "type": "Feature",
            "properties": {
                "reach_id": [
                    "12228200111",
                    "12228200111"
                ],
                "node_id": [
                    "12228200110861",
                    "12228200110861"
                ],
                "time_str": [
                    "2024-01-30T21:19:19Z",
                    "2024-02-06T08:37:09Z"
                ],
                "wse": [
                    "3389.616",
                    "1346.93836"
                ],
                "wse_units": [
                    "m",
                    "m"
                ]
            },
            "geometry": {
                "type": "Point",
                "coordinates": [
                    35.149314,
                    -10.256285
                ]
            }
        }
    ]
}

CSV accept header (text/csv)

curl --header "Accept: text/csv" --location 'https://soto.podaac.sit.earthdatacloud.nasa.gov/hydrocron/v1/timeseries?feature=Reach&feature_id=63470800171&start_time=2024-02-01T00:00:00%2b00:00&end_time=2024-10-30T00:00:00%2b00:00&fields=reach_id,time_str,wse'

"reach_id,time_str,wse,wse_units\n63470800171,2024-02-01T02:26:50Z,3386.9332,m\n63470800171,2024-02-08T13:48:41Z,1453.4136,m\n"

Sample CloudWatch log for authorizer (execution logs)

2024-06-04T20:37:15.241Z (xxx) Incoming identity: {method.request.header.User-Agent=*****8.6.0}
2024-06-04T20:37:15.241Z (xxx) Endpoint request URI: https://lambda.xxx.amazonaws.com/2015-03-31/functions/arn:aws:lambda:xxx:xxx:function:svc-hydrocron-sit-authorizer-lambda/invocations
2024-06-04T20:37:15.241Z (xxx) Endpoint request headers: [TRUNCATED]
2024-06-04T20:37:15.241Z (xxx) Endpoint request body after transformations: [TRUNCATED]
2024-06-04T20:37:15.241Z (xxx) Sending request to https://lambda.xxx.amazonaws.com/2015-03-31/functions/arn:aws:lambda:xxx:xxx:function:svc-hydrocron-sit-authorizer-lambda/invocations
2024-06-04T20:37:15.363Z (xxx) Authorizer result body before parsing: {"principalId": "default_user", "policyDocument": {"Version": "2012-10-17", "Statement": [{"Action": "execute-api:Invoke", "Effect": "Allow", "Resource": "arn:aws:execute-api:xxx:xxx:xxx/v1/GET/timeseries"}]}, "usageIdentifierKey": "xxx"}
2024-06-04T20:37:15.363Z (xxx) Using valid authorizer policy for principal: ******abc
2024-06-04T20:37:15.363Z (xxx) Successfully completed authorizer execution
2024-06-04T20:37:15.363Z (xxx) Verifying Usage Plan for request: xxx. API Key: **********************************123 API Stage: xxx/v1
2024-06-04T20:37:15.365Z (xxx) Usage Plan check succeeded for API Key **********************************123 and API Stage xxx/v1

Response when usage plan quota is hit:

{"message":"Limit Exceeded"}

PR checklist:

[X] Linted
[X] Updated unit tests
[X] Updated changelog
[X] Integration testing

See Pull Request Review Checklist for pointers on reviewing this pull request

nikki-t commented 3 weeks ago

This is a draft PR so that we can figure out starting defaults for throttling and the monthly requests quota.

In the meantime, I wanted to check on some decisions I made:

The location of the Lambda authorizer is in hydrocron.api.controllers.authorizer. Does this seem okay with the current organization of code?
Per this documentation,"A REQUEST authorizer receives the caller's identity in a combination of headers, query string parameters, stageVariables, and $context variables."
1. The identity source that I selected was the User-Agent header since that gets passed by most (all?) requests and end users won't have to modify their current requests. Are there any thoughts or concerns around using this header? Should we pick a different one?

nikki-t commented 2 weeks ago

I updated the Lambda authorizer to use a x-hydrocron-key header to differentiate between a default user and trusted partner.

I was able to confirm that the trusted partner key counted against the request quota defined in the trusted partner usage plan. I could not confirm request throttling but would like to deploy this to UAT and test using the benchmark tests. We can manually edit the throttling parameters to confirm throttling and the error response that is returned.

I also confirmed that the user does not need to pass in an API key which will count against the default usage plan.

I think we are ready to deploy this to UAT after we finalize the quota and throttle parameters for the different usage plans.

nikki-t commented 2 weeks ago

I updated the "timeseries" documentation to include a section on API keys. We need to decide:

1) What we consider heavy usage to be and when a user should request an API key. It is currently defined as: "Heavy usage can be defined as continued used with over x requests per day or continue use which require many requests per second or concurrent requests." What should be consider x to be?

2) How do users obtain API keys? The documentation currently says: "To request an API key or to discuss your use case, please contact us at x." Should we facilitate the request through an e-mail? If so what address do we use?

@frankinspace @torimcd @cassienickles - Interested in your thoughts!

nikki-t commented 2 weeks ago

I updated the quota and throttle settings per #186 discussion. I set the "trusted_partner" usage plan to have a quota of 5 requests per month and set the rate per second to 1 and the concurrent requests to 1. I don't think this should matter since we aren't rolling the trusted partner API key quite yet but wanted to keep the numbers low. I think this PR is ready for review!

nikki-t commented 1 week ago

@frankinspace - I think the updated changes are ready to be reviewed and then possibly deployed to UAT?

podaac / hydrocron