Allow lambda security group access to cache security group in CDK stack

[leothomas]

Closes #108

What I did:

Granted the lambda's security group access to the cache's security group through CDK.

How I did it:

I created a custom security group in the lambda function's VPC, and directly attached this security group to the (using the security_groups:[] property of the aws_lambda.Function class - this skips the automatic creation a default security group in the VPC.

I was then able to grant the lambda 's custom security group access to the cache security group (using the ec2.SecurityGroup.add_ingress_rule() property) so that the lambda function has access to the cache.

How you can test it:

Running this code snippet for the first time:

for e in ["", "global", "du", "be", "tk","la", "sf", "ny", "gl"]:
    start = time.time()
    r = re.get(f"https://08dvkxus0a.execute-api.us-east-1.amazonaws.com/v1/datasets/{e}") # staging url 
    end = time.time()
    print(f"Requested: /{e}, response time: {str(end-start)}, Response headers: {r.headers}")

produces the following output:

Requested: / , response time: 4.346861839294434, Response headers: {'Date': 'Mon, 25 Jan 2021 15:43:59 GMT', 'Content-Type': 'application/json', 'Content-Length': '4450', 'Connection': 'keep-alive', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkR5igsIAMEVFQ='}

Requested: /global, response time: 0.15545296669006348, Response headers: {'Date': 'Mon, 25 Jan 2021 15:43:59 GMT', 'Content-Type': 'application/json', 'Content-Length': '1758', 'Connection': 'keep-alive', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkR8jvLoAMEWDw='}

Requested: /du, response time: 0.7655160427093506, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:00 GMT', 'Content-Type': 'application/json', 'Content-Length': '2264', 'Connection': 'keep-alive', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkR9j87IAMEVGQ='}

Requested: /be, response time: 0.7159359455108643, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:01 GMT', 'Content-Type': 'application/json', 'Content-Length': '2501', 'Connection': 'keep-alive', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkSFjwEIAMEWDw='}

Requested: /tk, response time: 0.5140988826751709, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:01 GMT', 'Content-Type': 'application/json', 'Content-Length': '3190', 'Connection': 'keep-alive', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkSMjyqIAMEVcA='}

Requested: /la, response time: 0.6045031547546387, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:02 GMT', 'Content-Type': 'application/json', 'Content-Length': '2728', 'Connection': 'keep-alive', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkSRgmfIAMEVQg='}

Requested: /sf, response time: 0.6608572006225586, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:02 GMT', 'Content-Type': 'application/json', 'Content-Length': '3148', 'Connection': 'keep-alive', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkSYifeIAMEVaQ='}

Requested: /ny, response time: 0.6407091617584229, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:03 GMT', 'Content-Type': 'application/json', 'Content-Length': '3047', 'Connection': 'keep-alive', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkSejJQIAMEVTQ='}

Requested: /gl, response time: 0.39092206954956055, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:03 GMT', 'Content-Type': 'application/json', 'Content-Length': '1758', 'Connection': 'keep-alive', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkSkgQroAMEVew='}

Notice that:

• Each spotlight endpoint responds in ~0.5-0.6 seconds
• The full /datasets request takes ~4 seconds

Running the code snippet a second time produces:

Requested: / , response time: 0.19382977485656738, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:09 GMT', 'Content-Type': 'application/json', 'Content-Length': '4450', 'Connection': 'keep-alive', 'X-Cache': 'HIT', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkTajsqIAMEVEg='}

Requested: /global, response time: 0.11775016784667969, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:09 GMT', 'Content-Type': 'application/json', 'Content-Length': '1758', 'Connection': 'keep-alive', 'X-Cache': 'HIT', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkTchDioAMEVXw='}

Requested: /du, response time: 0.1087641716003418, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:09 GMT', 'Content-Type': 'application/json', 'Content-Length': '2264', 'Connection': 'keep-alive', 'X-Cache': 'HIT', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkTdgszoAMEVQg='}

Requested: /be, response time: 0.11829781532287598, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:09 GMT', 'Content-Type': 'application/json', 'Content-Length': '2501', 'Connection': 'keep-alive', 'vary': 'Accept-Encoding', 'X-Cache': 'HIT', 'content-encoding': 'gzip', 'Apigw-Requestid': 'ZtkTej5gIAMEVcA='}

Requested: /tk, response time: 0.12758183479309082, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:09 GMT', 'Content-Type': 'application/json', 'Content-Length': '3190', 'Connection': 'keep-alive', 'X-Cache': 'HIT', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkTfgSvoAMEVLw='}

Requested: /la, response time: 0.11828804016113281, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:09 GMT', 'Content-Type': 'application/json', 'Content-Length': '2728', 'Connection': 'keep-alive', 'X-Cache': 'HIT', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkTgixRIAMEVSQ='}

Requested: /sf, response time: 0.1327040195465088, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:09 GMT', 'Content-Type': 'application/json', 'Content-Length': '3148', 'Connection': 'keep-alive', 'X-Cache': 'HIT', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkThicDoAMEVMQ='}

Requested: /ny, response time: 0.10931611061096191, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:09 GMT', 'Content-Type': 'application/json', 'Content-Length': '3047', 'Connection': 'keep-alive', 'X-Cache': 'HIT', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkTjj8LIAMEV2A='}

Requested: /gl, response time: 0.12949299812316895, Response headers: {'Date': 'Mon, 25 Jan 2021 15:44:10 GMT', 'Content-Type': 'application/json', 'Content-Length': '1758', 'Connection': 'keep-alive', 'X-Cache': 'HIT', 'content-encoding': 'gzip', 'vary': 'Accept-Encoding', 'Apigw-Requestid': 'ZtkTkhMYIAMEVXg='}

Notice that:

• Full /datasets request takes ~0.2 seconds
• Each spotlight takes ~0.1 seconds
• Each response headers object includes 'X-Cache': 'HIT'

Note:

• I separated the vpc_access_policy_statement statement into vpc_log_access and vpc_ec2_access for clarity
• I renamed iam_policy_statement to s3_full_access_to_data_bucket for clarity