Open DingoEatingFuzz opened 6 years ago
Seems a lot of lamda functions and aws step funtions are fronted by aws api gateway but would need to look into how that plays with the civicpdx.org domain and sub domains.
This is good that there is a lamda example in hacko now. This can be used in the future to look at a more cost effective implamentation than containers/ecs (at least for smaller less compute/memory items that can be split off into simpler/descrete functions. (Future cost/performace project after this seasons projects)
Notes from discussion with Ian:
There may be a "direct-to-custom-path" approach that doesn't use API Gateway, but we're not sure that's viable here.
API Gateway is already set up and pointing to the lambda function, so it sounds like the bulk of the work remaining is getting a tls cert.
There are two reasonable paths forward:
Option 1: Use AWS Certificate Manager
Option 2: Use Let's Encrypt
If I read the pricing for cert manager correctly, it will cost us $0.75 per month per cert. Which is to say it will cost us $0.75 per month.
Given that low cost, I'm not opposed to it, but I do think Let's Encrypt shares values with Hack Oregon.
I'll leave the call up to the devops team, but I would like to at least explore what it will take to also use tls for our frontends and our other APIs. There is no private data being transferred, but https is the new norm.
Thank you for the specifics Michael, that’ll make this a lot less of a jungle hunt.
I’m gonna vote in favour of “working first, good second”:
If no one else is itching to take this on I’ll grab this on the weekend.
Checklist of possible things to do - updated to the new domain name:
civicSandboxPackages
Lambda functionsandbox.civicpdx.org
service.civicpdx.org
service.civicpdx.org
zone with the following attributes:
It took 5-10 minutes to validate after that record was created, but finally the ACM status for that cert flipped to Issued
My read of the AWS Lambda docs (e.g. Set up Custom Domain for an API in API Gateway) so far indicates that when setting up a custom domain name, the entire domain name gets directed to the Lambda side of the house (API Gateway I believe).
Looking in our current Route 53 config for service.civicpdx.org
it looks like that domain is already being redirected to our ECS cluster - this is the A record value:
dualstack.hacko-integration-658279555.us-west-2.elb.amazonaws.com.
I believe that we'll have to setup a separate custom domain (e.g. lambda.civicpdx.org
or functions.civicpdx.org
) to be able to call the Sandbox' Lambda function with a URL better than the usual nightmare-long value issued by AWS.
Michael has recommended sandbox.civicpdx.org
We'll need:
And there's instructions for setting this up in the Custom Domain Name instructions:
sandbox.civicpdx.org
namespace without having an A record for that namespace first. Turns out Route 53 doesn't do any more validation than necessary - it doesn't care if you create a CNAME for a namespace that doesn't have an A record. (DNS is not for the uninitiated 'round these parts.) And it further turns out that ACM really only needs that CNAME record to validate domain ownership - even though there's no associated A record.d-vih7gfr49i.execute-api.us-west-2.amazonaws.com
aws route53 change-resource-record-sets --hosted-zone-id Z1PP2R4G54X1EA --change-batch file://path/to/your/setup-dns-record.json
setup-dns-record.json
batch file:
{
"Changes": [
{
"Action": "CREATE",
"ResourceRecordSet": {
"Name": "sandbox.civicpdx.org",
"Type": "A",
"AliasTarget": {
"DNSName": "d-vih7gfr49i.execute-api.us-west-2.amazonaws.com",
"HostedZoneId": "Z2OJLYMUO9EFXC",
"EvaluateTargetHealth": false
}
}
}
]
}
{
"ChangeInfo": {
"Status": "PENDING",
"SubmittedAt": "2018-06-10T19:34:51.914Z",
"Id": "/change/CR2C5UA9N7KQN"
}
}
--hosted-zone-id
is the value for the HackO account (look at the Route 53 console > Hosted Zones > find the value under the Hosted Zone ID for the civicpdx.org
zone), and the HostedZoneId
in the JSON is the value from the Amazon API GatewayThe following URI: https://sandbox.civicpdx.org/civic-sandbox
successfully responds with good JSON:
{
statusCode: 200,
body: {
packages: {
transportation: {
description: "Transportation: This is a fake description. This is a fake description. This is a fake description. This is a fake description.",
foundations: [
"103",
"105",
"104"
],
default_foundation: "103",
slides: [
"102",
"106",
"001",
"002"
],
default_slide: "106"
},
affordable housing: {
description: "Transportation: This is a fake description. This is a fake description. This is a fake description. This is a fake description.",
foundations: [
"105",
"104",
"006"
],
default_foundation: "105",
slides: [
"101",
"102",
"003",
"004"
],
default_slide: "101"
},
neighborhoods: {
description: "Transportation: This is a fake description. This is a fake description. This is a fake description. This is a fake description.",
foundations: [
"104",
"006"
],
default_foundation: "006",
slides: [
"101",
"102",
"106",
"005"
],
default_slide: "102"
}
},
slides: {
101: {
name: "sweeps",
endpoint: "https://0uv7y2d29i.execute-api.us-east-2.amazonaws.com/mockslide101/",
description: "This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description.",
visualization: "ScreenGridMap"
},
102: {
name: "bus stops",
endpoint: "https://hsqxipv6u8.execute-api.us-east-2.amazonaws.com/mockslide102/",
description: "This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description.",
visualization: "ScatterPlotMap"
},
106: {
name: "bike lanes",
endpoint: "https://cekaghbj2k.execute-api.us-east-2.amazonaws.com/mockslide106",
description: "This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description.",
visualization: "PathMap"
},
001: {
name: "bike parking",
endpoint: "http://service.civicpdx.org/neighborhood-development/sandbox/slides/bikeparking/",
description: "Bike Parking: This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description.",
visualization: "ScatterPlotMap"
},
002: {
name: "bike lanes",
endpoint: "http://service.civicpdx.org/neighborhood-development/sandbox/slides/bikelanes/",
description: "Bike Lanes: This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description.",
visualization: "PathMap"
},
003: {
name: "parks",
endpoint: "http://service.civicpdx.org/neighborhood-development/sandbox/slides/parks/",
description: "Parks: This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description.",
visualization: "PolygonPlotMap"
},
004: {
name: "multi-use trails",
endpoint: "http://service.civicpdx.org/neighborhood-development/sandbox/slides/multiusetrails/",
description: "Multi-use Trails: This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description.",
visualization: "PathMap"
},
005: {
name: "community gardens",
endpoint: "http://service.civicpdx.org/neighborhood-development/sandbox/slides/communitygardens/",
description: "Community Gardens: This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description.",
visualization: "SmallPolygonMap"
}
},
foundations: {
103: {
name: "zip codes",
endpoint: "https://ctyhsin0r2.execute-api.us-east-2.amazonaws.com/mockfoundation103",
description: "This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description.",
visualization: "ChoroplethMap",
attributes: {
primary: {
field: "happy_index",
name: "average happy index rating"
},
secondary: {
field: "parks",
name: "number of parks"
}
}
},
104: {
name: "neighborhoods",
endpoint: "https://rnc45keyjk.execute-api.us-east-2.amazonaws.com/mockfoundation104",
description: "This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description.",
visualization: "ChoroplethMap",
attributes: {
primary: {
field: "name",
name: "neighborhood name"
},
secondary: {
field: null,
name: null
}
}
},
105: {
name: "voter precincts",
endpoint: "https://qfvf1wpc3l.execute-api.us-east-2.amazonaws.com/mockfoundation105",
description: "This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description. This is a fake description.",
visualization: "ChoroplethMap",
attributes: {
primary: {
field: "avg_voter_turnout_pct",
name: "average voter turnout"
},
secondary: {
field: "median_income",
name: "median income"
}
}
},
006: {
name: "Property Value",
endpoint: "http://service.civicpdx.org/neighborhood-development/sandbox/foundations/taxlotblockgroups/",
description: "Property Value: This is a fake description. This is a fake description. This is a fake description. This is a fake description. ",
visualization: "ChoroplethMap"
}
}
}
}
Goal: setup the requested /civic-sandbox/api/packages
route to answer with the Lambda function, rather than answering from /civic-sandbox
as is does right now.
Execution log for request 87463648-6ceb-11e8-ba1a-35a46d27507c
Sun Jun 10 20:19:07 UTC 2018 : Starting execution for request: 87463648-6ceb-11e8-ba1a-35a46d27507c
Sun Jun 10 20:19:07 UTC 2018 : HTTP Method: GET, Resource Path: /
Sun Jun 10 20:19:07 UTC 2018 : Method request path: {}
Sun Jun 10 20:19:07 UTC 2018 : Method request query string: {}
Sun Jun 10 20:19:07 UTC 2018 : Method request headers: {}
Sun Jun 10 20:19:07 UTC 2018 : Method request body before transformations:
Sun Jun 10 20:19:07 UTC 2018 : Endpoint request URI: https://lambda.us-west-2.amazonaws.com/2015-03-31/functions/arn:aws:lambda:us-west-2:845828040396:function:civicSandboxPackages/invocations
Sun Jun 10 20:19:07 UTC 2018 : Endpoint request headers: {x-amzn-lambda-integration-tag=87463648-6ceb-11e8-ba1a-35a46d27507c, Authorization=**************************************************************************************************************************************************************************************************************************************************************************************05224d, X-Amz-Date=20180610T201907Z, x-amzn-apigateway-api-id=43iqjy2gjk, X-Amz-Source-Arn=arn:aws:execute-api:us-west-2:845828040396:43iqjy2gjk/test-invoke-stage/GET/, Accept=application/json, User-Agent=AmazonAPIGateway_43iqjy2gjk, X-Amz-Security-Token=FQoDYXdzEB0aDGRYk21vDhnXiEnqyCK3AwcjWELWm+IuXhcsdFO6+KLwLLuK7200+sdM3tFc8H591odBeQ3SniDF6zl/pZw3EKuoRSznXUYs15I8PETyZox7kegk2J6WGiqpAoEQwvgds/C3080OOlebwAST8O3FRIw6Y+OGTO7g505lKM27vLVmRH7JWr1mdrbjOPaVGW5uqCAY5hSu/CHr+vuzsVnGpblOxj6CCFZ9dByZeiLRE8e+7EUCKMUIh70qn3ORJ5IyuLsCuxA5YJGu+Iv6qvHWHw/LveT4EMt6XWIpML0zlMK4MvpOxnwCdc+hyQfEI8S09gVtQgWTk5mRR4+DqJuXS5XTP7yGLf34Jnm+QCYsr/R9WVW0OS [TRUNCATED]
Sun Jun 10 20:19:07 UTC 2018 : Endpoint request body after transformations:
Sun Jun 10 20:19:07 UTC 2018 : Sending request to https://lambda.us-west-2.amazonaws.com/2015-03-31/functions/arn:aws:lambda:us-west-2:845828040396:function:civicSandboxPackages/invocations
Sun Jun 10 20:19:07 UTC 2018 : Received response. Integration latency: 245 ms
Sun Jun 10 20:19:07 UTC 2018 : Endpoint response body before transformations: {"statusCode": 200, "body": {"packages": {"transportation": {"description": "Transportation: This is a fake description. This is a fake description. This is a fake description. This is a fake description.", "foundations": ["103", "105", "104"], "default_foundation": "103", "slides": ["102", "106", "001", "002"], "default_slide": "106"}, "affordable housing": {"description": "Transportation: This is a fake description. This is a fake description. This is a fake description. This is a fake description.", "foundations": ["105", "104", "006"], "default_foundation": "105", "slides": ["101", "102", "003", "004"], "default_slide": "101"}, "neighborhoods": {"description": "Transportation: This is a fake description. This is a fake description. This is a fake description. This is a fake description.", "foundations": ["104", "006"], "default_foundation": "006", "slides": ["101", "102", "106", "005"], "default_slide": "102"}}, "slides": {"001": {"name": "bike parking", "end [TRUNCATED]
Sun Jun 10 20:19:07 UTC 2018 : Endpoint response headers: {X-Amz-Executed-Version=$LATEST, x-amzn-Remapped-Content-Length=0, Connection=keep-alive, x-amzn-RequestId=87479629-6ceb-11e8-a048-79fdeab69ef2, Content-Length=5847, Date=Sun, 10 Jun 2018 20:19:07 GMT, X-Amzn-Trace-Id=root=1-5b1d87bb-77e4919a70765cb35d547f84;sampled=0, Content-Type=application/json}
Sun Jun 10 20:19:07 UTC 2018 : Method response body after transformations: {"statusCode": 200, "body": {"packages": {"transportation": {"description": "Transportation: This is a fake description. This is a fake description. This is a fake description. This is a fake description.", "foundations": ["103", "105", "104"], "default_foundation": "103", "slides": ["102", "106", "001", "002"], "default_slide": "106"}, "affordable housing": {"description": "Transportation: This is a fake description. This is a fake description. This is a fake description. This is a fake description.", "foundations": ["105", "104", "006"], "default_foundation": "105", "slides": ["101", "102", "003", "004"], "default_slide": "101"}, "neighborhoods": {"description": "Transportation: This is a fake description. This is a fake description. This is a fake description. This is a fake description.", "foundations": ["104", "006"], "default_foundation": "006", "slides": ["101", "102", "106", "005"], "default_slide": "102"}}, "slides": {"001": {"name": "bike parking", "endpoi [TRUNCATED]
Sun Jun 10 20:19:07 UTC 2018 : Method response headers: {Access-Control-Allow-Origin=*, X-Amzn-Trace-Id=Root=1-5b1d87bb-77e4919a70765cb35d547f84, Content-Type=application/json}
Sun Jun 10 20:19:07 UTC 2018 : Successfully completed execution
Sun Jun 10 20:19:07 UTC 2018 : Method completed with status: 200
Execution log for request e6a41f57-6ceb-11e8-ba66-c7fdc9f70b8c
Sun Jun 10 20:21:47 UTC 2018 : Starting execution for request: e6a41f57-6ceb-11e8-ba66-c7fdc9f70b8c
Sun Jun 10 20:21:47 UTC 2018 : HTTP Method: GET, Resource Path: /api
Sun Jun 10 20:21:47 UTC 2018 : Method request path: {}
Sun Jun 10 20:21:47 UTC 2018 : Method request query string: {}
Sun Jun 10 20:21:47 UTC 2018 : Method request headers: {}
Sun Jun 10 20:21:47 UTC 2018 : Method request body before transformations:
Sun Jun 10 20:21:47 UTC 2018 : Endpoint request URI: https://lambda.us-west-2.amazonaws.com/2015-03-31/functions/arn:aws:lambda:us-west-2:845828040396:function:arn:aws:execute-api:us-west-2:845828040396:43iqjy2gjk/invocations
Sun Jun 10 20:21:47 UTC 2018 : Endpoint request headers: {x-amzn-lambda-integration-tag=e6a41f57-6ceb-11e8-ba66-c7fdc9f70b8c, Authorization=**************************************************************************************************************************************************************************************************************************************************************************************e79cfc, X-Amz-Date=20180610T202147Z, x-amzn-apigateway-api-id=43iqjy2gjk, X-Amz-Source-Arn=arn:aws:execute-api:us-west-2:845828040396:43iqjy2gjk/test-invoke-stage/GET/api, Accept=application/json, User-Agent=AmazonAPIGateway_43iqjy2gjk, X-Amz-Security-Token=FQoDYXdzEBwaDLuw3V9bOsZ2EzbA1yK3A4cP1ReSYk5jOO1nM1YoyM8ArF9pFfxnYYWvxYRsaZDi66TgsCbKx8BIdXpiV8NGwnxtYvy0dVlwoGRiBeB8lNJWapn0MhnPeDUkf0Tsq0pZ/XTf5LFxU2cJ1VTFyfKyfaOhJypW8ro08E2EJAVovRpeOIR2i+w0/8YpV51qPdenyJYfVgSEeEjwchOL/PY0yjq+8ezzWX3melKvL29O5hgMTPDkB6HzSXR0AqlYdv6x5SHChqn+AIE3Xmuq5mQzrFp5aI5zjjWpdPFhomgWUhrNfiFQ9B/o2dvDmjuG6iG4giB+/Zxb3I2zXZWq1KBXLu4by1XdTE7921zN5oE4AdHajaS [TRUNCATED]
Sun Jun 10 20:21:47 UTC 2018 : Endpoint request body after transformations:
Sun Jun 10 20:21:47 UTC 2018 : Sending request to https://lambda.us-west-2.amazonaws.com/2015-03-31/functions/arn:aws:lambda:us-west-2:845828040396:function:arn:aws:execute-api:us-west-2:845828040396:43iqjy2gjk/invocations
Sun Jun 10 20:21:47 UTC 2018 : Received response. Integration latency: 5 ms
Sun Jun 10 20:21:47 UTC 2018 : Endpoint response body before transformations: {"message":"1 validation error detected: Value 'arn:aws:lambda:us-west-2:845828040396:function:arn:aws:execute-api:us-west-2:845828040396:43iqjy2gjk' at 'functionName' failed to satisfy constraint: Member must satisfy regular expression pattern: (arn:(aws[a-zA-Z-]*)?:lambda:)?([a-z]{2}((-gov)|(-iso(b?)))?-[a-z]+-\\d{1}:)?(\\d{12}:)?(function:)?([a-zA-Z0-9-_\\.]+)(:(\\$LATEST|[a-zA-Z0-9-_]+))?"}
Sun Jun 10 20:21:47 UTC 2018 : Endpoint response headers: {Connection=keep-alive, x-amzn-RequestId=e6a49424-6ceb-11e8-b236-0ddc2d935ba0, x-amzn-ErrorType=ValidationException, Content-Length=398, Date=Sun, 10 Jun 2018 20:21:47 GMT, Content-Type=application/json} Sun Jun 10 20:21:47 UTC 2018 : Lambda invocation failed with status: 400 Sun Jun 10 20:21:47 UTC 2018 : Execution failed: 1 validation error detected: Value 'arn:aws:lambda:us-west-2:845828040396:function:arn:aws:execute-api:us-west-2:845828040396:43iqjy2gjk' at 'functionName' failed to satisfy constraint: Member must satisfy regular expression pattern: (arn:(aws[a-zA-Z-]*)?:lambda:)?([a-z]{2}((-gov)|(-iso(b?)))?-[a-z]+-\d{1}:)?(\d{12}:)?(function:)?([a-zA-Z0-9-.]+)(:(\$LATEST|[a-zA-Z0-9-]+))? Sun Jun 10 20:21:47 UTC 2018 : Method completed with status: 400
- THE DIFFERENCE:
- Good URI: https://lambda.us-west-2.amazonaws.com/2015-03-31/functions/arn:aws:lambda:us-west-2:845828040396:function:civicSandboxPackages/invocations
- Bad URI: https://lambda.us-west-2.amazonaws.com/2015-03-31/functions/arn:aws:lambda:us-west-2:845828040396:function:arn:aws:execute-api:us-west-2:845828040396:43iqjy2gjk/invocations
- Thus, I deleted the `/api` GET entry and created it with Lambda function = "arn:aws:lambda:us-west-2:845828040396:function:civicSandboxPackages/invocations"
- which throws a better-looking warning "You are about to give API Gateway permission to invoke your Lambda function: arn:aws:lambda:us-west-2:845828040396:function:civicSandboxPackages/invocations"
- but which then throws an error "An unknown error occurred"
- and when I test that with the "/api GET" Test function, returns an execution log of:
Execution log for request 0443eef3-6ced-11e8-9e05-0beac8d1f5d2
Sun Jun 10 20:29:46 UTC 2018 : Starting execution for request: 0443eef3-6ced-11e8-9e05-0beac8d1f5d2
Sun Jun 10 20:29:46 UTC 2018 : HTTP Method: GET, Resource Path: /api
Sun Jun 10 20:29:46 UTC 2018 : Method request path: {}
Sun Jun 10 20:29:46 UTC 2018 : Method request query string: {}
Sun Jun 10 20:29:46 UTC 2018 : Method request headers: {}
Sun Jun 10 20:29:46 UTC 2018 : Method request body before transformations:
Sun Jun 10 20:29:46 UTC 2018 : Endpoint request URI: https://lambda.us-west-2.amazonaws.com/2015-03-31/functions/arn:aws:lambda:us-west-2:845828040396:function:civicSandboxPackages/invocations/invocations
Sun Jun 10 20:29:46 UTC 2018 : Endpoint request headers: {x-amzn-lambda-integration-tag=0443eef3-6ced-11e8-9e05-0beac8d1f5d2, Authorization=**7269b0, X-Amz-Date=20180610T202946Z, x-amzn-apigateway-api-id=43iqjy2gjk, X-Amz-Source-Arn=arn:aws:execute-api:us-west-2:845828040396:43iqjy2gjk/test-invoke-stage/GET/api, Accept=application/json, User-Agent=AmazonAPIGateway_43iqjy2gjk, X-Amz-Security-Token=FQoDYXdzEB0aDNhCHBNH6tWnU+arPiK3A46CAVIAG58uLAkZ8+/WVi9FPnTAk5EXr9/YOPhu1L/eR0YsGQHoPXwzglRsKrWac4WPwvucMlyxsQbTkfAwkwQGa2OH/VCrbWbqVvjKdsC2qD95gWaYj73S1BTHUdWN+vJELTef1xOeCW1CACcMdQ1C753up66uXZBISg0zKVBmEU/tzLD/XURInCiR3JJ/J0RGJVM1nZdSYYlbk1N9YJf2aI4EF2QK/yzY0/hFq2k4Wii06oyQwp8xW2JnWIoomKbVS62aq/bMJhrylVR4HywTLZ54nAX3dj0DO4/REyAg6r8DT0Fbg8pKhQTg5p7MX4wmlCRrzbW19v+gVTIX/1bA1jd [TRUNCATED]
Sun Jun 10 20:29:46 UTC 2018 : Endpoint request body after transformations:
Sun Jun 10 20:29:46 UTC 2018 : Sending request to https://lambda.us-west-2.amazonaws.com/2015-03-31/functions/arn:aws:lambda:us-west-2:845828040396:function:civicSandboxPackages/invocations/invocations
Sun Jun 10 20:29:46 UTC 2018 : Received response. Integration latency: 2 ms
Sun Jun 10 20:29:46 UTC 2018 : Endpoint response body before transformations:
Sun Jun 10 20:29:46 UTC 2018 : Endpoint response headers: {Connection=keep-alive, x-amzn-RequestId=0444b245-6ced-11e8-845a-47dfea39f2c3, Content-Length=130, Date=Sun, 10 Jun 2018 20:29:46 GMT} Sun Jun 10 20:29:46 UTC 2018 : Lambda invocation failed with status: 403 Sun Jun 10 20:29:46 UTC 2018 : Execution failed due to configuration error: Sun Jun 10 20:29:46 UTC 2018 : Method completed with status: 500
### WORSE:
This has somehow made the URI https://sandbox.civicpdx.org/civic-sandbox return a Forbidden message, so I've deleted the entire `/api` Resource from the AWS Gateway - but that still hasn't fixed the 403 Forbidden error we're now getting from that request (even though the internal Test - from the "/ GET" method in the API Gateway console - is receiving a 200 response).
The Civic Sandbox should be seen as another piece of the Civic Platform. For this reason, despite the Sandbox API being hosted with AWS Lambda, it should have URLs that are familiar to our family of services.
civicSandboxPackages
Lambda functionI'm not sure what the best AWS service for creating this routing rule is, so this ticket is equal parts research and implementation.
cc @AraOshin