Lambda functions and internet access

[andrewkrug]

Turns out lambda functions don't get any access to the internet without the presence of a cost prohibitive NAT gateway. This means that lambda functions running inside of the ThreatResponse AWS account will need to POST their results in a different way than runtimes out in the wild.

Potential options are:

1. Profiler writes a {uuid.hex()}.json.gz file directly to S3 the same way the API does
2. Profiler writes to dynamo and we pick that up somewhere else ( seems unnecessary )
3. Deploy NAT gateway. ( Not cost effective ).
4. Deploy the Lambda in the same VPC as the API box and point it directly at the API instead.

So I'm sure that you've gathered option 1 is preferred. It's just a matter of writing a little logic that only does the S3 upload if you're running from within a lambda function. We'll still need the urllib2.Request method in the python profiler that @jeffbryner wrote. Oddly it doesn't actually cause the function to fail. Simply nothing ever happens...

Option 4. isn't a bad choice either but has implications for if / when we want to go multi-region and puts heavier requirements on the CI/CD pipeline to attach things.

[andrewkrug]

I should also note here that if the lambda functions are writing direct to s3 we'll need to use a custom lambda execution role that gives the lambda function s3:putObject and nothing more on the bucket "threatresponse.showdown"

[andrewkrug]

5th option... we could also consume these from CloudWatch events.

[jeffbryner]

s3 seems the simplest?

[andrewkrug]

@jeffbryner that's the direction I'm going ... I'm going to expand the scope of the CI/CD workflow this evening to accommodate.

You may have noticed I refactored the post call out into a function where I'm calling open on the request. From there can catch the exception and fall back to S3 if anything but a status code 200 is returned. https://github.com/ThreatResponse/python-lambda-inspector/blob/master/main.py#L134

[tnem]

Agreed with S3 - no need to add another component.

We might encounter similar problems with other cloud environments, but can deal with those when we come to them (or maybe use their equivalents).

[andrewkrug]

Closed in a81acf6412104bb0769891a033a1847d9e3a37d3