search

mozilla-services / python-dockerflow

A Python package to implement tools and helpers for Mozilla Dockerflow
https://python-dockerflow.readthedocs.io
Mozilla Public License 2.0
38 stars 22 forks source link

Current usability for AWS lambda? #11

Open hwine opened 7 years ago

hwine commented 7 years ago

Similar to #1, but with a non-app-framework in mind. In particular the logging functionality, but possibly others such as config.

I'm starting a new project intended to be deployed as an AWS Lambda function. Would your recommendation be to start with python-dockerflow, or stick with mozlog for now?

jezdez commented 7 years ago

@hwine That's a good question, as dockerflow is by design targeted at Docker based deployment workflows I'm not sure if that should be used for AWS Lambda deployments. It'd be best to ask the authors of Dockerflow @mostlygeek (and maybe @whd, @robotblake, @jasonthomas) what they think about it.

willkg commented 7 years ago

I just did a AWS Lambda function. I didn't use this package at all. The configuration in python-dockerflow is Django-specific, so that won't fly for Lambda (unless you're doing crazy things).

The logging bits might be helpful, but logs for AWS Lambda go to Cloudwatch, so for it to be really helpful, we'd have to have something that slurps from Cloudwatch and into our logging infrastructure.

Also, if you're using Datadog, then you print the statsd data to stdout and Datadog picks it up by reading Cloudwatch. You can't have that in mozlog format.

I have some rough infrastructure in Pigeon for configuration, logging, and statsd things if that helps. I'd be game for pulling that into python-dockerflow if that helps:

https://github.com/mozilla/socorro-pigeon/blob/735c183ecc626f1bc31648cc68f8f33870aea6c2/pigeon.py#L34

https://github.com/mozilla/socorro-pigeon/blob/735c183ecc626f1bc31648cc68f8f33870aea6c2/pigeon.py#L93

https://github.com/mozilla/socorro-pigeon/blob/735c183ecc626f1bc31648cc68f8f33870aea6c2/pigeon.py#L153

mostlygeek commented 7 years ago

The only time I would consider dockerflow for AWS lambda is if you could spin up docker containers in lambda. Definitely use mozlog for logging as all the downstream log processing is expecting it.

willkg commented 7 years ago

@mostlygeek Are you saying that logging 2.0 pulls logging events from Cloudwatch? I'm pretty sure Pigeon logs aren't going to the logging infrastructure and I was under the impression that wasn't a thing.

mostlygeek commented 7 years ago

@willkg @relud is best to answer that question.

lambda logs go to cloudwatch logs. I don't see why those can't make it into the loggion2.0 pipeline. I don't think this is a currently supported by logging2.0 yet.

hwine commented 7 years ago

I'm new to this group -- would there be value in having a Mozilla (which has MPL) setup for "all the things". If so, adding @willkg logging and other support for Lambda might be worthwhile.

jvehent commented 7 years ago

I'm pretty sure Pigeon logs aren't going to the logging infrastructure and I was under the impression that wasn't a thing.

I most certainly should be. All applications logs should eventually go into the logging pipeline. As for the timeline of doing so, that's a question for ops.

willkg commented 7 years ago

@jvehent @mostlygeek: I had no idea. Good to know. I updated Pigeon to use python-dockerflow for mozlog-style logging.