jack1902
commented
4 years ago Another alternative is to consider using a docker container. This exact issue is one I've ran into when using an XML parsing library within a Serverless Framework Project using the Python Requirements plugin which allows you to "dockerize pip" during the packaging process.
Perhaps the user can have the option of using docker or vagrant? So when running the deploy command it would detect if the environment in use is vagrant, otherwise fall onto the python3.7 lambci container
The containers in question: https://hub.docker.com/r/lambci/lambda/
Background
StreamAlert apps currently have some pre-packaged zips due to the need for natively compiled libraries in the lambda environment (shakes fist at cryptography lib). These are extracted as part of the lambda packaging process and injected into the other pip-installed OS independent libraries.
This is currently needed because we cannot guarantee that people are deploying from a Linux environment. For instance, if we did not have these packaged libs, and installed directly with
pip, users on macOS would run into issues when their code runs in the AWS lambda environment.The biggest problem with this is that it's very annoying to update dependencies when we need to, and requires some janky stuff. See the README.rst for apps.
Desired Change
Leverage the new vagrant stuff to build Lambda packages. We should also look at using Docker for this.
Notes
The vagrant environment would only need to be used for certain lambda packages, and we should create some sort of flag that says
skip_vagrant(with the default being using vagrant to be safe). I think the only package right now that needs it is theappsfunction. This would probably speed up deploys of functions that do not have OS specific dependencies.Regarding the above change - if (read: when) we move to a better management model, where the
streamalertandstreamalert_clicode is not stored with config / rules, we will want to the option to let users "toggle" the vagrant usage for any given function. This could be part of the clideploy --function...command or otherwise, but would need to exist.