provide hook for resolving binary dependencies and bundling them in zip file

[softprops]

see https://github.com/softprops/serverless-rust/issues/23 for context

This should be pretty straight forward. sketch thoughts, capture whats missing from the output of ldconfig -c -new -p from the list of dependencies provided by ldd {bin-name}. The most straightforward to expose such a hook is likely the presence of an env variable ZIP_DEPS or something similar.

[bernardobelchior]

I have been giving some thought on how to solve this and would like your opinion.

How would you feel of having a scripts folder that would contain hooks to run? This way downstream users would have a lot of flexibility. There could be default scripts like build.sh (that could be overriden), but also have other hooks, e.g., install.sh (which could install binary dependencies), package.sh (which could package the application).

By using inversion of control, the final user would have much more flexibility and we could also set sane defaults. What are your thoughts on this?

[triptec]

Any update on this?

[bernardobelchior]

I would still like to try to tackle this issue, but I'd like a green light from a contributor to make sure the effort wouldn't be for nothing.

@softprops you seem to be the main contributor, could you give your opinion here?

[softprops]

Sounds good. What's the approach you'd be leaning towards? I don't want to be a blocker here

[bernardobelchior]

What I'm imagining is something like this:

1. There would be a scripts/ directory inside the lambda-rust root directory with two scripts: one for building and one for packaging, which would be the current build.sh split into these two phases.
2. Inside the scripts/ directory, every script would start (by convention) with a 3-digit number so that scripts can be run in a predictable order. The order would be the one given by sort.

The consumers of lambda-rust could have their own scripts/ directory where they would also add their own scripts. If a user script has the exact same name of a lambda-rust script, then only the user script would be run.

Using the 3-digit notation before the name, I propose the initial scripts directory of lambda-rust to be something like:

• 300package.sh
• 700build.sh

The reason behind these numbers is the ability to add a significant amount of scripts before or after each step of the build process.

Example:

lambda-rust/scripts:

.
├── 300package.sh
└── 700build.sh

consumer-lambda/scripts:

.
├── 100install-native-dependencies.sh
└── 300package.sh

The scripts would be run in the following order:

1. consumer-lambda/scripts/100install-native-dependencies.sh
2. consumer-lambda/scripts/300package.sh
3. lambda-rust/scripts/700build.sh

If one scripts fails (i.e., returns non-zero) then the whole process would fail. Note that the lambda-rust/scripts/300package.sh was overridden and would not run.

---

Possible problems and solutions:

1. User could already have a scripts/ directory in their project. To solve this, the name of this directory would be configurable using a environment variable, .e.g SCRIPTS_DIR.
2. By adding this new functionality without a version bump, we could break consumers (e.g., if they already have a scripts directory that would be run without them knowing that would happen). To address this, we can either (1) bump the version of the docker image or (2) make the SCRIPTS_DIR functionality opt-in, meaning that if the environment variable is not set, then it will just run the default scripts.

I can't think of other problems or possible things that could break. Do you have any insights here or have I covered all the cases?

What is your opinion on the approach?

[softprops]

My preference would be to remove as much ceremony as possible so there's less to maintain for users that don't need this feature. I bit of the recent debug options have has started to feel that way. As much as possible only design for what you need and nothing more.

As few considerations. I separately maintain a serverless framework plugin for rust which leverages this builder for building rust lambdas consistent with the lambda runtime env. There are some discussions around adding support for building lambdas in a host env outside of docker. That becomes more complicated as this docker image does more than just a cargo build. A consideration there is how that works for windows users.

I see the default case for cargo is not needing anything at all. Cargo provides built-in support for a build script that will automatically get ran.

I like the idea of lifecycle hooks but I'm not sure about the need for having multiple copies of them. There operations I see are

Install

Currently there's a hook to yum install extra deps if you need them as docker run args https://github.com/softprops/lambda-rust/blob/cff2b3ad865d02c17b14ade7e14bc3a2234d597b/build.sh#L19. this impl could be a fallback in absence of an install script or env var

Build

Caro build is pretty basic as mentioned above it comes out of the box with build script support

https://github.com/softprops/lambda-rust/blob/cff2b3ad865d02c17b14ade7e14bc3a2234d597b/build.sh#L25 could be the fallback of a build script were absent but I'd like to hear more about how cargo build doesn't fit your rust bin needs

Package

This one is very has a very specific contract. The lambda runtime expects a zip file with an executable called "bootstrap" inside it. That's essentially all we're doing now https://github.com/softprops/lambda-rust/blob/cff2b3ad865d02c17b14ade7e14bc3a2234d597b/build.sh#L46. I'm not sure what a package script hook would look like but I'm open to suggestions. The problem to solve there is a need to generate a list to additional assets to collect into the zip.

"scripts" is pretty common name for a utility dir. I'd like to avoid surprises if you happened to have a scripts dir with a script sharing the same name has one of the hook scripts we'd be introducing and have it not be intended for use with this yool. I'd suggest something a bit more specific to the tool. In this case perhaps .lambda-rust/{install,build,package}

If you'd like to start sketching something out given the above feedback is be open to it. I'm a fan of iterating with small changes then learning how well they work in practice before making big changes. If there's a way to implement one of these hooks and see if it solves a problem that would be cool too.

[bernardobelchior]

I like your ideas. I'll your concerns point by point.

As few considerations. I separately maintain a serverless framework plugin for rust which leverages this builder for building rust lambdas consistent with the lambda runtime env. There are some discussions around adding support for building lambdas in a host env outside of docker. That becomes more complicated as this docker image does more than just a cargo build. A consideration there is how that works for windows users.

I don't understand what you mean. If there are discussions about adding support for building lambdas outside docker, what does it have to do with this project?

---

I like the idea of lifecycle hooks but I'm not sure about the need for having multiple copies of them.

The rationale was to provide users that need high flexibility the possibility to customize the process as they see fit. But I agree that it may be over-engineering at this point.

---

I'm not sure what a package script hook would look like but I'm open to suggestions. The problem to solve there is a need to generate a list to additional assets to collect into the zip.

It's possible to add files to a zip file using the zip command. This is how I'm adding files to the zip in my use case.

---

"scripts" is pretty common name for a utility dir. I'd like to avoid surprises if you happened to have a scripts dir with a script sharing the same name has one of the hook scripts we'd be introducing and have it not be intended for use with this yool. I'd suggest something a bit more specific to the tool. In this case perhaps .lambda-rust/{install,build,package}

I agree that scripts would not be a good name if hardcoded. Do you think having a SCRIPTS_DIR would be useful? If not, is there any reason to call it .lambda-rust rather than lambda-rust?

---

Thanks for the thorough feedback! I plan to start working on this shortly