RFC: Use existing Lambda Layers with `referenceFunctionLayer`

[josefaidt]

Hey folks :wave: we’re looking to introduce a mechanism to bind and consume existing Lambda Layers with Functions defined with defineFunction. This will not facilitate the creation and management of Lambda Layers, however provides you with a way to consume Node.js packages that require OS-specific addons without requiring a dependency on Docker to build Functions locally. This is a follow-up to the following issues:

• https://github.com/aws-amplify/amplify-backend/issues/1281
• https://github.com/aws-amplify/amplify-backend/issues/1432

The primary use case we’re seeking to address with this feature is the ability to use dependencies like sharp that require OS-specific addons to function as expected in the Node.js Lambda runtime environment, which requires linux addons. When building Functions locally (i.e. on deploy), Amplify uses esbuild, which will bundle modules that are imported in your source code. With layers, you need to prevent esbuild from bundling your layer code. Using CDK it is possible to force a Function build using Docker, however we would like to avoid requiring this dependency.

In addition to the bundler complications, you also need a way to provide a similar type-safe experience that you get when authoring your Function’s handler. For example, if you’re using a layer that provides access to specific npm packages like sharp, you should expect to see the same APIs and type-safety you’d see when installing and using the package normally. This is typically accomplished by marking certain dependencies as “external”.

import { defineFunction, referenceFunctionLayer } from "@aws-amplify/backend"

defineFunction({
  name: "my-function-with-layers",
  layers: {
    // keyed by module name
    sharp: referenceFunctionLayer("arn:.../my-sharp-layer"),
  },
})

The introduction of referenceFunctionLayer comes with an added layers prop on defineFunction . This object is keyed by the module name hosted on your existing layer. In the example above, we’re binding a layer that simply hosts the sharp dependency to the sharp key. When esbuild executes, sharp will be externalized and available via your layer at runtime.

For layers hosting multiple dependencies, you can create multiple key/value pairs with the same layer reference:

import { defineFunction, referenceFunctionLayer } from "@aws-amplify/backend"

defineFunction({
  name: "my-function-with-layers",
  layers: {
    // keyed by module name
    sharp: referenceFunctionLayer("arn:.../my-sharp-layer"),
    "some-other-dependency": referenceFunctionLayer("arn:.../my-sharp-layer"),
  },
})

Let us know what you think!

[MarlonJD]

Hey @josefaidt, on #1602 I'm trying to build using directly function itself in define function like in RFC #1543, we can use layers in there, I think, after finished this PR, I'll try this. It would be like this:

defineFunction((scope) => {
  return new NodejsFunction(scope, 'customFunction', {
    entry:
      './packages/backend-function/src/test-assets/default-lambda/handler.ts',
    layers: {
      // keyed by module name
      sharp: referenceFunctionLayer("arn:.../my-sharp-layer"),
    ]},
});

What are you think about this? Or should we add this directly to amplify-backend as defineFunction without callback ?

[josefaidt]

Hey @MarlonJD :wave: this would be a bit different and built onto the DefineFunctionProps as a way to specify external dependencies that are bound to a layer. For the sharp use case, it requires linux addons and those can only be installed when building on a linux host, which for pure CDK this is typically done with Docker. You can currently apply layers to Functions with CDK, however you need a way to specify which dependencies should be external to the bundle (to avoid bundling sharp with the macos addons, for example)

[mraspberry]

So this is purely intended for application dependencies, not extensions such as datadog or the AWS Parameters and Secrets extension?

[sjcengine]

This feature would be great! Is there any way we can use this now? If not, what's the current recommended work around for using packages like Sharp? Thanks in advance.