Fix: deploy layers when hot reload enabled

[joe4dev]

Layers are not deployed if hot reloading is enabled through mountCode: true. In the Serverless AWS provider, Lambda functions and layers are deployed together (uploadFunctionsAndLayers) but serverless-localstack has currently many limitations with hot reloading (e.g., global config, no layer support). This PR provides a workaround to unblock customers who want to hot-reload functions and regularly deploy layers.

Root cause

Hot reloading (i.e., mountCode) in serverless-localstack breaks any Lambda layers with the new Lambda provider:

custom:
  localstack:
    lambda:
      mountCode: true

When trying to deploy a Lambda function with a layer, it fails with an OperationalError:

OperationalError: ENOENT: no such file or directory, open '/Users/joe/Projects/LocalStack/issues/support_aaporomu_u04ba3n4bam/serverless-lambda-layers-minimal/.serverless/layerOne.zip'

When mountCode is enabled, layers do not get packaged and deployed: Skip plugin function Package.packageService (lambda.mountCode flag is enabled)

Therefore, the deployment fails.

Challenges

• mountCode: true is a global option but currently does not support hot-reloading of layers (new feature in LocalStack 2.0).
• Hot reloading (currently) only works with at most one layer per function. Hence, if we intend to apply mountCode to layers as well, it fails if there is any function with more than one layer in the serverless.yml
• mountCode is bad terminology because the current implementation does not do any mounting anymore but rather copies ZIP files (https://docs.localstack.cloud/user-guide/tools/lambda-tools/hot-reloading/). hotReload would be a better name
• We need to consider the scope of hot reloading because not everyone wants to enable hot reloading globally. Selectively enabling hot reloading for single functions or layers under development would be helpful.
• mountCode: true uses the current directory (assumed to be project directory) as default hot reload path or accepts relative paths (e.g., ./functions). This might lead to many unnecessary reloads.

Background

• Relevant code:
  • this.skipIfMountLambda('AwsDeploy', 'uploadFunctionsAndLayers'); disables layer uploading, which happens together with functions
  • this.skipIfMountLambda('Package', 'packageService'); disables packaging in general
• Serverless documentation on how to provide custom plugin configuration options: https://www.serverless.com/framework/docs/guides/plugins/custom-configuration
• It is possible to add custom function-level properties using "Function properties via defineFunctionProperties" via defineFunctionProperties
  • There is no well-defined mechanism to add layer-level properties. It might be possible to do manually using addPropertiesToSchema to add something like defineLayerProperties?! Might need to relax provider schema by allowing additional properties like for functions.patternProperties here https://github.com/serverless/serverless/blob/ed15cb27aee68954c93d875da96274914943ad71/lib/classes/config-schema-handler/index.js#L335
  • There is some API to extend configuration: https://www.serverless.com/framework/docs/guides/plugins/extending-configuration
• Serverless v3 comes with breaking changes regarding plugin variable resolving (e.g., comes with stricter validation): https://www.serverless.com/framework/docs/guides/upgrading-v3#new-variable-resolver-engine

Workaround

Removing the skipIfMountLambda circumvents the issue at the expense of slower initial deployment (with potential other deployment-related side-effects):

• this.skipIfMountLambda('Package', 'packageService');
• this.skipIfMountLambda('AwsDeploy', 'uploadFunctionsAndLayers');

Testing

• Works against customer sample layer_problem (shared in Pro support)
• Works against Serverless v3 tested with Python layer (just function code support hot reloading)
• Works with serverless-python-requirements plugin where dependencies are packaged as layer (but slow due to default root directory)

Side-effects

• Initial deployment takes longer because packaging and deployment are now done even for Lambda functions that are hot reloaded (that's unnecessary).

To fix this issue, we need to think about a new fine-grained API for hot reloading and handle selective deployments of all layers and functions that are not hot reloading.

Questions

1. Is there a dependency of AwsCompileFunctions.downloadPackageArtifacts ⇒ package (i.e., can we safely skip packaging without downloadPackageArtifacts?
2. Any other dependencies or implicit compatibility issues (e.g., related to other serverless plugins)?