Layers are packaged with absolute file paths in serverless-state.json

[HaaLeo]

Are you certain it's a bug?

• [X] Yes, it looks like a bug

Is the issue caused by a plugin?

• [X] It is not a plugin issue

Are you using the latest version?

• [X] Yes, I'm using the latest version

Is there an existing issue for this?

• [X] I have searched existing issues, it hasn't been reported yet

Issue description

Description

When packaging a layer serverless adds an absolute path to the .serverless/serverless-state.json file. For the lambda functions paths seem to be relative. The absolute path in the .serverless/serverless-state.json causes trouble in CI/CD environments.

Steps to reproduce

1. We are bundeling everything in a first step (Absolute path is added to the file).
2. Upload the .serverless directory temporary as build artifacts.
3. download the .serverless to our deployment environment and run the deployment with the command npm run deploy -- --stage ${ENVIRONMENT} --package ${CODEBUILD_SRC_DIR_BuildArtifact}/.serverless/ --region ${REGION}.

When doing this now the generated absolute path of course is wrong since we deploy all of this from a completely different instance.

Expected Result

when running sls package all path to zip archives should be relative including for layers

Actual Result

when running sls package all layers' paths are absolute

Example

A example for a generated serverless-state.json file

{
....
    "layers": {
        "swagger": {
          "path": "swaggerLayer",
          "name": "dev-fastify-swagger",
          "description": "fastify-swagger@4.12.6",
          "compatibleRuntimes": {
            "$ref": "$[\"service\"][\"provider\"][\"compiledCloudFormationTemplate\"][\"Resources\"][\"SwaggerLambdaLayer\"][\"Properties\"][\"CompatibleRuntimes\"]"
          },
          "package": {
            "artifact": "/Users/myUser/some/folder/.serverless/swagger.zip"
          }
        }
      }
}     

Workaround

Currently we fixed this by using sed to modify the file and replace the absolute path:

sed -i'' 's|"artifact"\: ".*\.serverless/swagger\.zip"|"artifact":"\.serverless/swagger\.zip"|g' .serverless/serverless-state.json

However, this obviously is a hacky workaround no sustainable solution to address this issue.

Service configuration (serverless.yml) content

service: ${env:APP_ID} # This value is present in the CI/CD environment

useDotenv: true # See https://www.serverless.com/framework/docs/environment-variables/

plugins:
  - serverless-webpack
  - serverless-offline
  - serverless-plugin-reducer

package:
  individually: true

provider:
  name: aws
  runtime: nodejs14.x
  region: ${opt:region, 'eu-central-1'}
  stage: ${opt:stage, 'dev'}
  endpointType: REGIONAL
  deploymentBucket:
    name: ${env:S3_BUCKET} # This value is present in the CI/CD environment
  stackTags:
    AppId: ${env:APP_ID}
    Environment: ${self:provider.stage}
  environment:
    REGION: ${self:provider.region}
    STAGE: ${self:provider.stage}
  iam:
    deploymentRole: ${env:CF_ROLE_ARN}
  lambdaHashingVersion: 20201221 # To prepare serverless v3.0 compatibility. https://serverless.com/framework/docs/deprecations/#LAMBDA_HASHING_VERSION_V2
  httpApi:
    cors:
      allowedOrigins:
        - '*'
      allowedHeaders:
        - '*'

layers:
  swagger:
    path: swaggerLayer
    name: ${self:provider.stage}-fastify-swagger
    description: fastify-swagger@4.12.6
    compatibleRuntimes:
      - nodejs12.x
      - nodejs14.x

functions:
  main:
    handler: src/main.handler
    memorySize: 128
    timeout: 30
    layers:
      - !Ref SwaggerLambdaLayer

    events:
      - httpApi: # ApiGatewayV2
          method: any
          path: /{proxy+}

resources:
  - extensions:
      HttpApiStage:
        Properties:
          StageName: ${self:provider.stage} # Due to https://github.com/serverless/serverless/issues/8367

# Other serverless files which are not important for the bug
  - ${file(serverless/mappings.yml)}

  - ${file(serverless/database.yml)}

  - ${file(serverless/main-lambda-resources.yml)}
  - ${file(serverless/api-gateway-domain.yml)}

Command name and used flags

npm run sls -- package

Command output

N/A

Environment information

Framework Core: 2.68.0 (local)
Plugin: 5.5.1
SDK: 4.3.0
Components: 3.18.1

[medikoo]

@HaaLeo great thanks for report. PR with a fix is welcome!

[HaaLeo]

@medikoo can you point me towards the right direction which files may must be touched to apply a fix?

[medikoo]

@HaaLeo I've just double-checked, and current behavior is consistent in all cases (service, function, layer) we store the absolute path to package.artifact in the state file.

I agree that it's not a great design, and ideally if in all cases paths are relative, but that may demand a more careful refactor.

We're open to PR that brings that in, but due to limited time constraints, I cannot also provide a comprehensive guide on how to go with it.

I'll start with coining the code points where the absolute path is assigned (it can be done by searching for .artifact = or artifact: phrases, then I'd change them to a relative, and ensure that in places they're used we resolve full path via path.resolve(servicePath, artifactPath). Technically we could skip the last part and most likely it'll work (as when relative path is provided, Node.js will resolve it against working directory, and currently in all cases it'll be a service directory). However we plan to allow customization of the service directory (so it's not necessary a current working directory), and for that, we need to ensure that those paths will always be resolved against service directory

[HaaLeo]

@medikoo thx for the quick response and the guidance.

Regarding:

I've just double-checked, and current behavior is consistent in all cases (service, function, layer) we store the absolute path to package.artifact in the state file.

I think it is not consistent. In my project's serverless-state.json the lambda function's path is relative:

{
...
    "package": {
      "artifact": ".serverless/main.zip"
    }
}

However, the layer's path is not

[medikoo]

In plain Serverless service (no plugins involved), definitely all paths are absolute

e.g. check this example of service with layers involved, with committed and generated state file:

https://github.com/medikoo/test-serverless/tree/layers

Function package.artifact: https://github.com/medikoo/test-serverless/blob/7011f933d3e9c9706ecba5eb0a21e129675760eb/.serverless/serverless-state.json#L340

Layer package.artifact:

https://github.com/medikoo/test-serverless/blob/7011f933d3e9c9706ecba5eb0a21e129675760eb/.serverless/serverless-state.json#L368

[HaaLeo]

Yep that's a good catch. I forgot about the plugins. In my case probably the serverless-webpack plugin causes the relative path.

[LeeGDavis]

Goodness, @medikoo thank you for the tip on the plugins! Typescript plugin broke this for us... Unfortunately I can't get that time back 😬 .

[rswheeldon]

Just ran into the same problem also with the typescript plugin. Commenting how that plugin allowed my layer to be added correctly but obviously then the code doesn't exist to run. @LeeGDavis Did you find a good workaround for this?

[LeeGDavis]

@rswheeldon I found the commits I made around this time and the specific fix is a little fuzzy now... It appears I moved to a prerelease Serverless Plugin TypeScript 3.0.0.xxx to get around this. I don't see any other hacks to work-around this particular issue.

[rswheeldon]

@LeeGDavis Awesome. Many thanks. That (almost) sorted it. For the record, I used 3.0.0-pre.82c9bc17 which at the time of writing seems to be the only preview available. This introduced another bug in which it produces the following error:

Error: EISDIR: illegal operation on a directory, unlink '/home/richard/cogcred/api-text_search/.build/node_modules'

Which I've hacked with the disgusting but effective line in my Makefile:

    sed -i 's/fs.unlinkSync(outModulesPath)/fs.removeSync(outModulesPath)/g' node_modules/serverless-plugin-typescript/dist/src/index.js

As I result I now have typescript code in a lambda calling a compiled exe from a layer. Hence, I hope / suspect a final 3.x release should fix this cleanly.