search

serverless / serverless

⚡ Serverless Framework – Effortlessly build apps that auto-scale, incur zero costs when idle, and require minimal maintenance using AWS Lambda and other managed cloud services.
https://serverless.com
MIT License
46.39k stars 5.7k forks source link

Layer Service: No file matches include / exclude patterns #5892

Open engmsaleh opened 5 years ago

engmsaleh commented 5 years ago

This is a Bug Report

Description

provider: name: aws

layers: psycopg2: buildScript: ./build.sh path: layer compatibleRuntimes:

Similar or dependent issues:

Additional Data

bweigel commented 5 years ago

Try running your build.sh manually before doing sls deploy. Serverless doesn't seem to run the buildscript. I had the same issue & building the layer folder beforehand fixed it.

dschep commented 5 years ago

Yup, buildScript isn't a framework option. Where did you see this documented?

bweigel commented 5 years ago

I guess when I started experimenting with layers I copied from here https://github.com/dschep/geoip-lambda-layer/blob/master/serverless.yml :smile:

But you are right. Nothing in the official docs.

dschep commented 5 years ago

Ahah! thanks. I knew it was my fault bc I had a plugin version i never relased which did have that.

engmsaleh commented 5 years ago

@bweigel Thanks a lot building it before the deploy did the trick 👍 @dschep It would be very nice to had this feature included :)

engmsaleh commented 5 years ago

Another question, Could I when deploy the layer be without the stage indicator ... for example dev as it makes no sense to me to have such suffix in the ARN of the layer as it will be all the same to me?

dogzzdogzz commented 5 years ago

My exclude/include in serverless.yml as below worked well before,

package:
  exclude:
    - ./**
  individually: true
functions:
  hello:
    package:
      include:
      - hello/**

until I added layers, I got " No file matches include / exclude patterns" error

layers:
  foo:
    path: bar
ghost commented 5 years ago

Same here. Using exclude and include when using layers leads to this error No file matches include / exclude patterns The configuration used is the following:

package:
  exclude:
    - "**"
  include:
    - functions/**

layers:
  layerOne:
    path: layerOne

The exclude and include works fine until I add the layers section. The problem seems to be the exclude: "**" since excluding specific directories works fine

gangster commented 5 years ago

I'm seeing the same behavior once I add the layers config. In my case, I am not declaring a package nor am I including/excluding anything explicitly anywhere in my yml file. This is on v1.4.0

sashokbg commented 5 years ago

Same problem here. I am trying to exclude node_modules but then add them to a layer and I get this error. Using 1.4.0

washedrepent commented 5 years ago

I am having the same issue. I am using CodeBuild and its causing this error. I am not even using layers. Is their any solution to this? Locally works fine. Issue only happens in container during build.

kjcsb1 commented 5 years ago

I'm experiencing the same issue.

service: docstore-api-v1
provider:
  name: aws

layers:
  super:
    path: /path/to/super-lambda-function/

package:
  individually: true

functions:
  create_doc:
    handler: src.doc_api.create_doc.lambda_handler
    description: Creates a document owned by the current user
    layers:
      - { Ref: SuperLambdaLayer }
    package:
      individually: true
      include:
        - src/doc_api/create_doc.py

Serverless Error ---------------------------------------

No file matches include / exclude patterns

Your Environment Information --------------------------- OS: linux Node Version: 8.15.1 Serverless Version: 1.42.2

src/doc_api/create_doc.py does indeed exist

henhal commented 5 years ago

I have the same problem, and building manually before invoking SLS is not really working for me. Sure I can wrap the SLS deploy in a script that runs build and then proceeds, but it doesn't fit my environment very well. For some other stuff I'm using hooks such as package:createDeploymentArtifacts to invoke commands that produce files that are then used by SLS. For layers, this hook doesn't seem to be called before the execution fails.

Is there any way to get SLS itself to run a script during deployment that produces the files to be put in a layer?

Edit: Strike the above, using the package:initialize hook worked!

custom:
  scripts:
    hooks:
      package:initialize: ./fetch_my-lib.js # fetches my-lib and puts it in .generated/src/my-lib
layers:
  myLib:
    path: .generated/src/my-lib
dchao19 commented 5 years ago

I had the same problem. I had ./** specified in the root (i.e top level - not the individual layer include/exclude) package:exclude option and I kept running into the No file matches include / exclude patterns bug.

I did some digging and it turns out that the error occurs while packaging the layer, not the functions. Even though the matching of the patterns occurs against the individual layer directory itself, Serverless also runs all of the layer files to package against the same patterns of the root service. Here's an example:

Let's say your serverless.yml has the following, root-level, package section:

package:
  individually: true
  exclude: 
    - ./**
  include:
    - ./functions/**/*

And your layers are structured in the service like this: 

serverless.yml
layers/
└─ node_modules
       └─ package-1/
       └─ package-2/
       └─ package-3/
functions/

When serverless is packaging your layer, it is only checking the files in the layers directory, so it is checking:

layers/node_modules/package-1/**
layers/node_modules/package-2/**

But serverless is checking those files against the patterns specified in the root package:exclude and because ./** matches every file and the include-pattern ./functions/**/* matches none, no files are actually included in the layer, which causes the error.

You can fix this (if you want to add everything in your layer directory to the package) by adding an additional package:include section to your layer configuration. For instance: 

layers:
  nodeModules:
    package:
      include:
        - ./**

Because the layers:nodeModules:package:include is not checked when packaging a function or an entire service, adding the ./** include to the layer only impacts how the layer is packaged. Serverless "scopes" the packaging of the layer to be checking files that are only in the layers:nodeModules:path so other packaging operations are not impacted.

lbiselli commented 5 years ago

Any news on this issue?

sanjay-zymr commented 5 years ago

I am also facing the same issue. Does anyone have a solution for the same?

ultrarunner commented 5 years ago

I am not sure this will help anyone but the issue resolved itself when I added a physical file in the folder that had the exclude path listed. I had the following in the serverless.yml file:

package:
  exclude:
    - nltk/**

And my folder structure was as such:

serverless.yml
nltk/
└─ nltk_data

but I did not have any files yet in it. As soon as I added a physical file, the error went away. Not sure it makes sense but I hope this helps.

igorrocha commented 5 years ago

I was having the same issue, even though I wasn't specifying any include/exclude directories. Turns out the path property in the layer definition cannot be the path of an empty directory.

layers: MyLayer: path: postgraphileCache/ # required, path to layer contents on disk -> CANNOT BE AN EMPTY FOLDER

@kjcsb1 @dogzzdogzz @ronaldonc @sashokbg @gangster @sanjay-zymr I believe that might be the case for you as well.

henhal commented 4 years ago

This issue seems to describe quite a few different symptoms, but to address the one with layers not being fetched/installed, hence no node_modules folder being present, which could result in an empty layer and the symptoms described, I wrote a plugin: https://www.npmjs.com/package/serverless-plugin-layer-manager

devalnor commented 4 years ago

Same issue here. I can't include any folder or file into the zip file. I don't use layer btw.

package:
  individually: true
...
functions:
  seed:
    package:
      include:
        - db/seeds/*.json
    handler: functions/seed.seed
    events:
      - http:
          path: seed
          method: get
          cors: true
          private: true
Siva1432 commented 4 years ago

I had the same issue. Not sure what the cause but its gone after adding node_modules back into the package exclude. I have a layer which packages one of the dependencies. Hopefully this will help someone.

Screen Shot 2019-12-11 at 7 24 38 PM
lobanov commented 4 years ago

The key missing piece for me was to understand that package include/exclude paths for a layer are resolved relative to the layer's own path directory, not the directory where the serverless.yml file is. This is very confusing because if you don't specify layer's package instructions, it takes service-level package include/exclude paths, which do not make sense for the layer.

I suppose it isn't really a bug in SLS, it's just lack of clarity in the documentation.

mottikadosh commented 4 years ago

I had this terrible issue and after some hours of investigation, I was discovered that the problem was using not include by an exclamation mark is some other functions in the same project.

This is an example of function f1 in my serverless project (serverless throws No file matches include/exclude patterns error)

before change: 

  f1:
    handler: myProj/myProjHandler.lambda_handler
    timeout: 20
    package:
      include:
        - "./myProj/myProjHandler.py"
        - "./myProj/common/**"
        - "./myProj/services/**"
        - "./myProj/data/**"
        #not including
        - "!**/tests/**"
        - "!**/.idea/**"
        - "!**/node_modules/**"
        - "!**/.serverless/**"
        - "!**/serverless/**"
        - "!**/venv/**"

Once I was replaced the not include (by an exclamation mark) to explicit exclude in all serverless project and the issue was solved :)

After change: 

  f1:
    handler: myProj/myProjHandler.lambda_handler
    timeout: 20
    package:
      exclude:
        - '**'
      include:
        - "./myProj/myProjHandler.py"
        - "./myProj/common/**"
        - "./myProj/services/**"
        - "./myProj/data/**"
faheel commented 4 years ago

I was facing the same No file matches include / exclude patterns issue with layers with the following:

package:
  exclude:
    - "**"
  include:
    - "!**"
    - functions/functionA/**
    - functions/functionB/**

layers:
  LayerA:
    path: layers/layerA
  LayerB:
    path: layers/layerB

Where the directory structure for my service was:

serverless.yml
layers
├── layerA
└── layerB
functions
├── functionA
└── functionB

What worked for me was specifying the package for each layer relative to the path. So I had to update the path and specify the package for the layers as follows:

layers:
  LayerA:
    path: layers
    package:
      exclude:
        - "**"
      include:
        - "!**"
        - layerA/**
  LayerB:
    path: layers
    package:
      exclude:
        - "**"
      include:
        - "!**"
        - layerB/**

@dchao19's comment helped me realize that the issue is because the service-level package was being applied to the layers, and didn't match anything. And when even after specifying the package for the layers I was getting the same error, I tried specifying the include paths relative to the layer's path, which finally worked!

It would be nice if the documentation for Lambda layers mentioned this.

akshaynarula commented 4 years ago

Encountered this issue today, was working fine earlier. Updated the serverless version Working fine without including package inside layer.

zoellner commented 4 years ago

I'm getting the same error even without a layers section to build layers. just pulling in a layer with a arn reference for one of the functions

Abhijith-Nagaraja commented 3 years ago

I am encountering this in the lastest serverless 2.21.1

I am trying some skeleton deployment using python. Here is my serverless.yaml

My folder structure is

serverless-test |_lambdas |handler.py |layers |common |somefunction.py

service: serverless-test

frameworkVersion: '2'

provider:
  name: aws
  runtime: python3.8
  lambdaHashingVersion: 20201221

  stage: test
  region: us-west-2

functions:
  hello:
    handler: lambdas/handler.hello

This works fine. Now as soon as I add a layer, I get the following error

No file matches include / exclude patterns

service: serverless-test

frameworkVersion: '2'

provider:
  name: aws
  runtime: python3.8
  lambdaHashingVersion: 20201221

  stage: test
  region: us-west-2

functions:
  hello:
    handler: lambdas/handler.hello
    layers:
      - {Ref: CommonLambdaLayer}

layers:
  common:
    path: layers/common
    name: common-module
    description: common set of functions

I also tried adding include and exclude patterns. But it didn't solve my problem

service: serverless-test

frameworkVersion: '2'

provider:
  name: aws
  runtime: python3.8
  lambdaHashingVersion: 20201221

  stage: test
  region: us-west-2

package:
  individually: true
  exclude: 
    - ./**
  include:
    - ./lambdas/**

functions:
  hello:
    handler: lambdas/handler.hello
    layers:
      - {Ref: CommonLambdaLayer}

layers:
  common:
    path: layers/common
    name: common-module
    description: common set of functions
    package:
      include:
        - ./**

how to solve this issue

humodz commented 3 years ago

@Abhijith-Nagaraja

Try setting your layer's package like this:

package:
  include:
    - '**/**'

For reference, my serverless.yml:

service: hello-serverless

provider:
  name: aws
  runtime: nodejs12.x
  lambdaHashingVersion: 20201221
  versionFunctions: false

package:
  excludeDevDependencies: false
  exclude:
    - '**/**'
  include:
    - handler.js

layers:
  dependencies:
    path: layers/dependencies
    name: dependencies
    description: "node_modules layer"

    # Overrides service-level package setting and prevents
    # the error "no files match include/exclude pattern"
    package:
      include:
        - '**/**'

functions:
  hello:
    handler: handler.hello
    layers:
      - !Ref DependenciesLambdaLayer

Serverless version: 2.23.0

yorjaggy commented 3 years ago

Hi everyone, I was facing the same situation and I solved it just by adding the package.json to the folder I specify in the path parameter. Lets say:

layers:
  testLayers:
    path: ./
    name: myFirstLayer
    description: Testing Layer to reduce package size in ImageUploaderService

So, I have all my dependencies in a folder called nodejs, then I was able to run sls deploy normally image

Important things to create Layers: Link

Manitej66 commented 3 years ago

Guys, finally I was able to get it worked!

here is my layer structure (name the folder as nodejs )

image

Now in the serverless.yml file add below code

layers:
  common:
    path: ./
    name: dev-layer
    compatibleRuntimes:
      - nodejs12.x
    retain: true
    package:
      patterns:
        - "!/*"
        - "nodejs/**"
pgrzesik commented 3 years ago

Hey @lizaw - could you share a minimal reproducible example? A lot of reported cases are for outdated versions and often caused by misconfiguration

lizaw commented 3 years ago

Hi @pgrzesik, there is mis-configuration on my serverless.yml. Original the layer path as `layers: layer1: path: mymodule compatibleRuntimes:

pgrzesik commented 3 years ago

Thanks for letting us know @lizaw :bow:

bitsofinfo commented 2 years ago

this is unbelievably annoying. I want my functions in one zip, and node_modules as a separate layer. My layers for the node_modules works fine, but no matter what I cannot exclude it from the functions zip. What gives?

arnm commented 2 years ago

still an issue

edit: I found this plugin which actually works, https://github.com/agutoli/serverless-layers