Cannot Deploy; EMFILE: too many open files

GrantPerkins commented 2 years ago

Are you certain it's a bug?

[X] Yes, it looks like a bug

Is the issue caused by a plugin?

[X] It is not a plugin issue

Are you using the latest version?

[X] Yes, I'm using the latest version

Is there an existing issue for this?

[X] I have searched existing issues, it hasn't been reported yet

Issue description

When running serverless deploy through CLI, I get some variation of the below error.

 Serverless Error ----------------------------------------

  Cannot read file node_modules\rxjs\operator\groupBy.d.ts due to: EMFILE: too many open files, open 'C:\Users\gcper\Code\190\Website\lambda\node_modules\rxjs\operator\groupBy.d.ts'

I cannot deploy to my lambda functions.

Service configuration (serverless.yml) content

service: team190

frameworkVersion: '2'

provider:
  name: aws
  runtime: python3.8
  iamRoleStatements:
    - Effect: "Allow"
      Action:
        - dynamodb:Query
        - dynamodb:Scan
        - dynamodb:GetItem
        - dynamodb:PutItem
        - dynamodb:UpdateItem
        - dynamodb:DeleteItem
      Resource: "*"
plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    dockerizePip: non-linux

functions:
  login:
    handler: handler/handler.login
  choose:
    handler: handler/handler.assign_role
  request:
    handler: handler/student_handler.request
  get_requests:
    handler: handler/uber_handler.get_requests
  get_records:
    handler: handler/handler.get_records
  confirm_requests:
    handler: handler/uber_handler.confirm_requests
  create_poll:
    handler: handler/uber_handler.create_poll
  get_users:
    handler: handler/uber_handler.get_all_users
  create_event:
    handler: handler/uber_handler.create_event
  get_pending_requests:
    handler: handler/student_handler.get_pending_requests
  delete_pending_requests:
    handler: handler/student_handler.delete_pending_requests

Command name and used flags

serverless deploy

Command output

Serverless: Running "serverless" installed locally (in service node_modules)
Serverless: Deprecation warning: Resolution of lambda version hashes was improved with better algorithm, which will be used in next major release.
            Switch to it now by setting "provider.lambdaHashingVersion" to "20201221"
            More Info: https://www.serverless.com/framework/docs/deprecations/#LAMBDA_HASHING_VERSION_V2
Serverless: Generated requirements from C:\Users\gcper\Code\190\Website\lambda\requirements.txt in C:\Users\gcper\Code\190\Website\lambda\.serverless\requirements.txt...
Serverless: Using static cache of requirements found at C:\Users\gcper\AppData\Local\UnitedIncome\serverless-python-requirements\Cache\b729430ab9c5421667932cf132fc2b7c75df8e7e25338978deedaf75b5fec293_slspyc ...
Serverless: Packaging service...
Serverless: Excluding development dependencies...

 Serverless Error ----------------------------------------

  Cannot read file node_modules\rxjs\operator\groupBy.d.ts due to: EMFILE: too many open files, open 'C:\Users\gcper\Code\190\Website\lambda\node_modules\rxjs\operator\groupBy.d.ts'

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information ---------------------------
     Operating System:          win32
     Node Version:              14.16.1
     Framework Version:         2.64.1 (local)
     Plugin Version:            5.5.0
     SDK Version:               4.3.0
     Components Version:        3.17.1

Environment information

Serverless: Running "serverless" installed locally (in service node_modules)
Framework Core: 2.64.1 (local)
Plugin: 5.5.0
SDK: 4.3.0
Components: 3.17.1

medikoo commented 2 years ago

@GrantPerkins great thanks for reporting. Can you post the output but when command is run with SLS_DEBUG=* flag?

DominikIlski commented 2 years ago

Hi i have the same problem and to my knowledge it is not dependend on config files. This error is really hard to reproduce, and the higher chance to do so is when node_module is filled with packages. My coleagues running the same command on the same repo does not have this problem. But unfortunetly it does exists and its stops me from developing :( For eg. part of my package.json file:

"devDependencies": {
    "@babel/core": "^7.15.5",
    "@babel/preset-env": "^7.15.6",
    "@babel/register": "^7.15.3",
    "prettier": "^2.4.1",
    "serverless-lift": "^1.9.0"
  },
  "dependencies": {
    "@sentry/serverless": "^6.13.3",
    "aws-sdk": "^2.997.0",
    "node-fetch": "^2.6.5",
    "node-fetch-retry": "^2.0.0",
    "yup": "^0.32.11"
  }

This is my output after running sls deploy -s prod:

Serverless: To ensure safe major version upgrades ensure "frameworkVersion" setting in service configuration (recommended setup: "frameworkVersion: ^2.64.1")

Serverless: Load command interactiveCli
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command config:tabcompletion
Serverless: Load command config:tabcompletion:install
Serverless: Load command config:tabcompletion:uninstall
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command upgrade
Serverless: Load command uninstall
Serverless: Load command lift
Serverless: Load command lift:eject
Serverless: Load command HooksQueue:logs
Serverless: Load command HooksQueue:send
Serverless: Load command HooksQueue:failed
Serverless: Load command HooksQueue:failed:purge
Serverless: Load command HooksQueue:failed:retry
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command generate-event
Serverless: Load command test
Serverless: Load command dashboard
Serverless: Load command output
Serverless: Load command output:get
Serverless: Load command output:list
Serverless: Load command param
Serverless: Load command param:get
Serverless: Load command param:list
Serverless: Load command studio
Serverless: Skipping variables resolution with old resolver (new resolver reported no more variables to resolve)
Serverless: Configuration warning at 'constructs.HooksQueue': unsupported configuration format
Serverless:  
Serverless: Learn more about configuration validation here: http://slss.io/configuration-validation
Serverless:  
Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...

 Serverless Error ----------------------------------------

  ServerlessError: Cannot read file node_modules\@types\lodash\fp\thru.d.ts due to: EMFILE: too many open files, open 'C:\PRACA_STUDIA\PRACACODEWAVE\webhooks-execution\node_modules\@types\lodash\fp\thru.d.ts'
      at C:\Users\DominioPianino\AppData\Roaming\nvm\v15.0.1\node_modules\serverless\lib\plugins\package\lib\zipService.js:126:15
      at tryCatcher (C:\Users\DominioPianino\AppData\Roaming\nvm\v15.0.1\node_modules\serverless\node_modules\bluebird\js\release\util.js:16:23)
      at Promise._settlePromiseFromHandler (C:\Users\DominioPianino\AppData\Roaming\nvm\v15.0.1\node_modules\serverless\node_modules\bluebird\js\release\promise.js:547:31)
      at Promise._settlePromise (C:\Users\DominioPianino\AppData\Roaming\nvm\v15.0.1\node_modules\serverless\node_modules\bluebird\js\release\promise.js:604:18)
      at Promise._settlePromise0 (C:\Users\DominioPianino\AppData\Roaming\nvm\v15.0.1\node_modules\serverless\node_modules\bluebird\js\release\promise.js:649:10)
      at Promise._settlePromises (C:\Users\DominioPianino\AppData\Roaming\nvm\v15.0.1\node_modules\serverless\node_modules\bluebird\js\release\promise.js:725:18)
      at _drainQueueStep (C:\Users\DominioPianino\AppData\Roaming\nvm\v15.0.1\node_modules\serverless\node_modules\bluebird\js\release\async.js:93:12)
      at _drainQueue (C:\Users\DominioPianino\AppData\Roaming\nvm\v15.0.1\node_modules\serverless\node_modules\bluebird\js\release\async.js:86:9)
      at Async._drainQueues (C:\Users\DominioPianino\AppData\Roaming\nvm\v15.0.1\node_modules\serverless\node_modules\bluebird\js\release\async.js:102:5)
      at Immediate.Async.drainQueues [as _onImmediate] (C:\Users\DominioPianino\AppData\Roaming\nvm\v15.0.1\node_modules\serverless\node_modules\bluebird\js\release\async.js:15:14)
      at processImmediate (node:internal/timers:462:21)

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information ---------------------------
     Operating System:          win32
     Node Version:              15.0.1
     Framework Version:         2.64.1
     Plugin Version:            5.5.0
     SDK Version:               4.3.0
     Components Version:        3.17.1

and when there are some fixes for a linux based environment, but it is hard to achieve on a windows. According to: https://stackoverflow.com/questions/8965606/node-and-error-emfile-too-many-open-files

Note: File that is shown in a error output is not always the same one.

medikoo commented 2 years ago

@DominikIlski we have EMFILE prevention baked in. Still it cannot handle eventual sync fs calls which we practice only when reading user configuration.

Other corner case scenario, where eventually EMFILE may sneak unhadled, is if any shared dependency between serverless and .e.g @servleress/components requires fs and attaches to it's functions in module initialization.

If there is such scenario anywhere, then this will be fixed with next release (as with https://github.com/serverless/serverless/pull/10131 we've ensured to not require other CLI's modules if they're not run)

migcoldev commented 2 years ago

I could deploy on my PC but not on the laptop. The next day, without making any updates, no changes, I can no longer deploy both. I get the same "EMFILE: too many open files" error

Today updated nodejs, serverless, and still with issue.

FrodoBaggyMcBaggins commented 2 years ago

FWIW - I was/am experiencing this with Python. I created a virtual environment for each Lambda handler - and it looks like serverless is including all of the /Lib and /Scripts directories for each virtual environment, thereby inflating the file count needed to package.

Adding exclusions "solved" this for me.

jvanalst commented 2 years ago

I just also ran into this while working on Windows 10 randomly. Nothing changed on my system or project except for an npm audit fix that upgraded a single package.

I tried reinstalling my packages from scratch but had no joy.

Framework Core: 2.71.0 (local) Plugin: 5.5.3 SDK: 4.3.0 Components: 3.18.1

jvanalst commented 2 years ago

I ended up using serverless-plugin-include-dependencies@v4.1.0 and it solved the problem. It also dramatically decreased build time. I'm not a fan of having the extra dependency though, but it does seem to be a viable work around.

medikoo commented 2 years ago

I've moved this issue to https://github.com/serverless/serverless-python-requirements repository, as it seems specific to this plugin

FrodoBaggyMcBaggins commented 2 years ago

@medikoo I don't believe that's the case - this issue can be encountered without having serverless-python-requirements installed. It seems more closely related to the combination of serverless with a virtual environment in the directory being packaged.

medikoo commented 2 years ago

@FrodoBaggyMcBaggins can you create a sample reproduction case, that doesn't involve any plugins, and provide a link?

jvanalst commented 2 years ago

@medikoo - I can guarantee it's not specific to the serverless-python-requirements plugin (I'm not using that one at all). I can't rule out other plugins having similar issues though, as I'm using:

plugins:
  - serverless-s3-sync
  - serverless-offline
  - serverless-export-env

medikoo commented 2 years ago

@jvanalst are you capable to prepare a reproduction test case that doesn't involve any plugins? Or if not, can you single out which plugin causes that?

jvanalst commented 2 years ago

@medikoo Checking now if I can replicate it if I roll back my project a few commits...

Yup... replicable. And I can trigger it without any Serverless Plugins at all:

org: bananaweb
app: bananarego
service: ${file(./config.js):common.service}
frameworkVersion: '2'
provider:
  name: aws
  runtime: nodejs14.x
  region: us-west-2
functions:
  api:
    handler: api/index.apiHandler

That api/index.js file can be empty, BTW. Nothing in there affects what I'm experiencing.

Currently looking through the package.json to see if anything in there affects it. Stripping the serverless-plugins from devDependencies (with clean npm install of course) ... no effect. Stripping ALL the other devDependencies... also no effect.

So... starting to strip out dependency dependencies.... also no effect. Well this is annoying.

{
  "name": "bananarego",
  "version": "0.0.0-alpha",
  "private": true,
  "dependencies": {
    "serverless": "^2.65.0"
  },
  "devDependencies": {
  }
}

Is it a problem with Serverless 2.65? If it is... 2.72.2 has the same exact bug in it. So that's not it probably. So far all we've managed to change is which node module shows up in the error message.

Serverless Error ---------------------------------------- Cannot read file node_modules\archiver-utils\node_modules\safe-buffer\index.js due to: EMFILE: too many open files, open 'C:\Users...\node_modules\archiver-utils\node_modules\safe-buffer\index.js'

I guess I'll start randomly removing files from the the project's base folder... Maybe it's trying to include something weird like include the frontend that's explicitly blocked in the original code.

Serverless Error ----------------------------------------

Cannot read file node_modules\aws-sdk\lib\services\cloudfront.d.ts due to: EMFILE: too many open files, open 'C:\Users...\node_modules\aws-sdk\lib\services\cloudfront.d.ts'

Okay... wiped the project down to package-lock.json, package.json, api/index.js, and node_modules containing only a serverless v2.27.2 install....

Serverless Error ----------------------------------------

Cannot read file node_modules\kafka-node\node_modules\debug.travis.yml due to: EMFILE: too many open files, open 'C:\Users...\node_modules\kafka-node\node_modules\debug.travis.yml'

Wow. Seriously? Still broken? Am I doing something crazy? Okay fine... blow away the package-lock and package.json just in case... so bare node_modules with an api/index.js (empty) and the serverless.yml above.

Serverless Error ----------------------------------------

Cannot read file node_modules\@sindresorhus\is\dist\index.d.ts due to: EMFILE: too many open files, open 'C:\Users...\node_modules\@sindresorhus\is\dist\index.d.ts'

Still broken..... Last shot. Gonna try it all in a completely different folder on a completely different hard drive. I'm also going to start from scratch in a totally new folder.

Serverless Error ----------------------------------------

Cannot read file node_modules\traverse\examples\stringify.js due to: EMFILE: too many open files, open 'C:\Users...\node_modules\traverse\examples\stringify.js'

Still? I'm not nuts am I? Okay... let's get out of the Users folder entirely. Maybe Windows is doing something weird like trying to sync OneDrive (even though I'm pretty sure that second SSD isn't sunk). Oh also, I figured out you don't actually need that api/index.js file. It'll still break on "Excluding development dependencies..."

Serverless Error ----------------------------------------

Cannot read file node_modules\d\node_modules\type\CHANGELOG.md due to: EMFILE: too many open files, open 'C:\test\node_modules\d\node_modules\type\CHANGELOG.md'

Okay... I'm at a total loss. All I can think of is that a bug got introduced to Serverless at some point. That or I'm a bizarre edge case.

jvanalst commented 2 years ago

Also tried installing a few older versions of Serverless: v2.50.0 - same issue. v2.25.0 - same issue. v2.0.0 - same issue.

I wonder if something in Windows changed.

medikoo commented 2 years ago

@jvanalst can you post a reproduction case on GitHub as a project (please also include the content of node_modules) ? That will allow us hopefully provide some solution if we happen to reproduce that on our side

jvanalst commented 2 years ago

https://github.com/jvanalst/emfile-serverless-win10-bug

fsobh commented 2 years ago

So i closed my project on VS code, and opened a terminal window in the project directory. I Ran sls deploy and everything worked fine. Could it possibly be a compatibility issue with VS code ?

jvanalst commented 2 years ago

I get the error regardless of VS Code vs PowerShell vs Command Prompt.

On Mon, Feb 7, 2022 at 10:49 AM 0xD1x0n @.***> wrote:

So i closed my project on VS code, and opened a terminal window in the project directory. I Ran sls deploy and everything worked fine. Could it possibly be a compatibility issue with VS code ?

— Reply to this email directly, view it on GitHub https://github.com/serverless/serverless-python-requirements/issues/662#issuecomment-1031746382, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACGSXKMFQ6W3LSAOWRI4SLU2AA4LANCNFSM5L3H26IA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.*** com>

oliverkoehler commented 2 years ago

Issues still persists, here is my output: Error: EMFILE: too many open files, open 'C:\Users\olli9\Documents\coding\graphql-ms\node_modules\@amplitude\types\dist\src\ampli.js'

fsobh commented 2 years ago

Also tried installing a few older versions of Serverless: v2.50.0 - same issue. v2.25.0 - same issue. v2.0.0 - same issue.

I wonder if something in Windows changed.

Dont bother with version 3. The frameworks STILL broken

piotr-szewczyk commented 2 years ago

Same here:

Cannot read file node_modules\rxjs\dist\types\internal\operators\sequenceEqual.d.ts due to: EMFILE: too many open files, open 'C:\Users\...\Desktop\...\...\node_modules\rxjs\dist\types\internal\operators\sequenceEqual.d.ts'

can someone look at this issue?

patrick-long commented 2 years ago

@medikoo Looks like this is still an issue -- I've run into a couple of times before and have been able to work around it with either one or both of the following plugins: serverless-plugin-common-excludes and serverless-plugin-include-dependencies.

However, this issue seems to just randomly come up when trying to run sls deploy. Have you been able to look into why this issue still persists despite saying multiple times that it should be fixed next release or that it's currently fixed? Because, truth be told it is not fixed yet.

As I was writing this comment, I was JUST able to get past the problem by running npm install serverless-plugin-include-dependencies@4.1.0 specifically. I was using this package at version 5.0.0 earlier with no luck. However, my project is now running properly using version 4.1.0.

I hope this helps someone else who is experiencing this very frustrating issue with serverless framework!

aerdeyn commented 2 years ago

@patrick-long thanks for the suggested work-around. I've also had this issue occurring from time to time. I had a good run of no issues and then had the EMFILE error pop up again. This was without any changes to dependencies in package.json. Reverting to serverless-plugin-include-dependencies@4.1.0 didn't do the trick in this case. Also, deleting and re-installing the packages in the node_modules folder didn't make a difference (although it seems to do so previously).

tobelesa commented 2 years ago

Hello, I got the EMFILE error when launching the serverless "deploy" script. I used graceful-fs like this to solve the issue :

install graceful-fs: npm install graceful-fs
go to file: node_modules/serverless/lib/plugins/package/lib/zip-service.js

replace this line : const fs = BbPromise.promisifyAll(require('fs')); by this:

var realFs = require('fs')
var gracefulFs = require('graceful-fs')
gracefulFs.gracefulify(realFs)
const fs = BbPromise.promisifyAll(realFs);

vijay-immanent commented 2 years ago

@medikoo ... have been able to work around it with either one or both of the following plugins: serverless-plugin-common-excludes and serverless-plugin-include-dependencies.

Just installing serverless-plugin-common-exludes worked for me. Thanks! 🙇

AGiantSquid commented 2 years ago

A colleague saw the EMFILE: too many files open error recently, and the error was only showing up on their computer. It took a while for us to figure it out what the issue was, but it ended up being a package glob issue.

We had manually listed exclude glob patterns in the serverless.yml like this:

package:
  individually: true
  exclude:
    - .git/**
    - node_modules/**
    - .venv/**

The colleague had created their virtual environment in a directory called venv instead of .venv, so serverless didn't know to skip their virtual environment when packaging the lambda. It was trying to read everything in their virtual environment which blew up the memory.

We updated the glob pattern to the following:

package:
  individually: true
  include:
    - '!*'
    - '!**'
    - src/*.py

This pattern prevents anything other than python files in the src dir from getting packaged. It turns out we got speed benefits with this new syntax as well, as the glob pattern is much faster when you use ! in an include rather than listing files in the exclude because the glob takes a while to find every matching file that you want to exclude.

Ksj14-kumar commented 1 year ago

Delete _node_modules _, package.lock.json file and .serverless . after that reinstall all dependencies by npm install. and problem fixed.

sanjaySP commented 1 year ago

A colleague saw the EMFILE: too many files open error recently, and the error was only showing up on their computer. It took a while for us to figure it out what the issue was, but it ended up being a package glob issue.

We had manually listed exclude glob patterns in the serverless.yml like this:
package:
  individually: true
  exclude:
    - .git/**
    - node_modules/**
    - .venv/**
The colleague had created their virtual environment in a directory called venv instead of .venv, so serverless didn't know to skip their virtual environment when packaging the lambda. It was trying to read everything in their virtual environment which blew up the memory.

We updated the glob pattern to the following:
package:
  individually: true
  include:
    - '!*'
    - '!**'
    - src/*.py
This pattern prevents anything other than python files in the src dir from getting packaged. It turns out we got speed benefits with this new syntax as well, as the glob pattern is much faster when you use ! in an include rather than listing files in the exclude because the glob takes a while to find every matching file that you want to exclude.

It worked like a charm 🥇

Lucassarantes commented 8 months ago

For me I've just deleted the node_modules and runned npm install again, after that the deploy worked.

desmondchou commented 3 months ago

A colleague saw the EMFILE: too many files open error recently, and the error was only showing up on their computer. It took a while for us to figure it out what the issue was, but it ended up being a package glob issue. We had manually listed exclude glob patterns in the serverless.yml like this:
package:
  individually: true
  exclude:
    - .git/**
    - node_modules/**
    - .venv/**
The colleague had created their virtual environment in a directory called venv instead of .venv, so serverless didn't know to skip their virtual environment when packaging the lambda. It was trying to read everything in their virtual environment which blew up the memory. We updated the glob pattern to the following:
package:
  individually: true
  include:
    - '!*'
    - '!**'
    - src/*.py
This pattern prevents anything other than python files in the src dir from getting packaged. It turns out we got speed benefits with this new syntax as well, as the glob pattern is much faster when you use ! in an include rather than listing files in the exclude because the glob takes a while to find every matching file that you want to exclude.
It worked like a charm 🥇

This reminded me to check the exclude path list and fixed the issue. Thank you.

s-nagesh commented 5 days ago

I also had this issue ,then i remove all node_modules , package.json and package.lock.json after install dependencys and again install npm i serverless

Alan-Graton commented 19 hours ago

So i closed my project on VS code, and opened a terminal window in the project directory. I Ran sls deploy and everything worked fine. Could it possibly be a compatibility issue with VS code ?

I don't think so. I'm having issues while trying to deploy my project in github actions

serverless / serverless-python-requirements