File table overflow on shared package with large number of dependencies

[chrisfischer]

Seen when running firelink deploy

[Error: ENFILE: file table overflow, open '.packages/api-util/node_modules/stream-events/readme.md'] {
  errno: -23,
  code: 'ENFILE',
  syscall: 'open',
  path: '.packages/api-util/node_modules/stream-events/readme.md'
}

[Stradivario]

@chrisfischer

Thank you for writing this ticket!

Hey man can you please post a version that you are running the problem ? What OS are you running ?

I think it may happen since the last version that i have published because i don't ignore some folders.

I will introduce a way in the next version to not copy packages and use them as they are and provide a way to ignore some files using .fireignore file.

Found some useful things related to this topic.

On MAC OS try the following

$ echo kern.maxfiles=65536 | sudo tee -a /etc/sysctl.conf
$ echo kern.maxfilesperproc=65536 | sudo tee -a /etc/sysctl.conf
$ sudo sysctl -w kern.maxfiles=65536
$ sudo sysctl -w kern.maxfilesperproc=65536
$ ulimit -n 65536 

Cheers!

[chrisfischer]

0.7.56

[Stradivario]

0.7.56

So it is really related to rsync deprecation https://github.com/rxdi/firelink/pull/23

Recently i have switched to native fs for compatability issues in different OS's

Please for now try to use 0.7.55 and tell me if the problem persists!

npm i -g @rxdi/firelink@0.7.55

Regards!

[Stradivario]

I think here https://github.com/rxdi/firelink/pull/23/files#diff-1f0c15eb37693c60d98a391cf8a98de708f96ab13f119be3d9caae1a3f7d5d25R30

it is collecting and trying to copy multiple files at once and it just overflows the limit of the file system itself.

Possible fix can be to execute them concurrent in batches by 100 files for example so the OS will remove the listeners from the file system. This may lead to a bit slower performance in terms of copying the packages but it should get rid of the error.

Since in my time is 12:30 AM i need to sleep and will investigate after i wake up in 6-7 hours.

Cheers!

[Stradivario]

This PR should resolve the bug introduced in this isssue also adds a really nice functionality where you can ignore certain files from being copied inside .packages folder

More details can be found at:

https://github.com/rxdi/firelink/issues/25

In about an hour new version will be deployed.

Cheers!

[Stradivario]

@chrisfischer Hello there!

Reverted logic to use rsync again for performance.

New version published

npm i -g @rxdi/firelink@0.7.57

[chrisfischer]

That seems to have fixed it!

I am running into a separate issue i think (and lmk if i should make a separate issue. tho my guess is i've just configured it wrong)

Build failed: npm ERR! cipm can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.
npm ERR! 
npm ERR! 
npm ERR! Invalid: lock file's @foo/api-util@file:../../api-util does not satisfy @foo/api-util@file:./.packages/api-util
npm ERR! 

snippet from my package.json:

  "dependencies": {
    "@foo/api-util": "file:../../api-util",
    "firebase-admin": "^10.0.0",
    "firebase-functions": "^3.16.0",
    "source-map-support": "^0.5.19"
  },
  "fireDependencies": {
    "@foo/api-util": "../../../packages/api-util"
  },

[Stradivario]

The actual idea for firelink is to not write manually local path like file:../../api-util

Instead strict version should be defined

"@foo/api-util": "^1.0.0"

The correct package json should be

  "dependencies": {
    "@foo/api-util": "^1.0.0",
    "firebase-admin": "^10.0.0",
    "firebase-functions": "^3.16.0",
    "source-map-support": "^0.5.19"
  },
  "fireDependencies": {
    "@foo/api-util": "../../../packages/api-util"
  },

When you hit deploy it will copy the packages inside .packages folder and deployment will install it locally and be replaced with file:../../../packages/api-util

Feel free to contact me if you need help resolving the issue and we can make a Google meet to share screen.

kristiqn.tachev@gmail.com

[Stradivario]

@chrisfischer closing this one if need some further help write a separate ticket or check example folder onto how firelink it is suppose to work.

Re-Open if needed!

https://github.com/rxdi/firelink/tree/master/example

Cheers!

[chrisfischer]

Apologies, it’s been a crazy week and haven’t had a chance to check that fix out yet. I really appreciate your help tho! I will follow up next week