isaacs / node-graceful-fs

fs with incremental backoff on EMFILE
ISC License
1.27k stars 148 forks source link

Still maintained? Alternatives? #186

Closed mifi closed 11 months ago

mifi commented 4 years ago

As this module is the core dependency of a lot of other packages used by thousands (millions) of people who need a normalized fs behavior, it is (along with fs-extra which uses this module) the go-to module. However there are many unresolved issues regarding Windows oddities and many open issues and pull requests needing to be merged or declined. As it's such an important core module for the node ecosystem I find it important that it gets maintained.

So is this module still being maintained? I understand that the module needs to be careful with breaking alot of downstream packages, but we could at least release major semversions with changes that people can begin explicitly upgrading to and testing.

I maintain a few other packages and am happy to co-maintain this if needed. Alternatively does anyone know if there are any other packages similar to this one that is more actively maintained that new projects can use instead?

balupton commented 11 months ago

What exactly do you want it to do that it doesn't already do?

It also seems in newer node versions, graceful-fs may not be necessary due to their builtins having a maxRetries option: https://kapeli.com/dash_share?docset_file=NodeJS&docset_name=Node.js&path=nodejs/api/fs.html%23fsrmpath-options-callback&platform=nodejs&repo=Main&version=21.1.0

If you want an oldschool wrapper with some utility functions, see https://github.com/bevry/safefs

isaacs commented 11 months ago

Maintaining this library has not been a priority for me, no. I don't use it in my own code much, since I've found over the years that promise parallelism limits are a better approach to preventing enfile/emfile than this sort of queueing at the low level fs, and most of the other polyfills are no longer necessary.

I don't recommend any alternatives. I recommend refactoring so that you don't need emfile protections because you're not opening thousands of files at the same time.

hatton commented 11 months ago

@isaacs If I understand you correctly, you're saying that the problem you were solving is contention within your own program, and thus there are better ways to avoid that.

I use libraries like this is instead to deal with contention with other processes running on the system, such as cloud backup programs. Once you get to thousands of users, even a very rare instance of such a program holding on to your file for a fraction of a second is enough to create many bug reports.

isaacs commented 11 months ago

I use libraries like this is instead to deal with contention with other processes running on the system

That is not what this library does.

The file descriptor ulimit is per-process, not for the system or login as a whole. If some other process is running on your system that is using file descriptors, then that doesn't affect any other process (because they don't share file descriptors) and is not handled by this library.

hatton commented 11 months ago

Thanks for that