search

thecodrr / fdir

⚡ The fastest directory crawler & globbing library for NodeJS. Crawls 1m files in < 1s
https://thecodrr.github.io/fdir/
MIT License
1.51k stars 58 forks source link

Infinite RAM usage can exhaust system #100

Open paulmillr opened 4 months ago

paulmillr commented 4 months ago

RAM usage seems to be unlimited with the module. RAM requirements need to be clarified somewhere, because crashing apps is not ok.

thecodrr commented 4 months ago

Fdir just crawls the directories, and yes, if you have an insane amount of items it may end up OOM-ing your app. What's the suggested outcome here?

paulmillr commented 4 months ago

Readdirp have limits on ram usage due to its architecture. It can’t exceed X.

43081j commented 2 months ago

readdirp will just emit files/dirs, right? so it won't really use much memory, as it isn't building up a list of found paths

if the user turns the result into an array, they will hit the same issue fdir has.

you can set maxFiles possibly to limit the array size, so you won't hit any OOM issues

alternatively, maybe we could introduce an option to pass a callback which will be called each time we visit a path, and we skip building up the internal set of paths (which means you can't use group etc)

paulmillr commented 2 months ago

Well, stream APIs exist for a reason. Not always users want an array of 400K files. etc.