One of the known issues with keeping file system objects in memory is that that v8 heap memory grows towards a limit (~2GB max) with increased CPU usage as garbage collection becomes more expensive.
I've started looking into using different approaches of solving the problem, including using native libs (eg. hashset) and embed databases (sqlite, leveldb, lmdb, ejdb), and with each approach comes its own set of advantages and drawbacks.
One of the known issues with keeping file system objects in memory is that that v8 heap memory grows towards a limit (~2GB max) with increased CPU usage as garbage collection becomes more expensive.
I've started looking into using different approaches of solving the problem, including using native libs (eg. hashset) and embed databases (sqlite, leveldb, lmdb, ejdb), and with each approach comes its own set of advantages and drawbacks.
Co-incidentally, there's has been some interesting discussion on disk visualization tools on hacker news and led me to find out about other work with regards to dealing with performance and huge number of files. In particularly,
http://duc.zevv.nl/ https://github.com/jarun/nnn/wiki/performance-factors https://github.com/zevv/duc/issues/161