TC encountered a problem when a large number of files (+1000) are pushed, in that the operating system only allows a certain number of files to be open at any given time.
This is a problem for the push command, because the filestreams are created asynchronously. Node will try to open all the filestreams at the same time, and receives an error from the OS.
Solution: I added the filequeue module from NPM, which allows you to create file streams asynchronously, while setting a limit to the number of open streams in a given context. If node tries to open more than this number of files, it will queue up those requests, and make them as previous requests are completed.
TC encountered a problem when a large number of files (+1000) are pushed, in that the operating system only allows a certain number of files to be open at any given time.
This is a problem for the
pushcommand, because the filestreams are created asynchronously. Node will try to open all the filestreams at the same time, and receives an error from the OS.Solution: I added the filequeue module from NPM, which allows you to create file streams asynchronously, while setting a limit to the number of open streams in a given context. If node tries to open more than this number of files, it will queue up those requests, and make them as previous requests are completed.