In order to accommodate super large configurations (anything over 50MB?), it has been suggested to stream file input into the initial parsing function.
The intent is to parse each chunk as it is streamed in. This would prevent the entire file from being loaded into memory, then parsed.
This could also work into the multiple file input. If we get a ucs (or mini ucs from the vscode extension), then we could asyncronously crawl the archive and stream the files we need, one at a time.
This may also require that we input files after class instantiation.
In order to accommodate super large configurations (anything over 50MB?), it has been suggested to stream file input into the initial parsing function.
The intent is to parse each chunk as it is streamed in. This would prevent the entire file from being loaded into memory, then parsed.
This could also work into the multiple file input. If we get a ucs (or mini ucs from the vscode extension), then we could asyncronously crawl the archive and stream the files we need, one at a time.
This may also require that we input files after class instantiation.
https://www.freecodecamp.org/news/node-js-streams-everything-you-need-to-know-c9141306be93/ https://stackoverflow.com/questions/54981119/how-to-properly-walk-a-disk-directory-recursively-and-stream-all-the-paths-in-re https://qwtel.com/posts/software/async-generators-in-the-wild/?referrer=https%3A%2F%2Fstackoverflow.com%2Fquestions%2F5827612%2Fnode-js-fs-readdir-recursive-directory-search%2F45130990%2345130990 https://stackoverflow.com/questions/5827612/node-js-fs-readdir-recursive-directory-search https://stackoverflow.com/questions/25757293/how-to-stream-read-directory-in-node-js