Efficiency

[smilinazzo]

https://github.com/Ikshi/assignment/blob/cdde9f40d9217ba45bb7440288e2cf3fa4e71ae3/test/test.js#L9

readFileSync reads the file synchronously effectively halting the event loop for other tasks. It also reads the file in its entirety. And, we are doing it a number of times in each test for the same files.

How would you improve the efficiency here? What if the file was for 100M events? Or the events were quite large. Are there other considerations we need to take into account when testing this?

[Ikshi]

@smilinazzo I would consider the memory usage when reading large files to avoid any excessive consumption of memory. I would rather use fs.createReadStream instead of fs.readFileSync to process the data in chucks and not reading the entire file at once. Also, as we are reading the same file, I could cache the data in memory after reading it.