hisabimbola / slack-history-export

A NPM module that allows slack users export their history
MIT License
287 stars 45 forks source link

Handle large history export #20

Open hisabimbola opened 8 years ago

hisabimbola commented 8 years ago

If the history being exported is very large, memory consumption might be very large because we storing all history in memory and then writing it to a file after all histories have been fetched. This can be very costly and if an error occurs when fetching messages, all messages will be lost.

Also, another risk for this is when #5 is implemented, memory usage can be very high.

A simple solution is to write the history to file in batch, once gotten from slack API and append subsequent messages to end of the file.

Another elegant solution is to open a NodeJS writable stream and close when done. Writing to file will be faster.