Previously, when importing Slack files into Discord, the file was fully downloaded from Slack into memory, then uploaded to Discord. This not only could use a ton of memory for large files, but because the download blocks the event loop, the program would stop being able to respond to the Discord heartbeat requests while the file was being downloaded.
This commit adds a shim that dynamically downloads chunks of data from Slack as the bot attempts to upload them to Discord. This means that only a small buffer of data has to be stored in memory at a time, as well as doesn't block the event loop for as long since it's only downloading a small chunk of data before allowing the event loop to continue.
Previously, when importing Slack files into Discord, the file was fully downloaded from Slack into memory, then uploaded to Discord. This not only could use a ton of memory for large files, but because the download blocks the event loop, the program would stop being able to respond to the Discord heartbeat requests while the file was being downloaded.
This commit adds a shim that dynamically downloads chunks of data from Slack as the bot attempts to upload them to Discord. This means that only a small buffer of data has to be stored in memory at a time, as well as doesn't block the event loop for as long since it's only downloading a small chunk of data before allowing the event loop to continue.
Fixes #15