You may have come across streaming if you have experience outside of Apps Script, for example in Node JS. In Apps Script, you’d typically ingest all the data from a data source (eg a file), possibly transform it, and then write the copy you’ve made somewhere else – for example from Drive to Drive or Drive to cloud storage.
This is what the download/upload/export/convert methods in the drv and gcs Superfetch plugins do by default. (tldr; Actually they use a technique called resumable upload which splits the upload into chunks – allowing it to bypass the Drive and UrlFetch payload size limits)
However, what happens if the file size is just too big for Apps Script to handle all in one go, or if the data is arriving in chunks from a network service? In platforms that support it, you’d use a streaming technique – chunks arrive as input and are immediately piped to the output – so you never need to worry about having the complete data in memory at any one time.
One way is to use Html Service to handle client side streaming – but that would mean bouncing the data down to your client, so we probably don’t want to do that. Another would be to step outside Apps Script and use cloud functions or cloud run (see Blistering fast file streaming between Drive and Cloud Storage using Cloud Run https://ramblings.mcpher.com/gcp/file-drive-and-cloud-storage/). But if we want to stick to plain Server side Apps Script – the Tank Plugin is a pseudo streamer for Apps Script to emulate that behavior.
You may have come across streaming if you have experience outside of Apps Script, for example in Node JS. In Apps Script, you’d typically ingest all the data from a data source (eg a file), possibly transform it, and then write the copy you’ve made somewhere else – for example from Drive to Drive or Drive to cloud storage.
This is what the download/upload/export/convert methods in the drv and gcs Superfetch plugins do by default. (tldr; Actually they use a technique called resumable upload which splits the upload into chunks – allowing it to bypass the Drive and UrlFetch payload size limits)
However, what happens if the file size is just too big for Apps Script to handle all in one go, or if the data is arriving in chunks from a network service? In platforms that support it, you’d use a streaming technique – chunks arrive as input and are immediately piped to the output – so you never need to worry about having the complete data in memory at any one time.
One way is to use Html Service to handle client side streaming – but that would mean bouncing the data down to your client, so we probably don’t want to do that. Another would be to step outside Apps Script and use cloud functions or cloud run (see Blistering fast file streaming between Drive and Cloud Storage using Cloud Run https://ramblings.mcpher.com/gcp/file-drive-and-cloud-storage/). But if we want to stick to plain Server side Apps Script – the Tank Plugin is a pseudo streamer for Apps Script to emulate that behavior.
https://ramblings.mcpher.com/apps-script/superfetch-proxy/plugins-tank/