Upload large binaries with lower memory usage

[djeikyb]

Has support for lower memory video uploads been considered? It would make the caller code a little more complex, in exchange for not having to spend — at most, the twitter maximum of — 512mb memory per concurrent upload. The interface changes below I think might be one way the change could go.

I imagine the caller code would need to provide an enumeration of the chunks. Those could be byte arrays, but accepting streams would allow greater flexibility. For example, copying a large object from cloud storage to twitter. Slices of the file could be read on demand (System.Collections.IEnumerator.MoveNext) and copied from the cloud response stream to the twitter request stream.

From 0d0daef4db7997b6f11456627da690384ea378dc Mon Sep 17 00:00:00 2001
From: Jacob <jace42 at gmail>
Date: Tue, 23 Feb 2021 18:55:30 -0800
Subject: [PATCH] WIP propose lower memory usage upload

---
 .../Public/Client/Requesters/IUploadRequester.cs    | 13 ++++++++++++-
 .../Parameters/Upload/UploadBinaryParameters.cs     | 11 ++++++++++-
 2 files changed, 22 insertions(+), 2 deletions(-)

diff --git a/src/Tweetinvi.Core/Public/Client/Requesters/IUploadRequester.cs b/src/Tweetinvi.Core/Public/Client/Requesters/IUploadRequester.cs
index 15398424..41d29ac6 100644
--- a/src/Tweetinvi.Core/Public/Client/Requesters/IUploadRequester.cs
+++ b/src/Tweetinvi.Core/Public/Client/Requesters/IUploadRequester.cs
@@ -10,13 +10,24 @@ namespace Tweetinvi.Client.Requesters
     public interface IUploadRequester
     {
         /// <summary>
-        /// Upload a binary in chunks and waits for the Twitter to have processed it
+        /// Upload a binary in chunks and waits for the Twitter to have processed it.
+        /// Suitable for smaller binaries.
+        /// The entire binary is loaded into memory for the duration of the upload.
         /// <para>INIT : https://dev.twitter.com/en/docs/media/upload-media/api-reference/post-media-upload-init</para>
         /// <para>APPEND : https://dev.twitter.com/en/docs/media/upload-media/api-reference/post-media-upload-append</para>
         /// <para>FINALIZE : https://dev.twitter.com/en/docs/media/upload-media/api-reference/post-media-upload-finalize</para>
         /// </summary>
         /// <returns>Uploaded media information</returns>
         Task<IChunkUploadResult> UploadBinaryAsync(IUploadParameters parameters);
+
+        /// <summary>
+        /// Upload a binary in chunks and waits for the Twitter to have processed it.
+        /// Suitable for larger binaries.
+        /// At most, per invocation, only one chunk of the binary is in memory at a time.
+        /// </summary>
+        /// <param name="parameters"></param>
+        /// <returns></returns>
+        Task<IChunkUploadResult> UploadBinaryAsync(IIncrementalUploadParameters parameters);

         /// <summary>
         /// Add metadata to an uploaded media
diff --git a/src/Tweetinvi.Core/Public/Parameters/Upload/UploadBinaryParameters.cs b/src/Tweetinvi.Core/Public/Parameters/Upload/UploadBinaryParameters.cs
index 7a99c016..e64049e7 100644
--- a/src/Tweetinvi.Core/Public/Parameters/Upload/UploadBinaryParameters.cs
+++ b/src/Tweetinvi.Core/Public/Parameters/Upload/UploadBinaryParameters.cs
@@ -1,4 +1,7 @@
-namespace Tweetinvi.Parameters
+using System.Collections.Generic;
+using System.IO;
+
+namespace Tweetinvi.Parameters
 {
     /// <summary>
     /// For more description visit : https://dev.twitter.com/rest/media/uploading-media
@@ -27,4 +30,10 @@

         public byte[] Binary { get; set; }
     }
+
+    public interface IIncrementalUploadParameters : IUploadOptionalParameters
+    {
+        int TotalSize { get; }
+        IEnumerable<Stream> Chunks { get; }
+    }
 }
\ No newline at end of file
-- 
2.30.1