I am working on a use case where I do not have all of the data for a multipart upload at the time I begin the part upload. I can't hold all of it in memory due to potential memory exhaustion nor can I first write it to disk because the S3 happens to be faster than local disk.
The sender calls a send() function to give me more data. The sender is driving this.
The callback option won't work because once I am in the callback loop I can't get out of it. S3_upload_part() will not return until all the data is sent. If it doesn't return, I am not available to receive new calls to send().
I have taken the approach to do the S3_upload_part() in a background thread. I am just wondering if there is a way to escape from the callback and continue as I get more data. (I was looking at CURL and couldn't figure out a way to send data without the callback either.)
This is more of a question not an issue. Apologies if this isn't the correct forum for it.
I am working on a use case where I do not have all of the data for a multipart upload at the time I begin the part upload. I can't hold all of it in memory due to potential memory exhaustion nor can I first write it to disk because the S3 happens to be faster than local disk.
The sender calls a send() function to give me more data. The sender is driving this.
The callback option won't work because once I am in the callback loop I can't get out of it. S3_upload_part() will not return until all the data is sent. If it doesn't return, I am not available to receive new calls to send().
I have taken the approach to do the S3_upload_part() in a background thread. I am just wondering if there is a way to escape from the callback and continue as I get more data. (I was looking at CURL and couldn't figure out a way to send data without the callback either.)
This is more of a question not an issue. Apologies if this isn't the correct forum for it.