search

bji / libs3

Other
154 stars 151 forks source link

Is there a way to send multipart upload data without using callback mechanism? #97

Closed JustinKyleJames closed 3 years ago

JustinKyleJames commented 4 years ago

I am working on a use case where I do not have all of the data for a multipart upload at the time I begin the part upload. I can't hold all of it in memory due to potential memory exhaustion nor can I first write it to disk because the S3 happens to be faster than local disk.

The sender calls a send() function to give me more data. The sender is driving this.

The callback option won't work because once I am in the callback loop I can't get out of it. S3_upload_part() will not return until all the data is sent. If it doesn't return, I am not available to receive new calls to send().

I have taken the approach to do the S3_upload_part() in a background thread. I am just wondering if there is a way to escape from the callback and continue as I get more data. (I was looking at CURL and couldn't figure out a way to send data without the callback either.)

This is more of a question not an issue. Apologies if this isn't the correct forum for it.

JustinKyleJames commented 3 years ago

I no longer need this so I am closing it.