demergent-labs / azle

A WebAssembly runtime for TypeScript and JavaScript on ICP
MIT License
209 stars 36 forks source link

Get rid of onBeforeExit handler for file uploader #1996

Open bdemann opened 3 months ago

bdemann commented 3 months ago

Jordan doesn't want to use the onBeforeExit handler. Instead he wants to use await Promise.all() to know when the process is completed.

Additional feature request, if one of the chunks fails to upload, try again N times

Additional feature request, if one of the chunks fails even after N attempts then it would be great if we could cancel uploading the subsequent chunks from that file

As our uploader stands right now it does not do any of the additional features, so it would be technically a step forward if they only thing we accomplished was to use Promises.all().

Things to consider

I'm sure this is possible but I was having a hard time wrapping my head around it. As it stands we need to await the throttle but not await the actual chunk upload. This is why onBeforeExit works very well. We can have the uploading function be awaited and within we can await the throttle and and not await the chunk upload. But returning a list of promises instead is a little more challenging to make sure we are firing off the chunks at a rate that the replica can handle. We will need to make sure we tests this on mainnet before finalizing the design since this throttling is not needed to the same extent on a local replica.

bdemann commented 3 months ago

WIP For the associated branch, there is (at the time of writing this) a couple of WIP commits, one of which has a little proof of concept for canceling uploaded called cancel.ts. It is supposed to simulate uploading a bunch of chunks and having one fail and seeing if we can cancel the rest