🐛 Bug Report: S3 device should retry request on 503 errors

[metadaddy]

👟 Reproduction steps

This is a transient issue with S3 and compatible cloud object storage services such as Backblaze B2. Such services occasionally return 503 Service Unavailable or even 500 Internal Server Error due to a transient fault. Clients should retry the request; doing so works in the majority of cases.

👍 Expected behavior

The device should retry the request, with exponential backoff. Parameters should be configurable, but a good starting point is to retry 3 times, after 1, 2 and 4 seconds.

👎 Actual Behavior

On receiving a 503 error, an exception is thrown:

Now another problem. Sometimes it just stops the upload and says this ```10.76aeb2c7.js:5 File upload error: AppwriteException: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Error>
    <Code>ServiceUnavailable</Code>
    <Message>Service temporarily unavailable, please try again.</Message>
</Error>

    at W.<anonymous> (https://vlsdroneimaging.com/_app/immutable/chunks/public.888089e6.js:1:14808)
    at Generator.next (<anonymous>)
    at d (https://vlsdroneimaging.com/_app/immutable/chunks/public.888089e6.js:1:8610)```

🎲 Utopia Storage Version

Version 0.5.x

💻 Operating system

Linux

🧱 Your Environment

No response

👀 Have you spent some time to check if this issue has been raised before?

• [X] I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

• [X] I have read the Code of Conduct

[metadaddy]

Full disclosure: I am Pat Patterson, Chief Technical Evangelist at Backblaze.

I filed this in response to a post in the Appwrite support channel: https://discord.com/channels/564160730845151244/1227800965277552672/1227855634049925181

I'll look at coding a fix for this. It's a tricky one to reproduce against a production service, but I can use a proxy that returns a 503 every so often.

[CannonPhelps]

I am able to get this to happen when uploading files that are 1.5GB or so.