Multiple fails uploads from Network lost issues causing CredentialProviderError.noProvider error to be thrown

[YoavGro]

Describe the bug A clear and concise description of what the bug is. If you can include a link to redacted requests or responses, that would be very helpful as well!

To Reproduce Steps to reproduce the behavior:

1. have multiple uploads in a queue
2. start uploading
3. Turn off connection for a very low connection
4. Wait for 30 seconds
5. Turn connection on
6. All upload calls to multipartUpload( throws an error with CredentialProviderError.noProvider

Expected behavior

• Have a callback to refresh the credentials incase of timeout expiration
• Have an ability to test if the credentials are valid

Setup (please complete the following information):

• OS: 12.5.1
• Version of soto: 6.0.0 (Next major)
• Authentication mechanism [hard-coded credentials, IAM Instance Profile on EC2, etc]

Additional context

After app restart uploads continue regularly

[adam-fowler]

Hi, a couple of questions

• Which credential provider are you using?
• Are you turning off the connection after some parts have uploaded fine?

[YoavGro]

Hi

• we are using this call: let credentialProvider: CredentialProviderFactory = .cognitoIdentity(identityPoolId: , logins: nil, region: )
• Yes

[adam-fowler]

Where are you getting your identityPoolId from? Any chance this could have expired? Could you post the code that is doing the upload?

Also if you put a break point here https://github.com/soto-project/soto/blob/eda662e974ab718a1932c3e4f3f0d63d2024b67b/Sources/Soto/Extensions/CognitoIdentity/CognitoIdentity%2BCredentialProvider.swift#L54 do you hit it and then if so what is the contents of response when you do?

[YoavGro]

• The identity pool id we are getting from a config file at run time (don't think it could have expired)
• This is the code that us dong the upload

 public func uploadRecordingSegment(fileName: String, segmentLocalPath: String) async throws {
        let bucket = InjectedValues[\.enviromentService].selectedEnvironment.awsBucket
        let request = S3.CreateMultipartUploadRequest(bucket: bucket, key: fileName)
        _ = try await self._s3.multipartUpload(request, filename: segmentLocalPath)
    }

• Yeah I hit this throw and the error response on my catch is SotoCore.CredentialProviderError._CredentialProviderError.noProvider

[YoavGro]

I'm seeing this error, only after we are getting timeouts on multiple parts

[adam-fowler]

Yeah I hit this throw and the error response on my catch is SotoCore.CredentialProviderError._CredentialProviderError.noProvider

By response I meant the contents of the response variable in the same function that throws the error. It appears AWS is returning an empty response when I ask it for credentials from your identity.

[YoavGro]

Ohh got it, sorry guess I misunderstood that. Checked it now, and it looks like it is not throwing the error from this line

[adam-fowler]

Hmm not totally sure where that noProvider error is coming from. There is only two places which throw it and given the description of the bug I cannot see how either are reached. Here are some options to play with

• Do you have a global AWSClient you use for every transaction with AWS. This is preferable to creating a new one for every operation, especially if you are using a credential provider that runs network operations like the cognito ones.
• There are two ways to manage retries of network connections.
  • By default multipartUpload should throw a S3ErrorType.multipart.abortedUpload which can be used in resumeMultipartUpload to resume a multipart upload. See https://soto.codes/user-guides/s3-multipart-upload.html. Although if you are getting a noProvider error I imagine you won't be able to use the client as the credential provider attached to it won't work.
  • The second way to manage retries is when you are creating you AWSClient add your own retryPolicy. It defaults to using jitter with a maximum number of retries of 4. Maybe you could increase the number of retries so it is in effect infinite. eg

    let client = AWSClient(
    credentialProvider: credentialProvider,
    retryPolicy: .jitter(maxRetries: 100000),
    ...
    )

[adam-fowler]

Closing as I did not get a response