aajtodd
commented
9 months ago Thanks for the issue.
we run into frequent, unhandled networking exceptions when talking to just about any AWS service, and especially S3.
I'd be interested in understanding what types of exceptions you are seeing. Are they all the same as the one you included in this issue (e.g. java.net.SocketException: Broken pipe (Write failed)) or are there others? Do they differ by service?
To consider java.net.SocketException retryable.
The issue is that knowing if a network exception is retryable generically for any operation depends on whether the operation is idempotent. Take the example given, you hit something like a socket closed issue (perhaps the server closed it or there was some other network issue). What if this failure happened after the server modified state? How do we know it's safe to retry at that point?
The answer is we don't (unless the operation is marked as @idempotent). We don't special case idempotent operations yet, that would be a good addition/feature request.
It's possible we may be able to special case S3's retry policy to be more relaxed but again we'd need a good way to classify which networking errors are retryable.
In the meantime you can always customize the retry policy for the SDK to retry whatever kind of error you want. This doesn't require changing the RetryStrategy, just the policy for what errors are considered retryable.
e.g.
class CustomRetryPolicy : RetryPolicy<Any?> {
override fun evaluate(result: Result<Any?>): RetryDirective {
return when(result.exceptionOrNull()) {
is IOException -> RetryDirective.RetryError(RetryErrorType.Transient)
else -> AwsDefaultRetryPolicy.evaluate(result)
}
}
}
fun main(): Unit = runBlocking {
val s3 = S3Client.fromEnvironment {
retryPolicy = CustomRetryPolicy()
}
...Note: You'll need a dependency on aws.sdk.kotlin:aws-http:<sdk-version> to access AwsDefaultRetryPolicy
berlix
ianbotsf
madisp
Describe the bug
I am aware of the fix in #896. However, even using SDK versions that include the fix (AWS Kotlin SDK 0.32.0-beta), we run into frequent, unhandled networking exceptions when talking to just about any AWS service, and especially S3.
Expected Behavior
To retry the request, so that application code doesn't have to.
Current Behavior
The request is not retried, and the exception is propagated up to the application:
As a side note: we tried switching to the CRT client, but there we've also observed at least one instance of an exception that looks retryable:
Steps to Reproduce
We observe this mostly (but not only) in long-running tasks that make many concurrent requests to AWS services.
Possible Solution
To consider
java.net.SocketExceptionretryable.Your Environment