IllegalStateException "Unbalanced enter/exit" in okio AsyncTimeout

[madisp]

Describe the bug

Running lots of SQS client calls in parallel throws java.lang.IllegalStateException at AsyncTimeout.kt:58 when canceling the scope. Follow-up on my last few comments in #962 .

Expected Behavior

Exception not thrown

Current Behavior

Exception thrown :)

Steps to Reproduce

Start with an empty c7gd.xlarge instance running ami-0c5789c17ae99a2fa

Install java and docker:

sudo apt install default-jdk
sudo snap install docker
sudo groupadd docker
sudo usermod -aG docker $USER

(Reboot the ec2 instance at this point)

Clone the example project and run the tests:

git clone https://github.com/madisp/aws-kotlin-threads.git
cd aws-kotlin-threads
./gradlew test

Possible Solution

I think the issue is the Job#invokeOnCompletion call here - https://github.com/smithy-lang/smithy-kotlin/blob/5dddc09697d2dfc2c883f7a963392428e7338b8f/runtime/protocol/http-client-engines/http-client-engine-okhttp/jvm/src/aws/smithy/kotlin/runtime/http/engine/okhttp/OkHttpEngine.kt#L54. The completion handler will probably be called on the dispatcher thread of whoever is doing the cancel. ResponseBody.close will try to consume any unconsumed bytes and this can lead to multiple threads working with the same ResponseBody, correctly triggering a precondition check failure in AsyncTimeout.

Context

I'm using aws-sdk-kotlin in a backend project and was experimenting with the limits of SQS clients in a single process.

Your Environment

• Smithy Kotlin version used: 1.1.1
• Platform (JVM/JS/Native): JVM, Linux ARMv8
• Operating System and version: Ubuntu 22.04 LTS Jammy running on c7gd.xlarge.