search

awslabs / aws-sdk-kotlin

Multiplatform AWS SDK for Kotlin
Apache License 2.0
414 stars 49 forks source link

Unable to mock S3's presignGetObject call #1238

Open hugoncosta opened 8 months ago

hugoncosta commented 8 months ago

Describe the bug

I am unable to mock a call to the S3Client's presignGetObject function as part of a unit test.

Expected behavior

The test should run successfully.

Current behavior

The test fails with the following stacktrace

java.lang.IllegalArgumentException:  is not a valid inet host

    at aws.smithy.kotlin.runtime.net.HostKt.hostParseImpl(Host.kt:35)
    at aws.smithy.kotlin.runtime.net.HostKt.access$hostParseImpl(Host.kt:1)
    at aws.smithy.kotlin.runtime.net.Host$Companion.parse(Host.kt:14)
    at aws.smithy.kotlin.runtime.http.operation.OperationEndpointKt$setResolvedEndpoint$1.invoke(OperationEndpoint.kt:70)
    at aws.smithy.kotlin.runtime.http.operation.OperationEndpointKt$setResolvedEndpoint$1.invoke(OperationEndpoint.kt:66)
    at aws.smithy.kotlin.runtime.http.request.HttpRequestBuilderKt.url(HttpRequestBuilder.kt:71)
    at aws.smithy.kotlin.runtime.http.operation.OperationEndpointKt.setResolvedEndpoint(OperationEndpoint.kt:66)
    at aws.smithy.kotlin.runtime.auth.awssigning.PresignerKt.presignRequest(Presigner.kt:35)
    at aws.sdk.kotlin.services.s3.presigners.PresignersKt.presignGetObject(Presigners.kt:52)
    at aws.sdk.kotlin.services.s3.presigners.PresignersKt.presignGetObject$default(Presigners.kt:40)
    at aws.sdk.kotlin.services.s3.presigners.PresignersKt.presignGetObject-exY8QGI(Presigners.kt:30)
    at com.amazon.networkvalidator.s3.S3AccessorTest$getObjectPresigned$1$1.invokeSuspend(S3AccessorTest.kt:74)
    at com.amazon.networkvalidator.s3.S3AccessorTest$getObjectPresigned$1$1.invoke(S3AccessorTest.kt)
    at com.amazon.networkvalidator.s3.S3AccessorTest$getObjectPresigned$1$1.invoke(S3AccessorTest.kt)
    at io.mockk.impl.eval.RecordedBlockEvaluator$record$block$2$1.invokeSuspend(RecordedBlockEvaluator.kt:27)
    at io.mockk.impl.eval.RecordedBlockEvaluator$record$block$2$1.invoke(RecordedBlockEvaluator.kt)
    at io.mockk.impl.eval.RecordedBlockEvaluator$record$block$2$1.invoke(RecordedBlockEvaluator.kt)
    at io.mockk.InternalPlatformDsl$runCoroutine$1.invokeSuspend(InternalPlatformDsl.kt:23)
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
    at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
    at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:277)
    at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:95)
    at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:69)
    at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
    at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:48)
    at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
    at io.mockk.InternalPlatformDsl.runCoroutine(InternalPlatformDsl.kt:22)
    at io.mockk.impl.eval.RecordedBlockEvaluator$record$block$2.invoke(RecordedBlockEvaluator.kt:27)
    at io.mockk.impl.eval.RecordedBlockEvaluator$enhanceWithRethrow$1.invoke(RecordedBlockEvaluator.kt:76)
    at io.mockk.impl.recording.JvmAutoHinter.autoHint(JvmAutoHinter.kt:23)
    at io.mockk.impl.eval.RecordedBlockEvaluator.record(RecordedBlockEvaluator.kt:39)
    at io.mockk.impl.eval.EveryBlockEvaluator.every(EveryBlockEvaluator.kt:30)
    at io.mockk.MockKDsl.internalCoEvery(API.kt:100)
    at io.mockk.MockKKt.coEvery(MockK.kt:169)
    at com.amazon.networkvalidator.s3.S3AccessorTest$getObjectPresigned$1.invokeSuspend(S3AccessorTest.kt:74)
    at com.amazon.networkvalidator.s3.S3AccessorTest$getObjectPresigned$1.invoke(S3AccessorTest.kt)
    at com.amazon.networkvalidator.s3.S3AccessorTest$getObjectPresigned$1.invoke(S3AccessorTest.kt)
    at kotlinx.coroutines.test.TestBuildersKt__TestBuildersKt$runTest$2$1$1.invokeSuspend(TestBuilders.kt:316)
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
    at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
    at kotlinx.coroutines.test.TestDispatcher.processEvent$kotlinx_coroutines_test(TestDispatcher.kt:24)
    at kotlinx.coroutines.test.TestCoroutineScheduler.tryRunNextTaskUnless$kotlinx_coroutines_test(TestCoroutineScheduler.kt:99)
    at kotlinx.coroutines.test.TestBuildersKt__TestBuildersKt$runTest$2$1$workRunner$1.invokeSuspend(TestBuilders.kt:322)
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
    at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
    at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:277)
    at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:95)
    at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:69)
    at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
    at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:48)
    at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
    at kotlinx.coroutines.test.TestBuildersJvmKt.createTestResult(TestBuildersJvm.kt:10)
    at kotlinx.coroutines.test.TestBuildersKt__TestBuildersKt.runTest-8Mi8wO0(TestBuilders.kt:310)
    at kotlinx.coroutines.test.TestBuildersKt.runTest-8Mi8wO0(Unknown Source)
    at kotlinx.coroutines.test.TestBuildersKt__TestBuildersKt.runTest-8Mi8wO0(TestBuilders.kt:168)
    at kotlinx.coroutines.test.TestBuildersKt.runTest-8Mi8wO0(Unknown Source)
    at kotlinx.coroutines.test.TestBuildersKt__TestBuildersKt.runTest-8Mi8wO0$default(TestBuilders.kt:160)
    at kotlinx.coroutines.test.TestBuildersKt.runTest-8Mi8wO0$default(Unknown Source)
    at com.amazon.networkvalidator.s3.S3AccessorTest.getObjectPresigned(S3AccessorTest.kt:73)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
    at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
    at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
    at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
    at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
    at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
    at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
    at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
    at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
    at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
    at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
    at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
    at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
    at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
    at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
    at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
    at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
    at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
    at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:57)
    at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
    at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
    at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
    at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
    at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)

Steps to Reproduce

@Disabled
@Test
fun getObjectPresigned_exception() = runTest {
    coEvery { client.presignGetObject(GET_OBJECT_REQUEST, any()) } throws S3Exception("Error")
    assertThrows<IllegalStateException> { s3Accessor.getObjectPresigned(S3BUCKET, KEY) }
}

the function it calls is the following

class S3Accessor(
    private val client: S3Client = S3Client { region = "us-east-1" }
) {

suspend fun getObjectPresigned(
        s3Bucket: String,
        keyName: String
    ): String {
        val request = GetObjectRequest {
            bucket = s3Bucket
            key = keyName
        }

        return try {
            client.presignGetObject(request, 6.hours).url.toString()
        } catch (e: S3Exception) {
            log.error(e)
            error("There was an issue creating the presigned link for $keyName from ${s3Bucket}: ${e.message}")
        }
    }    
}

Possible Solution

No response

Context

I am trying to create a unit test to ensure that if an error is thrown, it is thrown captured as I expect it. I am also facing the issue if I want to mock that the function returns some ByteStream.

AWS Kotlin SDK version used

1.0.67 (?)

Platform (JVM/JS/Native)

JVM (JDK17)

Operating System and version

macOS Sonoma 14.3.1

lauzadis commented 8 months ago

Thanks for the report! I am able to replicate the failure. The reason it throws this exception is because MockK tries to instantiate an empty Host for mocking which then causes a parse failure in the SDK.

...
2024-03-14 16:24:17 TRACE io.mockk.impl.instantiation.AbstractMockFactory:17 - Building proxy for Host hashcode=dde6f87
2024-03-14 16:24:17 TRACE io.mockk.proxy.jvm.ProxyMaker:17 - Class class aws.smithy.kotlin.runtime.net.Host is sealed, will use its subclass class aws.smithy.kotlin.runtime.net.Host$Domain to build proxy
2024-03-14 16:24:17 TRACE io.mockk.proxy.jvm.transformation.JvmInlineInstrumentation:17 - Retransforming classes aws.smithy.kotlin.runtime.net.Host$Domain, aws.smithy.kotlin.runtime.net.Host
2024-03-14 16:24:17 TRACE io.mockk.proxy.jvm.ProxyMaker:17 - Taking instance of class aws.smithy.kotlin.runtime.net.Host$Domain itself because it is final.
2024-03-14 16:24:17 TRACE io.mockk.proxy.jvm.ProxyMaker:17 - Instantiating proxy for class aws.smithy.kotlin.runtime.net.Host$Domain via instantiator
2024-03-14 16:24:17 TRACE io.mockk.proxy.jvm.ObjenesisInstantiator:17 - Creating new empty instance of class aws.smithy.kotlin.runtime.net.Host$Domain
hostname:

Note: hostname is an empty string at this point , so parsing can never succeed.

This seems to be an issue with MockK / your configuration of it rather than the SDK. I'm not too familiar with using MockK, is this enough extra information to help resolve your issue?

Also, do you need to use MockK? If not, you could consider writing an interceptor to throw this exception rather than have MockK do it.

hugoncosta commented 8 months ago

I didn't include any configuration, and for mocking purposes, I wouldn't expect any needed. Is it not possible to include a default hostname so that MockK doesn't fail? From all the s3 methods, this is the only one that fails.

As to why use MockK, this function fails as well

@Disabled
@Test
fun getObjectPresigned() = runTest {
    coEvery { client.presignGetObject(GET_OBJECT_REQUEST, 6.hours) } returns HTTP_REQUEST
    val output = s3Accessor.getObjectPresigned(S3BUCKET, KEY)
    assertEquals(Url.toString(), output)
}
ianbotsf commented 7 months ago

@hugoncosta I can't see how you're configuring your mocks but all of the presigner methods (e.g., presignGetObject) are extension methods on the service client interface and MockK requires an additional mock set up for extension methods. I suggest adding the following call to your test code before setting the expectation on the extension method:

mockStatic("aws.sdk.kotlin.services.s3.presigners.PresignersKt")

The SDK does not currently provide explicit support for any specific mocking framework. We strive to generally make our code as mockable as possible but each mocking framework will have its own set up, requirements, and likely some scenarios which cannot be mocked. We're always open to suggested improvements to our code which would improve mockability but we cannot promise 100% mockability via any specific framework.

I'm resolving this issue for now. Feel free to open a new issue if you encounter any SDK-specific bugs in your testing.

github-actions[bot] commented 7 months ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

ianbotsf commented 2 months ago

In light of https://github.com/awslabs/aws-sdk-kotlin/issues/1396 we should re-examine this request and either provide clearer instructions for how to mock presigning operations and/or improve the mockability of presigning code.