Inconsistent behavior between S3AsyncClient with and without VertxSdkClient wrapper

[dennismohan]

Hi, I've noticed some strange behavior around S3AsyncClient when wrapping it with the reactiverse aws-sdk wrapper.

I'm currently running into an issue where trying to upload files where the s3 key contains a special character, in my scenario +, results in a signature mismatch expection. java.util.concurrent.CompletionException: software.amazon.awssdk.services.s3.model.S3Exception: The request signature we calculated does not match the signature you provided. Check your key and signing method.

I dug a bit deeper, and to summarize my findings:

• s3 does in-fact support these special characters
• removing the special character results in a success
• aws cli and aws web console allow this file to be uploaded to the expected key
• The synchronous S3 client works
• The async s3 client without the aws-sdk wrapper works

I was able to narrow down a JUnit 5 test scenario to outline the inconsistent results. The only thing you'll need to do to run it is update the bucket variable at the top with a bucket of the test.

import io.reactiverse.awssdk.VertxSdkClient;
import io.vertx.core.Future;
import io.vertx.core.Vertx;
import io.vertx.core.buffer.Buffer;
import io.vertx.junit5.VertxExtension;
import io.vertx.junit5.VertxTestContext;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3AsyncClient;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.model.PutObjectResponse;

import java.nio.file.Path;
import java.util.UUID;

@ExtendWith(VertxExtension.class)
public class ReactiverseS3AsyncClientBug
{
    private static final AwsCredentialsProvider provider = DefaultCredentialsProvider.create();
    private static final Region region = Region.US_EAST_2;
    private static final String bucket = "my-bucket";
    private static final Path filePath = Path.of("sample_file.txt");

    @BeforeAll
    static void setup(Vertx v, VertxTestContext ctx)
    {
        final String text = "Some text data to store to disk";
        v.fileSystem().writeFile(filePath.toString(), Buffer.buffer(text)).onComplete(ctx.succeedingThenComplete());
    }

    @Test
    void uploadStuff_withSpecialChar_Async_reactiverseWrapper(final Vertx v, final VertxTestContext ctx)
    {
        final S3AsyncClient s3AsyncClient =
                VertxSdkClient.withVertx(S3AsyncClient.builder().credentialsProvider(provider).region(region),
                                         v.getOrCreateContext()).build();

        saveFileToS3(s3AsyncClient).onComplete(ctx.succeedingThenComplete());
    }

    @Test
    void uploadStuff_withSpecialChar_Async(final Vertx v, final VertxTestContext ctx)
    {

        final S3AsyncClient s3AsyncClient =
                S3AsyncClient.builder().credentialsProvider(provider).region(region).build();

        saveFileToS3(s3AsyncClient).onComplete(ctx.succeedingThenComplete());
    }

    private Future<PutObjectResponse> saveFileToS3(final S3AsyncClient client)
    {
        final String s3Key = "specialCharTest/Test_Sync_With_Special_Char&$@=;:+_" + UUID.randomUUID() + "_.txt";

        final PutObjectRequest reqObj = PutObjectRequest.builder().bucket(bucket).key(s3Key).build();
        return Future.fromCompletionStage(client.putObject(reqObj, filePath));
    }
}

The S3AsyncClient built with the VertxSdkClient.withVertx fails, where as the default one does not.

I ran into a similar case a while back where I abused the executeBlocking function on vertx to run the synchronous client as a work around. That scenario had similar behavior when trying to run s3AsyncClient.listObjectsV2, where it was failing when using the client wrapped by VertxSdkClient however passing without the wrapper.

I have not dove too deep into the investigation. Without special characters the putObject functionality works fine. From what I've read so far, special characters need to be url encoded when doing a http request. Its possible the encoding part is not working as intended causing the signature mismatch.

Let me know if I'm simply doing something wrong here, or if I can provide additional details to reproduce.

Thanks!

[aesteve]

Thanks a lot for the reproducer, that will certainly help a lot here. I'll try to give it a look when possible. For sure there must be something missing with encoding as you're suggesting.

[dennismohan]

Hey @aesteve,

I did a bit of digging into this, though did not have the time to actually confirm/verify this is the cause.

Diving through aws it seems they have a very particular way of doing UriEncoding, and recommend to override/implement the encoding instead of relying on the default.

Info can be found here: https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html

Cliffnotes:

URI encode every byte. UriEncode() must enforce the following rules:

URI encode every byte except the unreserved characters: 'A'-'Z', 'a'-'z', '0'-'9', '-', '.', '_', and '~'.

The space character is a reserved character and must be encoded as "%20" (and not as "+").

Each URI encoded byte is formed by a '%' and the two-digit hexadecimal value of the byte.

Letters in the hexadecimal value must be uppercase, for example "%1A".

Encode the forward slash character, '/', everywhere except in the object key name. For example, if the object key name is photos/Jan/sample.jpg, the forward slash in the key name is not encoded.

Important The standard UriEncode functions provided by your development platform may not work because of differences in implementation and related ambiguity in the underlying RFCs. We recommend that you write your own custom UriEncode function to ensure that your encoding will work.

The following is an example UriEncode() function in Java.

public static String UriEncode(CharSequence input, boolean encodeSlash) {
          StringBuilder result = new StringBuilder();
          for (int i = 0; i < input.length(); i++) {
              char ch = input.charAt(i);
              if ((ch >= 'A' && ch <= 'Z') || (ch >= 'a' && ch <= 'z') || (ch >= '0' && ch <= '9') || ch == '_' || ch == '-' || ch == '~' || ch == '.') {
                  result.append(ch);
              } else if (ch == '/') {
                  result.append(encodeSlash ? "%2F" : ch);
              } else {
                  result.append(toHexUTF8(ch));
              }
          }
          return result.toString();
      }

I just wanted to add a bit more information to help potentially narrow the problem down, hope it helps.