minio-java: how to upload file use presign url with user matadata? getPresignedObjectUrl? How to?

[MrFeng1993]

NOTE

minio-java version:8.5.4 after change minio from 2022-09-25T15:44:53Z to 2023-09-23T03:47:50Z when I use the url generate by getPresignedObjectUrl to upload a file an error occurred: My code is as follows:

• url generate

  public String getUploadUrl(String bucket, String ossFileName, String appId, Integer expiry, String flag, String preFileId) {
  
      Map<String, String> reqParams = new HashMap<>();
      reqParams.put(FsConstant.MINIO_BUCKET_NOTIFICATION_USERMETADATA_APP_ID_KEY, appId);
      reqParams.put(FsConstant.MINIO_BUCKET_NOTIFICATION_USERMETADATA_FLAG_KEY, flag);
      reqParams.put(FsConstant.MINIO_BUCKET_NOTIFICATION_USERMETADATA_File_ID_KEY, preFileId);
      try {
          return minioClient.getPresignedObjectUrl(GetPresignedObjectUrlArgs.builder()
                  .bucket(bucket)
                  .object(ossFileName)
                  .method(Method.PUT)
                  .extraQueryParams(reqParams)
                  .expiry(expiry)
                  .build());
      } catch (Exception e) {
          log.error("minio gen upload url failed!", e);
          throw new FsException("minio gen upload url failed!" );
      }
  
  }

• URL http://192.168.60.190:9000/fenggn0926/fenggn0926.jpg?X-Amz-Meta-App=1706960953436901378&X-Amz-Meta-File=4028a8ee8ad63795018ad9b591810006&X-Amz-Meta-Flag=2c9eb08188521a600188521a60390000&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minioadmin%2F20230928%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230928T025429Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=cffd43c49be232de5eae49378f28c35f239a0eba09d334cb44f515de5f33ae13

Expected Behavior

Upload success

Current Behavior

http status = 400 code = AccessDenied Message = There were headers present in the request which were not signed user metadata in extraQueryParams

Possible Solution

How can X-Amz-SignedHeaders contain X-Amz-Meta-* I looked at a lot of issues and couldn't find an answer

Steps to Reproduce (for bugs)

1. 2. 3. 4.

Context

Regression

Your Environment

• Version used (minio --version):2023-09-23T03:47:50Z
• Server setup and configuration:
• Operating System and version (uname -a):Linux a31a36f11894 3.10.0-1160.el7.x86_64 minio/minio#1 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[jiuker]

We have test example like:

 public static void getPresignedPostFormData() throws Exception {
    String methodName = "getPresignedPostFormData()";
    if (!mintEnv) {
      System.out.println(methodName);
    }
    long startTime = System.currentTimeMillis();
    try {
      String objectName = getRandomName();

      PostPolicy policy = new PostPolicy(bucketName, ZonedDateTime.now().plusDays(7));
      policy.addEqualsCondition("key", objectName);
      policy.addEqualsCondition("content-type", "image/png");
      policy.addContentLengthRangeCondition(1 * MB, 4 * MB);
      Map<String, String> formData = client.getPresignedPostFormData(policy);

      MultipartBody.Builder multipartBuilder = new MultipartBody.Builder();
      multipartBuilder.setType(MultipartBody.FORM);
      for (Map.Entry<String, String> entry : formData.entrySet()) {
        multipartBuilder.addFormDataPart(entry.getKey(), entry.getValue());
      }
      multipartBuilder.addFormDataPart("key", objectName);
      multipartBuilder.addFormDataPart("Content-Type", "image/png");
      multipartBuilder.addFormDataPart(
          "file",
          objectName,
          RequestBody.create(readAllBytes(new ContentInputStream(1 * MB)), null));
      Request.Builder requestBuilder = new Request.Builder();
      String urlString =
          client.getPresignedObjectUrl(
              GetPresignedObjectUrlArgs.builder()
                  .method(Method.GET)
                  .bucket(bucketName)
                  .object("x")
                  .build());
      urlString = urlString.split("\\?")[0]; // Remove query parameters.
      // remove last two characters to get clean url string of bucket.
      urlString = urlString.substring(0, urlString.length() - 2);
      Request request = requestBuilder.url(urlString).post(multipartBuilder.build()).build();
      OkHttpClient transport = newHttpClient();
      Response response = transport.newCall(request).execute();
      Assert.assertNotNull("no response from server", response);
      try {
        if (!response.isSuccessful()) {
          String errorXml = response.body().string();
          throw new Exception(
              "failed to upload object. Response: " + response + ", Error: " + errorXml);
        }
      } finally {
        response.close();
      }
      client.removeObject(RemoveObjectArgs.builder().bucket(bucketName).object(objectName).build());
      mintSuccessLog(methodName, null, startTime);
    } catch (Exception e) {
      handleException(methodName, null, startTime, e);
    }
  }

[balamurugana]

The error from the server is There were headers present in the request which were not signed. This means you are passing additional headers to obtained URL. This is wrong way to use Presigned URL.

[MrFeng1993]

So the problem is still unresolved.In fact, the '2022-09-25T15:44:53Z' version can use the header in this way, but after merge 17737 can not be used in this way is not too reasonable? I just want to know how I can add user meta data to the getPresignedObjectUrl method after 17737 without the AccessDenied error.if This is wrong way to use Presigned URL then what is the right way? thank U very much!

[balamurugana]

According to S3 specification, there is no way to pass headers to getPresignedObjectUrl() and the API returns URL and the consumer of the URL has no idea what headers should be passed at the time of HTTP execution. If you are passing headers to presigned URL, the error is evident.

Use getPresignedPostFormData() API where you have more control

[MrFeng1993]

ok, I got it. that means I need changgetPresignedObjectUrl() to getPresignedPostFormData(). Thank you for your patience

[MrFeng1993]

According to the previous discussion, I still did not understand how to modify the code. Let's get back to the essence of the problem! So now I want to ask, if I want to get a presign url that carries user matadata, can I do it? How to do that? before RELEASE.2023-08-04T17-40-21Z it`s ez. but now I am confused.

[jiuker]

@MrFeng1993

     PostPolicy policy = new PostPolicy(bucketName, ZonedDateTime.now().plusDays(7));
      policy.addEqualsCondition("key", objectName);
      policy.addEqualsCondition("content-type", "image/png");
      policy.addContentLengthRangeCondition(1 * MB, 4 * MB);
      Map<String, String> formData = client.getPresignedPostFormData(policy);

https://github.com/minio/minio-java/issues/1493#issuecomment-1738615275 Check the example plz. You should set policy.addEqualsCondition("X-Amz-Meta-Flag", "Your Flag"); maybe.

[MrFeng1993]

Okay, I'll try. thank U

[MrFeng1993]

By the way, Doesn't getPresignedObjectUrl work for uploading files anymore?

[jiuker]

By the way, Doesn't getPresignedObjectUrl work for uploading files anymore?

Not really. It work more like AWS S3 doc. @MrFeng1993

[MrFeng1993]

Ok, so should I check the S3 documentation? Because all I have to do is return a url to the other person to upload the file, I don't really know what I'm getting back to them with getPresignedPostFormData(), and if the data structure changes too much, it adds to their workload. I hope he doesn't perceive my change for the best. @jiuker

[jiuker]

The address you return includes policy, which contains various values that you thought were correct before you uploaded it. minio will check to see if these values have been tampered with. So I'm not sure how complicated it will get, it will just make the upload more secure. @MrFeng1993