googleapis / java-storage

Apache License 2.0
107 stars 77 forks source link

Support renaming files on Transfer Manager Upload #2638

Open sydney-munro opened 4 months ago

chalmagr commented 3 months ago

Having this same issue... adding some more details

Environment details

OS type and version: MacOS 14.6.1 / Windows 11 (issue happens on both) Java version: Temurin-11.0.19+7 (build 11.0.19+7) Version(s): 2.42.0

Steps to reproduce

  1. Implement the shared code in the documentation to upload multiple files (https://cloud.google.com/storage/docs/uploading-objects#storage-upload-object-java)
  2. Run the code trying to upload one directory (e.g. /Users/me/source/dir) to some location in a bucket: e.g. gs://my-bucket-name/destination (directory contents are file1.txt, file2.txt) (run the code with arguments

Code example

package org.example;

import com.google.cloud.storage.transfermanager.ParallelUploadConfig;
import com.google.cloud.storage.transfermanager.TransferManager;
import com.google.cloud.storage.transfermanager.TransferManagerConfig;
import com.google.cloud.storage.transfermanager.UploadResult;

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List;
import java.util.stream.Stream;

public class Main {
    public static void main(String[] args) throws IOException {
        uploadDirectoryContents(args[0], Paths.get(args[1]), args[2]);
    }

    private static void uploadDirectoryContents(String bucketName, Path sourceDirectory, String remotePath) throws IOException {
        TransferManager transferManager = TransferManagerConfig.newBuilder().build().getService();
        ParallelUploadConfig parallelUploadConfig = ParallelUploadConfig.newBuilder()
                .setBucketName(bucketName)
                .setPrefix(remotePath)
                .build();

        List<Path> filePaths = new ArrayList<>();
        try (Stream<Path> pathStream = Files.walk(sourceDirectory)) {
            pathStream.filter(Files::isRegularFile).forEach(filePaths::add);
        }
        List<UploadResult> results = transferManager.uploadFiles(filePaths, parallelUploadConfig).getUploadResults();

        for (UploadResult result : results) {
            System.out.println(
                    "Upload for "
                            + result.getInput().getName()
                            + " completed with status "
                            + result.getStatus());
        }
    }
}

Run the above example using arguments my-bucket-name /Users/me/source/dir destination

I would expect that the bucket contains the following files now

gs://my-bucket-name/destination/file1.txt gs://my-bucket-name/destination/file2.txt

However, the bucket contains the following (for a MacOS scenario)

gs://my-bucket-name/destination/Users/me/source/dir/file1.txt gs://my-bucket-name/destination/Users/me/source/dir/file2.txt

For Windows, the /Users/me/source/dir/file1.txt gets changed to C:\Users\me\source\dir\file1.txt