Add support for multipart uploads and/or expose XMLMultiPartUploadHelper

[clintonb]

Is your feature request related to a problem? Please describe. I am creating an S3 proxy to GCS so that I can use a custom authentication and authorization scheme. The client, Litestream, uses multipart uploads. GCS only exposes support for this via XML. I would prefer not to create a GCS XML client from scratch since you've already done it via XMLMultiPartUploadHelper.

Describe the solution you'd like Provide a client for the XML API. I don't want TransferManager trying to do this for me. I need access to the raw API so that I can get the upload ID, and upload individual parts.

Describe alternatives you've considered

1. Switch to S3.

2. Create my own XML API client.

   Additional context N/A

[ddelgrosso1]

Hi @clintonb thanks for opening an issue. A bit of explanation for why we don't expose the XMLMultiPartUploadHelper, currently there is only one operation in the NodeJS client that utilizes XML and that was within transfer manager. There was some discussion of the JSON API potentially supporting multipart uploads in the future and we were hesitant to expose the helper publicly if it would be replaced (i.e. we didn't want a breaking change).

All of that said, let me have some discussions internally and see how we feel about exposing this class. In its current form do you think it would meet your needs or do you see the need for changes?

[clintonb]

@ddelgrosso1 XMLMultiPartUploadHelper ended up being more helpful as an example of how to make authenticated calls to the XML API than as a client. No, it doesn't need to be exposed. I think providing sample code for how to make authenticated calls would be sufficient.

Here is an example of initiating a multipart upload:

// TODO Change to whatever makes sense for your integration
const bucket = this.getBucket(bucketName);

const url = `https://${bucket.name}.${
  new URL(bucket.storage.apiEndpoint).hostname
}/${name}?uploads`;

// Proxy directly to/from the GCS XML API
try {
  const res = await bucket.storage.authClient.request({
    headers: {
     // TODO Add other headers, if needed, such as Content-Type
      'Content-Length': 0,
    },
    method: 'POST',
    url,
  });

  if (res.data && res.data.error) {
    throw res.data.error;
  }

  // TODO This is an Express response. Change to accommodate your server
  return response.set(res.headers).status(HttpStatus.OK).send(res.data);
} catch (err) {
  this.logger.error(
    { err, bucketName, name },
    'Failed to initiate multipart upload'
  );

  throw err;
}

[ddelgrosso1]

Thanks for letting us know @clintonb. I'm going to close this issue out. If there is anything else please feel free to reopen or file a new issue.