search

googleapis / java-storage

Apache License 2.0
106 stars 77 forks source link

Search: java.net.MalformedURLException when using Blob WriteChannel #2034

Closed bergwerf closed 1 year ago

bergwerf commented 1 year ago

It appears I am currently not able to implement streamed file uploading to Google Cloud Storage via the Java API.

OS type/version: Debian 11.

Java version: Java 17 OpenJDK.

Steps to reproduce

Run the Google Cloud Storage mock server provided by https://github.com/fsouza/fake-gcs-server

Run the program at https://github.com/bergwerf/cratedb_gcs/tree/main/gcs_demo

Code example

See https://github.com/bergwerf/cratedb_gcs/blob/3a0a1521f7d82286f2a0ba51f0939dff369c8b13/gcs_demo/src/main/java/nl/hbergwerf/example/Main.java#L83

final Blob blob = getBlob(blobName, true);
final WriteChannel channel = blob.writer();
final OutputStream outputStream = Channels.newOutputStream(channel);
final long transferred = content.transferTo(outputStream);
channel.close();
System.out.println("Transferred " + transferred + " bytes to `" + blobName + "`.");

Stack trace

Exception in thread "main" com.google.cloud.storage.StorageException: java.lang.IllegalArgumentException: java.net.MalformedURLException: no protocol: /upload/storage/v1/b/bucket/o?uploadType=resumable&name=world&upload_id=b25b68443dfb4363e11f0d58761fc6be
        at com.google.cloud.storage.StorageException.translateAndThrow(StorageException.java:81)
        at com.google.cloud.storage.BlobWriteChannel.flushBuffer(BlobWriteChannel.java:281)
        at com.google.cloud.BaseWriteChannel.close(BaseWriteChannel.java:151)
        at nl.hbergwerf.example.Main.writeViaStream(Main.java:88)
        at nl.hbergwerf.example.Main.main(Main.java:50)
Caused by: java.lang.IllegalArgumentException: java.net.MalformedURLException: no protocol: /upload/storage/v1/b/bucket/o?uploadType=resumable&name=world&upload_id=b25b68443dfb4363e11f0d58761fc6be
        at com.google.api.client.http.GenericUrl.parseURL(GenericUrl.java:679)
        at com.google.api.client.http.GenericUrl.<init>(GenericUrl.java:125)
        at com.google.api.client.http.GenericUrl.<init>(GenericUrl.java:108)
        at com.google.cloud.storage.spi.v1.HttpStorageRpc.writeWithResponse(HttpStorageRpc.java:847)
        at com.google.cloud.storage.BlobWriteChannel.transmitChunk(BlobWriteChannel.java:73)
        at com.google.cloud.storage.BlobWriteChannel.access$1400(BlobWriteChannel.java:34)
        at com.google.cloud.storage.BlobWriteChannel$1.run(BlobWriteChannel.java:258)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
        at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
        at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
        at com.google.cloud.storage.BlobWriteChannel.flushBuffer(BlobWriteChannel.java:189)
        ... 3 more
Caused by: java.net.MalformedURLException: no protocol: /upload/storage/v1/b/bucket/o?uploadType=resumable&name=world&upload_id=b25b68443dfb4363e11f0d58761fc6be
        at java.base/java.net.URL.<init>(URL.java:674)
        at java.base/java.net.URL.<init>(URL.java:569)
        at java.base/java.net.URL.<init>(URL.java:516)
        at com.google.api.client.http.GenericUrl.parseURL(GenericUrl.java:677)
        ... 14 more

External references such as API reference guides

Any additional information below

Related: https://github.com/googleapis/java-storage/issues/40

jjaakola-aiven commented 1 year ago

See fake-gcs-server issue, https://github.com/fsouza/fake-gcs-server/issues/1166

BenWhitehead commented 1 year ago

When a resumable upload session is started in Google Cloud Storage, the redirect Location returned for all subsequent operations is a fully qualified URL. It seems that "fake-gcs-server" is providing a relative Location instead of a fully qualified one.

The returned Location must specify the scheme and host http://localhost:8080/upload/storage/v1/b/bucket/o?uploadType=resumable&name=world&upload_id=b25b68443dfb4363e11f0d58761fc6be, once that is done the java library will PUT to the running fake-gcs-server.

bergwerf commented 1 year ago

An update was pushed to "fake-gcs-server" which resolved this issue for me.