GCP InputStream in GCP actually readsAllBytes

[eahrold]

When fetching larger files, we are seeing throwIfFatal detected a jvm fatal exception, which is thrown and logged below: java.lang.OutOfMemoryError: Java heap space

It looks like what's happening is that the underlying method used in objectstorage, is actually just pulling all bytes from the Blob into memory, and not creating a usable stream.

https://github.com/micronaut-projects/micronaut-object-storage/blob/e0ec2d931569fdd9252c00e9147a7f2f22e63edb/object-storage-gcp/src/main/java/io/micronaut/objectstorage/googlecloud/GoogleCloudStorageEntry.java#L49-L51

The relevant stack.

        at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
    at com.google.cloud.storage.Retrying.run(Retrying.java:65)
    at com.google.cloud.storage.StorageImpl.run(StorageImpl.java:1527)
    at com.google.cloud.storage.StorageImpl.readAllBytes(StorageImpl.java:622)
    at com.google.cloud.storage.Blob.getContent(Blob.java:778)
    at io.micronaut.objectstorage.googlecloud.GoogleCloudStorageEntry.getInputStream(GoogleCloudStorageEntry.java:50)

What we've got under the hood is similar to this.

7.2. Download endpoint from https://guides.micronaut.io/latest/micronaut-object-storage-gcp-gradle-java.html

[eahrold]

There may be some autoclose foot-guns here, but I've implemented the equivalent of this, and seems to work for files of any size

    @NonNull
    @Override
     public InputStream getInputStream() {
          return Channels.newInputStream(blob.reader());
     }

[alvarosanchez]

@eahrold thank you very much! Would you be willing to send a PR so that you get credit for the fix?

[eahrold]

@alvarosanchez here you go, https://github.com/micronaut-projects/micronaut-object-storage/pull/467