GCS Private GeoTIFFs

[justin-lawless]

Is anyone successfully adding an authenticated Google Cloud Storage GeoTiff store using the Cog plugin? After following the installation instruction here, we found this initial error

it.geosolutions.imageioimpl.plugins.cog.CogSourceSPIProvider.createRangeReaderInstance Unable to create a RangeReader of type it.geosolutions.imageioimpl.plugins.cog.GSRangeReader on uri

And after dumping the stack trace from that function, we realized many dependencies were also missing, so we've now added them. However, now when we attempt to initialize the store, the server becomes unresponsive until rebooted. Server busy, please wait

We've set the system wide environment variable GOOGLE_APPLICATION_CREDENTIALS to the file path of a service account key with permissions on the resource, and have replicated this behavior locally in docker.

Is authenticated Google Cloud storage using COG actually supported?

Geoserver version 2.23.0 Steps: Login > Stores > Add new Store > GeoTIFF > Cloud Optimized GeoTiff using authenticated URL as well as the gsutil URI

[samodadela]

@justin-lawless I was wondering if you could explain what were the missing dependencies that you are mentioning above - I think this is the last piece of my puzzle...

As for the "Server busy, please wait" check out these issues on JIRA. https://osgeo-org.atlassian.net/browse/GEOS-11034 and this one https://osgeo-org.atlassian.net/browse/GEOS-11028

I've set it.geosolutions.cog.default.header.length to 512000 bytes and got rid of the server busy issue. Also ImageMosaic granules load correctly.

This works for me on a local GeoServer built from sources, but does not work in my docker container running kartoza/geoserver + cog-plugin. So I though that if I load the correct dependencies things would start working.

[justin-lawless]

Thanks, @samodadela. Good to know.

We solved the "server busy" issue by preprocessing a subset of problematic COGs we received from a specific vender through rio-cogeo with the --web-optimized flag. This was sufficient for our limited use case, so we did not pursue it further.

Regarding dependencies, we noticed after dumping the full stack trace from this method that google-cloud-storage was not in the class path.

After checking the WAR and the COG nightly builds, not only is google-cloud-storage missing, but many other dependencies that it relies on as well.
Looking over the build logs on the same Jenkins instance, we saw that google-cloud-storage was built with the project as we'd expect and as happens locally, however it's never packaged into the nightly build repository, and therefore never makes it to the server when using those repositories.

We assumed this is an oversight and instead build the project from source in CI, copy all dependencies and imageio JARs to our own artifact repository, and install from there instead. Works as expected now.

[aaime]

Missing dependencies are a problem in the assembly of GeoServer, nothing to do with imageio-ext. Can someone open a dedicated ticket in the GeoServer Jira tracker?

About needing to reprocess the files, I'm wondering if there is some tool out there creating exceptionally small tiles, which would cause the header to become very large as a consequence (there is one offset per tile, if someone uses 32x32 tiles, for example, rather than 256x246, that would make the tiles offset section of the header 64 times bigger).

Or maybe the file was just not a COG to start with.... many possibilities there.

[samodadela]

@aaime Jira ticket regarding the missing dependencies: https://osgeo-org.atlassian.net/browse/GEOS-11037

[samodadela]

We make our tiles COG compliant using https://github.com/airbusgeo/cogger. It's several times faster then rio cogeo and there is no need to tune parameters - it just copies what the source has.

[justin-lawless]

We confirmed it was indeed a proper COG with the same library, rio-cogeo, but again, it was only an issue with a specific vender. The structure looked nearly identical to other known good COGs as well as the examples here

I wish I still had outputs and logs to share, but unfortunately this is as much as I can contribute.

[samodadela]

That's what gdalinfo reports for our granules.

Band 1 Block=256x256 Type=Float32, ColorInterp=Gray
  NoData Value=-1e+04
  Overviews: 4594x6246, 2297x3123, 1149x1562, 575x781, 288x391, 144x196, 72x98, 36x49  

[oddbjorn-larsen]

Thanks for the tips about missing dependencies, this is how I solved it (not familiar with java and maven), but I created a pom.xml with this code:

<?xml` version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.example</groupId>
    <artifactId>google-cloud-storage-dependencies</artifactId>
    <version>1.0-SNAPSHOT</version>

    <dependencies>
        <dependency>
            <groupId>com.google.cloud</groupId>
            <artifactId>google-cloud-storage</artifactId>
            <version>2.29.0</version>
        </dependency>
        <!-- Add other dependencies here -->
    </dependencies>
</project>

Installed maven and ran mvn dependency:copy-dependencies Copied all .jar files created by maven into geoserver's lib directory in Dockerfile # Copy all JAR files from the host's 'libs' directory to the container's GeoServer lib directory COPY libs/* /usr/local/tomcat/webapps/geoserver/WEB-INF/lib/

Im using kartoza docker image, so rebuilt the image and ran docker-compose up --build. Now the cog-plugin works with both public and private gs:// links. Also added GOOGLE_APPLICATION_CREDENTIALS to the build.

[aaime]

Closing, not an imageio-ext issue