How to obtain sample data from S3 and launch the ogc-example test locally

[RunBoo]

I'm new to geotrellis-server and currently attempting to launch the sample test for ogc-example.

I noticed that the sample data is located on S3, but I don't have an AWS S3 account to access this data. Can someone please send me a copy of the test data, such as the data from this link: gt+s3://azavea-datahub/catalog?layer=us-ned-tms-epsg3857&zoom=14&band_count=1 https://github.com/geotrellis/geotrellis-server/blob/2a1fab9eb967bad1e8c3d1257a10dfc32182380b/ogc-example/src/main/resources/application.conf#L277

My email address is runbono13@gmail.com.

And if I have the data locally, how should I configure the application.conf file?

[RunBoo]

I have currently switched the test data to COG, but I am still getting the same error as before.

I suspect it's not an issue with the sample data, but rather a problem with the underlying environment. I am running on the Windows operating system, and my gdal-warp-bindings.jar version is 1.1.1. The installed GDAL version is 3.0.4.

The error message I'm receiving is as follows: 09:51:51.740 [blaze-selector-0] WARN org.http4s.server.blaze.BlazeServerBuilder - HTTP/2 support requires TLS. Falling back to HTTP/1. 09:51:51.852 [raster-io-0] DEBUG geotrellis.server.ogc.Main - GetCapabilities: /?SERVICE=WMS&REQUEST=GetCapabilities 09:51:51.947 [raster-io-0] ERROR org.http4s.server.service-errors - Error servicing request: GET / from 127.0.0.1 geotrellis.raster.gdal.MalformedDataException: Unable to construct dataset dimensions. GDAL Error Code: 4 at geotrellis.raster.gdal.GDALDataset$.$anonfun$dimensions$1(GDALDataset.scala:160) at geotrellis.raster.gdal.GDALDataset$.$anonfun$dimensions$1$adapted(GDALDataset.scala:157) at geotrellis.raster.gdal.GDALDataset$.errorHandler$extension(GDALDataset.scala:422) at geotrellis.raster.gdal.GDALDataset$.dimensions$extension1(GDALDataset.scala:157) at geotrellis.raster.gdal.GDALDataset$.rasterExtent$extension1(GDALDataset.scala:197) at geotrellis.raster.gdal.GDALRasterSource.gridExtent$lzycompute(GDALRasterSource.scala:93) at geotrellis.raster.gdal.GDALRasterSource.gridExtent(GDALRasterSource.scala:93) at geotrellis.server.ogc.wms.CapabilitiesView$.$anonfun$modelAsLayer$2(CapabilitiesView.scala:277) at scala.collection.immutable.List.map(List.scala:293) at geotrellis.server.ogc.wms.CapabilitiesView$.$anonfun$modelAsLayer$1(CapabilitiesView.scala:265) at map @ geotrellis.server.ogc.wms.CapabilitiesView$.modelAsLayer(CapabilitiesView.scala:264) at mapN @ geotrellis.server.ogc.wms.CapabilitiesView$.modelAsLayer(CapabilitiesView.scala:291) at mapN @ geotrellis.server.ogc.wms.CapabilitiesView$.modelAsLayer(CapabilitiesView.scala:291) at map @ geotrellis.server.ogc.wms.CapabilitiesView.toXML(CapabilitiesView.scala:111) at flatMap @ geotrellis.server.ogc.wms.WmsView.$anonfun$responseFor$5(WmsView.scala:142) at delay @ io.chrisdavenport.log4cats.slf4j.internal.Slf4jLoggerInternal$Slf4jLogger.$anonfun$debug$4(Slf4jLoggerInternal.scala:68) at delay @ io.chrisdavenport.log4cats.slf4j.internal.Slf4jLoggerInternal$Slf4jLogger.isDebugEnabled(Slf4jLoggerInternal.scala:50) at ifM$extension @ io.chrisdavenport.log4cats.slf4j.internal.Slf4jLoggerInternal$Slf4jLogger.info(Slf4jLoggerInternal.scala:76) at >>$extension @ geotrellis.server.ogc.wms.WmsView.responseFor(WmsView.scala:141) at sequence @ org.http4s.HttpRoutes$.$anonfun$of$2(HttpRoutes.scala:79) at defer @ org.http4s.HttpRoutes$.$anonfun$of$1(HttpRoutes.scala:79) at $anonfun$combineK$1 @ org.http4s.syntax.KleisliResponseOps.$anonfun$orNotFound$1(KleisliSyntax.scala:49) at getOrElse @ org.http4s.syntax.KleisliResponseOps.$anonfun$orNotFound$1(KleisliSyntax.scala:49) at defer @ org.http4s.server.blaze.Http1ServerStage$$anon$2.run(Http1ServerStage.scala:200) at flatMap @ org.http4s.server.blaze.Http1ServerStage$$anon$2.run(Http1ServerStage.scala:202)

Is there anyone who can help take a look at this issue？

[pomadchin]

Hi! I am not sure what's the gt+s3://azavea-datahub/ aviability at this point, it's better to check with Azavea / E84 folks.

About the second comment: what is the application.conf configuration you have for this layer?

GDAL Error Code: 4 means smth very generic ~ could not read the file for some reason; could be path / format / contents of the file.

[RunBoo]

My configuration file is as follows： application.zip

I now believe that the main reason for GDAL Error Code:4 is an issue with the underlying GDAL environment, which is a common occurrence on the Windows platform. Do you have any compatible versions available? I am using the latest version of geotrellis-server. And I am also trying on Linux machine.

By the way, can tile data sliced through Geotrellis be published as WMTS or WMS services in geotrellis-server? Is changing the source in the configuration file the only way to do it, or are there other methods?

[pomadchin]

That's the supported GDAL matrix: https://github.com/geotrellis/gdal-warp-bindings#installation GDAL 3.0.4 is the compatible one. However, if no proper GDAL is found than a different kind of error will be thrown (smth related to linking)

Try without GDAL to factor GDAL out for now. (replace gdal+file:// => gtiff+file://)

The data stored as GeoTrellis layer can be served as well, see layer definition example

As is (example projects) the way to deal with the source is via configuration files only. However, you may use this repository as a repository of components / reference server implementation to implement smth that fits requirements better.

[RunBoo]

Thank you for your patient response.

I tried not to use GDAL and changed the source to gtiff+file:///E:/Geotrellis/raster_data/LC08_B7_Memphis_COG.tiff, but then I encountered the following error:

14:40:52.222 [raster-io-0] ERROR org.http4s.server.service-errors - Error servicing request: GET / from 127.0.0.1 java.nio.file.InvalidPathException: Illegal char <:> at index 2: /E:/Geotrellis/raster_data/LC08_B7_Memphis_COG.tiff at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182) at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153) at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77) at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94) at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255) at java.nio.file.Paths.get(Paths.java:84) at geotrellis.util.FileRangeReaderProvider.rangeReader(FileRangeReaderProvider.scala:46) at geotrellis.util.FileRangeReaderProvider.rangeReader(FileRangeReaderProvider.scala:23) at geotrellis.util.RangeReader$.apply(RangeReader.scala:63) at geotrellis.util.RangeReader$.apply(RangeReader.scala:65) at geotrellis.raster.geotiff.GeoTiffRasterSource.$anonfun$tiff$1(GeoTiffRasterSource.scala:38) at scala.Option.getOrElse(Option.scala:189) at geotrellis.raster.geotiff.GeoTiffRasterSource.tiff$lzycompute(GeoTiffRasterSource.scala:37) at geotrellis.raster.geotiff.GeoTiffRasterSource.tiff(GeoTiffRasterSource.scala:34) at geotrellis.raster.geotiff.GeoTiffRasterSource.gridExtent$lzycompute(GeoTiffRasterSource.scala:55) at geotrellis.raster.geotiff.GeoTiffRasterSource.gridExtent(GeoTiffRasterSource.scala:55)

I'm not sure if this is a specific bug in Windows or not.

[RunBoo]

The data stored as GeoTrellis layer can be served as well, see layer definition example

I also tried publishing the Geotrellis layer from local or HBase as a service, but it still throws the same error：Unable to construct dataset dimensions. GDAL Error Code: 4

My configuration is source = "file:///E:/Geotrellis/Tiles/attributes?layers=tiles&zoom=10&band_count=1"

And I can show you my local Geotrellis layers:

Is there something wrong?

[pomadchin]

Indeed Windows support was never our focus, that's likely an unhandled cases in GeoTrellis; there's been https://github.com/locationtech/geotrellis/pull/3507 that addressed a windows parsing case, it looks like it may help you. Any Windows PRs against GeoTrellis a very much welcome!

To use GeoTrellis layers the gt+ prefix should be used, like in the layer definition above:

• gtiff+ - Java Tiff Reader
• gdal+ GDAL Reader (not only TIFF, any file format)
• gt+ GeoTrellis layers

[RunBoo]

Thank you, I found that the latest release version 3.7.0 of GeoTrellis has not included the Windows portion of this code https://github.com/locationtech/geotrellis/pull/3507 yet. And I will try it on Linux.

I've tried to use gt+GeoTrellis layers, but I got this error: Unable to find RasterSource for gt+file:///E:/Geotrellis/Tiles/attributes?layers=tiles&zoom=10&band_count=1

So my question is whether the gt+GeoTrellis layers path refers to JSON files or those tile data, and my layers data is normal or not:

My configuration is source = "file:///E:/Geotrellis/Tiles/attributes?layers=tiles&zoom=10&band_count=1"

And I can show you my local Geotrellis layers

[pomadchin]

@RunBoo it should point to the directory with the catalog, not to the attributes folder; in your case gt+file:///E:/Geotrellis/Tiles/?layer=tiles&zoom=10&band_count=1 Also a typo layerS => layer; please refer to the examples conf files and to the source code.

Interesting error you got: https://github.com/locationtech/geotrellis/blob/133a2d076bdba4dff4a760a4bf785268924dbde4/raster/src/main/scala/geotrellis/raster/RasterSource.scala#L210

Means that no GeoTrellisRasterSourceProvider is available at runtime.

Double check the class path / if all of the dependencies are properly loaded, this portion of the code relies on SPI. In the worst case try manually checking if its there.

How do you start / build the server?

You may try using WSL if staying on Windows is important.

[RunBoo]

@pomadchin Thank you very much! Now I am able to publish COG data as a WMS service on my Linux machine and it can be browsed successfully!

And then I tried to change the source to Geotrellis tiles, which is source = "gt+file:////home/dell/runboo/raster_service/data/Tiles?layer=tiles&zoom=10&band_count=1", then I encountered this error: java.lang.Exception: Unable to construct EPSG code from tmerc-CS at geotrellis.server.ogc.wms.CapabilitiesView$RasterSourceMethods.$anonfun$toLayer$3(CapabilitiesView.scala:189) at scala.Option.getOrElse(Option.scala:189) at geotrellis.server.ogc.wms.CapabilitiesView$RasterSourceMethods.$anonfun$toLayer$1(CapabilitiesView.scala:189) at scala.collection.immutable.List.map(List.scala:297) at geotrellis.server.ogc.wms.CapabilitiesView$RasterSourceMethods.toLayer(CapabilitiesView.scala:186) at geotrellis.server.ogc.wms.CapabilitiesView$.$anonfun$modelAsLayer$6(CapabilitiesView.scala:289) at scala.collection.immutable.List.map(List.scala:293) at geotrellis.server.ogc.wms.CapabilitiesView$.$anonfun$modelAsLayer$1(CapabilitiesView.scala:289) at map @ geotrellis.server.ogc.wms.CapabilitiesView$.modelAsLayer(CapabilitiesView.scala:264) at mapN @ geotrellis.server.ogc.wms.CapabilitiesView$.modelAsLayer(CapabilitiesView.scala:291) at mapN @ geotrellis.server.ogc.wms.CapabilitiesView$.modelAsLayer(CapabilitiesView.scala:291) at map @ geotrellis.server.ogc.wms.CapabilitiesView.toXML(CapabilitiesView.scala:111) at flatMap @ geotrellis.server.ogc.wms.WmsView.$anonfun$responseFor$5(WmsView.scala:142)

Initially, I suspected that the issue was due to GDAL not having EPSG installed correctly. However, upon checking, GDAL was able to read the projection system of the data. Now, I suspect that there might be an issue with the code I used for slicing in Geotrellis. I am using Geotrellis version 3.6.1, and my core code is as follows:

    val inputRdd: RDD[(ProjectedExtent, MultibandTile)] = sc.hadoopMultibandGeoTiffRDD(inputPath)

    val layoutScheme: ZoomedLayoutScheme = ZoomedLayoutScheme(WebMercator, tileSize = 512)
    val layoutDefinition: LayoutDefinition = layoutScheme.levelForZoom(zoomLevel).layout
    val zoom: Int = layoutScheme.levelForZoom(zoomLevel).zoom

    val rasterMetaData: TileLayerMetadata[SpatialKey] =
      inputRdd.collectMetadata[SpatialKey](layoutDefinition)

    val tiled: RDD[(SpatialKey, MultibandTile)] =
      inputRdd
        .tileToLayout(rasterMetaData.cellType, rasterMetaData.layout, Bilinear)
        .repartition(100)

    val reprojected: MultibandTileLayerRDD[SpatialKey] =
      MultibandTileLayerRDD(tiled, rasterMetaData)

    val attributeStore = FileAttributeStore(outputPath)
    val writer = FileLayerWriter(attributeStore)

    // Pyramiding up the zoom levels, write our tiles out to the local file system.
    Pyramid.upLevels(reprojected, layoutScheme, zoom, Bilinear) { (rdd, z) =>
      val layerId = LayerId("tiles", z)
      // If the layer exists already, delete it out before writing
      if (attributeStore.layerExists(layerId)) {
        new FileLayerManager(attributeStore).delete(layerId)
      }
      writer.write(layerId, rdd, ZCurveKeyIndexMethod)
    }

Is it because the coordinate system is incorrect when I perform the slicing, or is the slicing method incorrect?

[pomadchin]

@RunBoo what's the proj4 string of the source data? I assume the issue is in the proj4 unsable to map it into epsg code to construct a proper wms layer: https://github.com/geotrellis/geotrellis-server/blob/main/ogc/src/main/scala/geotrellis/server/ogc/wms/CapabilitiesView.scala#L147

Two things can be improved: 1. get removal 2. proj4 ehancement potentially to understand that proj4 string epsg code in case its unsupported

This is nothing to do with GDAL, GDAL is used in case of a GDALRasterSource usage only.

[RunBoo]

The spatial coordinate of my source data is CGCS2000, of which the proj4 string is"+proj=tmerc +lat_0=0.0 +lon_0=114.0 +k=1.0 +x_0=3.85E7 +y_0=0.0 +a=6378137.0 +b=6356752.314140356 +units=m ". It seems that this proj4 string epsg code is not supported.

Now I've changed my source data to another data with WGS84 spatial coordinate, and its proj4 string is "+proj=longlat +datum=WGS84 +no_defs ". Then I got a new error: geotrellis.raster.GeoAttrsError: invalid cols: 0 at geotrellis.raster.GridExtent$mcI$sp.(GridExtent.scala:45) at geotrellis.raster.RasterExtent.(RasterExtent.scala:79) at geotrellis.raster.RasterExtent$.apply(RasterExtent.scala:158) at geotrellis.server.ogc.SimpleOgcLayer$.$anonfun$simpleOgcHasRasterExtents$3(OgcLayer.scala:109) at scala.collection.immutable.List.map(List.scala:297) at geotrellis.server.ogc.SimpleOgcLayer$.$anonfun$simpleOgcHasRasterExtents$2(OgcLayer.scala:108) at delay @ geotrellis.server.ogc.SimpleOgcLayer$.$anonfun$simpleOgcHasRasterExtents$1(OgcLayer.scala:107) at traverse @ geotrellis.server.LayerHistogram$.$anonfun$apply$1(LayerHistogram.scala:57) at map @ geotrellis.server.LayerHistogram$.$anonfun$apply$1(LayerHistogram.scala:58) at flatMap @ geotrellis.server.LayerHistogram$.$anonfun$apply$1(LayerHistogram.scala:55) at flatMap @ geotrellis.server.LayerHistogram$.apply(LayerHistogram.scala:54) at flatMap @ geotrellis.server.ogc.wms.GetMap.$anonfun$build$4(GetMap.scala:68)

The related code is: https://github.com/locationtech/geotrellis/blob/133a2d076bdba4dff4a760a4bf785268924dbde4/raster/src/main/scala/geotrellis/raster/GridExtent.scala#L45

Now my data and the underlying environment appear to be normal, and I'm using the method of slicing with Geotrellis before publishing the service. I'm puzzled by this new error.

[RunBoo]

@pomadchin Thank you very much for patiently helping me with each new question. I understand now what the issue is. I have been publishing Geotrellis tile data, and instead of publishing it as WMS, it should be published as WMTS.

Now I have successfully published WMTS, but currently I can only retrieve attribute information through GetCapabilities. When I try to load the WMTS in QGIS, I can see the layer information：

but the images cannot be loaded, and the program throws this error：

20:26:24.649 [raster-io-3] ERROR org.http4s.server.service-errors - Error servicing request: GET / from 127.0.0.1 scala.MatchError: GeoTrellisOgcSource(us-ned,US NED,gt+file:///E:/Geotrellis/WGS84Tiles?layer=tiles&zoom=10&band_count=1,Some(elevation-ramp),List(ColorRampStyle(elevation-ramp,Elevation Ramp,geotrellis.raster.render.ColorRamp@7e59dff8,Some(90),None,None,false,List()), ColorRampStyle(elevation-ramp-clamped,Elevation Ramp: 1000 - 3000m,geotrellis.raster.render.ColorRamp@6a703765,Some(90),Some(1000.0),Some(3000.0),true,List())),NearestNeighbor,AutoHigherResolution,Some(times),Default,Oldest) (of class geotrellis.server.ogc.GeoTrellisOgcSource) at geotrellis.server.ogc.wmts.WmtsModel.$anonfun$getLayer$3(WmtsModel.scala:53) at scala.collection.immutable.List.map(List.scala:293) at geotrellis.server.ogc.wmts.WmtsModel.$anonfun$getLayer$2(WmtsModel.scala:50) at map @ geotrellis.server.ogc.wmts.WmtsModel.$anonfun$getLayer$1(WmtsModel.scala:49) at traverseN @ geotrellis.server.ogc.wmts.WmtsModel.getLayer(WmtsModel.scala:48) at map @ geotrellis.server.ogc.wmts.WmtsModel.getLayer(WmtsModel.scala:68) at flatMap @ geotrellis.server.ogc.wmts.WmtsView.responseFor(WmtsView.scala:116)

My tiles data and configuration file is as follows: Tiles data and configuration.zip Could you please help me take a look and see if there are any issues?

[RunBoo]

And if I change the tiles data from HBase, like this: source = "gt+hbase:///znkgtz01:2181?master=X.X.X.X&attribute=attributes&layer=myTiles&zoom=10&band_count=1" I will get this error:

Exception in thread "raster-io-0" java.lang.NoClassDefFoundError: org/apache/log4j/Level at org.apache.hadoop.mapred.JobConf.(JobConf.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2332) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:95) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:79) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137) at org.apache.hadoop.security.Groups.(Groups.java:105) at org.apache.hadoop.security.Groups.(Groups.java:101) at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:449) at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:434) at org.apache.hadoop.hbase.security.UserProvider.(UserProvider.java:56) at org.apache.hadoop.hbase.AuthUtil.loginClient(AuthUtil.java:106) at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:128) at geotrellis.store.hbase.HBaseInstance.getConnection(HBaseInstance.scala:58) at geotrellis.store.hbase.HBaseInstance.withAdminDo(HBaseInstance.scala:84) at geotrellis.store.hbase.HBaseAttributeStore.(HBaseAttributeStore.scala:45)

Is this issue related to Hadoop data retrieving?

[pomadchin]

@RunBoo

• geotrellis.raster.GeoAttrsError: invalid cols: 0 - smth related to sampling; for some reason no image is produced for the given zoom level, projection, area on a map; could be a bug or data specifics. Hard to tell, I'd think of a too low zoom lvl.

• It looks like WMTS does not support GT Layer for some reason, this can be fixed, any PRs / improvements are welcome! https://github.com/geotrellis/geotrellis-server/blob/main/ogc/src/main/scala/geotrellis/server/ogc/wmts/WmtsModel.scala#L54-L63

• Exception in thread "raster-io-0" java.lang.NoClassDefFoundError: org/apache/log4j/Level related to the class path; no logger found; just add it as a dependency! Most likely it is related to loggers exclusions, related to the CVE-2021-44228

[RunBoo]

• It looks like WMTS does not support GT Layer for some reason, this can be fixed, any PRs / improvements are welcome! https://github.com/geotrellis/geotrellis-server/blob/main/ogc/src/main/scala/geotrellis/server/ogc/wmts/WmtsModel.scala#L54-L63

If WMTS doesn't support GT layer, use geotrellis-server, how can I publish GT layers data to service? Should I publish to WMS? But I really have sliced source data to tiles.

[RunBoo]

@pomadchin I'm planning to submit a PR, but realized that I don't have the necessary permission. Could you please add this for me? And here is the code I want to push:

   source match {
            case mas: MapAlgebraSource =>
              val (name, title, algebra, resampleMethod, overviewStrategy) =
                (mas.name, mas.title, mas.algebra, mas.resampleMethod, mas.overviewStrategy)

              val simpleLayers = mas.sources.mapValues { rs =>
                SimpleTiledOgcLayer(name, title, crs, layout, rs, style, resampleMethod, overviewStrategy)
              }
              MapAlgebraTiledOgcLayer(name, title, crs, layout, simpleLayers, algebra, style, resampleMethod, overviewStrategy)
            case ss: SimpleSource =>
              SimpleTiledOgcLayer(ss.name, ss.title, crs, layout, ss.source, style, ss.resampleMethod, ss.overviewStrategy)
            case gos: GeoTrellisOgcSource =>
              SimpleTiledOgcLayer(gos.name, gos.title, crs, layout, gos.source, style, gos.resampleMethod, gos.overviewStrategy)
          }

However, it appears that the situation is not as straightforward as I initially thought. I encountered another error:

geotrellis.vector.ExtentRangeError: Invalid Extent: xmin must be less than xmax (xmin=9783.93962050256, xmax=9783.939620502026) at geotrellis.vector.Extent.(Extent.scala:94) at geotrellis.vector.Extent.buffer(Extent.scala:244) at geotrellis.store.GeoTrellisResampleRasterSource.$anonfun$read$2(GeoTrellisResampleRasterSource.scala:105) at scala.Option.map(Option.scala:230) at geotrellis.store.GeoTrellisResampleRasterSource.read(GeoTrellisResampleRasterSource.scala:105) at geotrellis.layer.LayoutTileSource.$anonfun$read$1(LayoutTileSource.scala:80) at scala.Option.flatMap(Option.scala:271) at geotrellis.layer.LayoutTileSource.read(LayoutTileSource.scala:79) at geotrellis.layer.LayoutTileSource.read(LayoutTileSource.scala:61) java.lang.Exception: Unable to retrieve layer SimpleTiledOgcLayer(us-ned,My Tiles,EPSG:4326,LayoutDefinition(Extent(-2.0037508342789244E7, -2.0037508342789244E7, 2.0037508342789244E7, 2.0037508342789244E7),CellSize(19567.87924100512,19567.87924100512),4x4 tiles,2048x2048 pixels),GeoTrellisRasterSource(file:///E:/Geotrellis/WGS84Tiles, Layer(name = "tiles", zoom = 10)),Some(ColorRampStyle(elevation-ramp,Elevation Ramp,geotrellis.raster.render.ColorRamp@23da75d,Some(90),None,None,false,List())),NearestNeighbor,AutoHigherResolution) at XY of (2, 0) at geotrellis.server.ogc.SimpleTiledOgcLayer$.$anonfun$simpleTiledReification$5(TiledOgcLayer.scala:109) at scala.Option.getOrElse(Option.scala:189) at geotrellis.server.ogc.SimpleTiledOgcLayer$.$anonfun$simpleTiledReification$3(TiledOgcLayer.scala:109) at delay @ geotrellis.server.ogc.SimpleTiledOgcLayer$.$anonfun$simpleTiledReification$2(TiledOgcLayer.scala:98) at map @ geotrellis.server.LayerTms$.$anonfun$apply$10(LayerTms.scala:69) at parTraverse$extension @ geotrellis.server.LayerTms$.$anonfun$apply$9(LayerTms.scala:65) at parTraverse$extension @ geotrellis.server.LayerTms$.$anonfun$apply$9(LayerTms.scala:65) at parTraverse$extension @ geotrellis.server.LayerTms$.$anonfun$apply$9(LayerTms.scala:65) at map @ geotrellis.server.LayerTms$.$anonfun$apply$9(LayerTms.scala:70) at flatMap @ geotrellis.server.LayerTms$.$anonfun$apply$9(LayerTms.scala:65)

I have been debugging for a long time, but I still can't find the issue. Could you help me take a look? And the data I used is：

My tiles data and configuration file is as follows: Tiles data and configuration.zip

Thank you!

[RunBoo]

I'm planning to submit a PR, but realized that I don't have the necessary permission. Could you please add this for me? And here is the code I want to push:

After I added this code and tried another two data, now I cannot get any error anymore. But in the QGis software, I also cannot load images, the screen is totally white. One of my data is: Tiles.zip

So I suspect that if there is smth wrong with my code of slicing GT layers? In our chat logs you can find my core code.

[pomadchin]

@RunBoo the way contributions work on GitHub is by making a fork and then a PR based on a push into the forked repo, will be happy to review the PR.

The error is likely to be of the similar roots as the geotrellis.raster.GeoAttrsError: invalid cols: 0 that was mentioned above.

Most likely a too low (not high enough resolution) zoom level on viz. I'd recommend a higher zoom level (11-12) and WCS to work on the issue. Most likely some projection / data quality bug / feature.

If WMTS doesn't support GT layer, use geotrellis-server, how can I publish GT layers data to service? Should I publish to WMS? But I really have sliced source data to tiles.

It does not really matter much, wmts / wcs / wms are specs and behave as views to the original data source that can be of any shape (rasters, stac api + rasters, geotrellis layers, whatever custom file format is needed)

I suspect that if there is smth wrong with my code of slicing GT layers

Totally not clear, have you tried looking at the sliced chunk without gt server?

I have been debugging for a long time, but I still can't find the issue. Could you help me take a look?

I'll try to find time, but no guarantees on the timeline.

[RunBoo]

I have submitted the PR here: https://github.com/geotrellis/geotrellis-server/pull/395 You can try it when you have time to see if there are any issues.

Most likely a too low (not high enough resolution) zoom level on viz. I'd recommend a higher zoom level (11-12) and WCS to work on the issue. Most likely some projection / data quality bug / feature.

I already have the slicing level set to 12. Or are you suggesting that I should set this parameter("gt+file:///E:/Geotrellis/Tiles?layer=tiles&zoom=10&band_count=1") to a higher zoom level?

Last question, could you share a code snippet for slicing multi-band Geotiff data using Geotrellis 3.6.0 version? I haven't been able to find such code online either.

[RunBoo]

@pomadchin I'm sorry to bother you again, but I set the zoom level to 12, and nothing happened. There are no error messages now, but the map still cannot be loaded. So I really have no idea.

Allow me to summarize. I have a GeoTIFF image, I want to slice it using Geotrellis, and then publish it as a WMTS / WCS or WMS service using Geotrellis-server to visualization. Is this technically feasible? I don't think it should be a difficult process, but it actually has taken a long time.

[pomadchin]

yea it should be a straightforward process; try just printing out a layer, wondering if thats a data issue; GT layer is chunked as z/x/y.

[pomadchin]

@RunBoo what is the original source tiff?

The layer is definitely not correct; There is only one tile 12/6990506 of an 1 x 2 size; I converted it to png and tiff - the result aligns with the gt server results you got (nothing is displayed) (:

$ gdalinfo 6990506.tiff
Driver: GTiff/GeoTIFF
Files: 6990506.tiff
Size is 2, 1
PROJ.4 string is:
'+proj=longlat +datum=WGS84 +no_defs'

6990506.zip

[pomadchin]

@RunBoo can you share the original tiff so I can take a look at the tiling logic?

It is hardly possible that it is related to the GT versions, there were no changes done in [3.6; 3.7] around this logic.

[RunBoo]

Sure, the source Geotiff and layers data is: Testdata.zip

Maybe my code is based on 2.x version, I'm not very sure. But I'm clear that 3.x version is different from 2.x version.

[pomadchin]

I'd recommend you just to regenerate the layer, smth is off with the one you tried to visualize.

tiles-new.zip

[pomadchin]

The ingest code new style via RasterSources (I generated tiles-new.zip this way):

val paths = "file:///tmp/origin.tif" :: Nil
val layerName = "tiles"

implicit val sc: SparkContext = createSparkContext("IngestRasterSource", new SparkConf(true))
val targetCRS = LatLng
val method = Bilinear
val layoutScheme = ZoomedLayoutScheme(targetCRS, tileSize = 256)

val sourceRDD: RDD[RasterSource] =
sc.parallelize(paths, paths.length)
  .map(uri => RasterSource(uri).reproject(targetCRS, method = method): RasterSource)
  .cache()

val summary = RasterSummary.fromRDD(sourceRDD)
val LayoutLevel(zoom, layout) = summary.levelFor(layoutScheme)
val contextRDD = RasterSourceRDD.tiledLayerRDD(sourceRDD, layout, KeyExtractor.spatialKeyExtractor, rasterSummary = summary.some)

val attributeStore = FileAttributeStore("/tmp/tiles-new")
val writer = FileLayerWriter(attributeStore)

Pyramid.upLevels(contextRDD, layoutScheme, zoom, method) { (rdd, z) =>
  val layerId = LayerId(layerName, z)
  if (attributeStore.layerExists(layerId)) FileLayerDeleter(attributeStore).delete(layerId)
  writer.write(layerId, rdd, ZCurveKeyIndexMethod)
}

The ingest code (old style like in your examples), I generated tiles using this one as well, should match the result of the code above; visually its the same result, difference can be in some tiny details on edges:

val inputPath = "/tmp/origin.tif"
val layerName = "tiles"
implicit val sc: SparkContext = createSparkContext("Ingest", new SparkConf(true))
val targetCRS = LatLng
val method = Bilinear
val layoutScheme = ZoomedLayoutScheme(targetCRS, tileSize = 256)

val inputRdd: RDD[(ProjectedExtent, MultibandTile)] = sc.hadoopMultibandGeoTiffRDD(inputPath) // can be single

val (_, rasterMetaData) = CollectTileLayerMetadata.fromRDD(inputRdd, FloatingLayoutScheme(512))

val tiled: RDD[(SpatialKey, MultibandTile)] = inputRdd.tileToLayout(rasterMetaData.cellType, rasterMetaData.layout, Bilinear)

val (zoom, reprojected): (Int, RDD[(SpatialKey, MultibandTile)] with Metadata[TileLayerMetadata[SpatialKey]]) =
  MultibandTileLayerRDD(tiled, rasterMetaData)
    .reproject(targetCRS, layoutScheme, method)

val attributeStore = FileAttributeStore("/tmp/tiles-new-old")
val writer = FileLayerWriter(attributeStore)

Pyramid.upLevels(reprojected, layoutScheme, zoom, method) { (rdd, z) =>
  val layerId = LayerId(layerName, z)
  if (attributeStore.layerExists(layerId)) FileLayerDeleter(attributeStore).delete(layerId)
  writer.write(layerId, rdd, ZCurveKeyIndexMethod)
}

The createSparkContext function definition:

def createSparkContext(appName: String, sparkConf: SparkConf = createSparkConf): SparkContext = {
  sparkConf
    .setAppName(appName)
    .setIfMissing("spark.master", "local[*]")
    .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
    .set("spark.kryo.registrator", classOf[geotrellis.spark.store.kryo.KryoRegistrator].getName)

  new SparkContext(sparkConf)
}

You may use https://github.com/pomadchin/vlm-performance/tree/feature/gt-3.x/src/main/scala/geotrellis/contrib/performance as a source of inspiration for the ingest code.

[RunBoo]

@pomadchin Thank you very much, it's very kind of you. I'll try it.

[RunBoo]

@pomadchin Hi~ I tried the tile data and slicing code you provided, thanks a lot. I found that publishing as WCS works fine, but publishing as WMS or WMTS still doesn't work. There are still some data-related errors that prevent the image from loading. Our requirement is to smoothly browse the data on the web, so we hope to publish it as WMTS. Here is my configuration file for your data:

application.zip

I'd recommend you just to regenerate the layer, smth is off with the one you tried to visualize.

tiles-new.zip

Can you help me check if there is a problem with the tile-matrix settings or other conf? And just to confirm, the current version of gt-server is able to support publishing GT layer data as WMTS, right?

[pomadchin]

I think the problem is somewhere on the reprojection / source data side; It is a bit time consuming figuing our what's the actual issue there; WCS display most likely works for you by a lucky mistake / smth done with the projections.

[abcwuhang]

The ingest code new style via RasterSources (I generated tiles-new.zip this way):

val paths = "file:///tmp/origin.tif" :: Nil
val layerName = "tiles"

implicit val sc: SparkContext = createSparkContext("IngestRasterSource", new SparkConf(true))
val targetCRS = LatLng
val method = Bilinear
val layoutScheme = ZoomedLayoutScheme(targetCRS, tileSize = 256)

val sourceRDD: RDD[RasterSource] =
sc.parallelize(paths, paths.length)
  .map(uri => RasterSource(uri).reproject(targetCRS, method = method): RasterSource)
  .cache()

val summary = RasterSummary.fromRDD(sourceRDD)
val LayoutLevel(zoom, layout) = summary.levelFor(layoutScheme)
val contextRDD = RasterSourceRDD.tiledLayerRDD(sourceRDD, layout, KeyExtractor.spatialKeyExtractor, rasterSummary = summary.some)

val attributeStore = FileAttributeStore("/tmp/tiles-new")
val writer = FileLayerWriter(attributeStore)

Pyramid.upLevels(contextRDD, layoutScheme, zoom, method) { (rdd, z) =>
  val layerId = LayerId(layerName, z)
  if (attributeStore.layerExists(layerId)) FileLayerDeleter(attributeStore).delete(layerId)
  writer.write(layerId, rdd, ZCurveKeyIndexMethod)
}

The ingest code (old style like in your examples), I generated tiles using this one as well, should match the result of the code above; visually its the same result, difference can be in some tiny details on edges:

val inputPath = "/tmp/origin.tif"
val layerName = "tiles"
implicit val sc: SparkContext = createSparkContext("Ingest", new SparkConf(true))
val targetCRS = LatLng
val method = Bilinear
val layoutScheme = ZoomedLayoutScheme(targetCRS, tileSize = 256)

val inputRdd: RDD[(ProjectedExtent, MultibandTile)] = sc.hadoopMultibandGeoTiffRDD(inputPath) // can be single

val (_, rasterMetaData) = CollectTileLayerMetadata.fromRDD(inputRdd, FloatingLayoutScheme(512))

val tiled: RDD[(SpatialKey, MultibandTile)] = inputRdd.tileToLayout(rasterMetaData.cellType, rasterMetaData.layout, Bilinear)

val (zoom, reprojected): (Int, RDD[(SpatialKey, MultibandTile)] with Metadata[TileLayerMetadata[SpatialKey]]) =
  MultibandTileLayerRDD(tiled, rasterMetaData)
    .reproject(targetCRS, layoutScheme, method)

val attributeStore = FileAttributeStore("/tmp/tiles-new-old")
val writer = FileLayerWriter(attributeStore)

Pyramid.upLevels(reprojected, layoutScheme, zoom, method) { (rdd, z) =>
  val layerId = LayerId(layerName, z)
  if (attributeStore.layerExists(layerId)) FileLayerDeleter(attributeStore).delete(layerId)
  writer.write(layerId, rdd, ZCurveKeyIndexMethod)
}

The createSparkContext function definition:

def createSparkContext(appName: String, sparkConf: SparkConf = createSparkConf): SparkContext = {
  sparkConf
    .setAppName(appName)
    .setIfMissing("spark.master", "local[*]")
    .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
    .set("spark.kryo.registrator", classOf[geotrellis.spark.store.kryo.KryoRegistrator].getName)

  new SparkContext(sparkConf)
}

You may use https://github.com/pomadchin/vlm-performance/tree/feature/gt-3.x/src/main/scala/geotrellis/contrib/performance as a source of inspiration for the ingest code.

I wonder whether the resolution of the last layer of RasterSourceRDD can be specified, manually or relevant to the original tiff. For example, it is good if the last layer (or next to last layer) is the same as the original tiff. How can I use parameters to meet the demand?