Wuchsbezirke cannot be uploaded

[ridoo]

Reported by @ahmdthr while uploading Wuchsbezirke 2011.

Failed with message tuple.index(x): x not in tuple. Request: 75122b9b-2610-410e-bdaa-e638ba42b0e6. Complete error response:

{
  "exec_id": "75122b9b-2610-410e-bdaa-e638ba42b0e6",
  "status": "failed",
  "func_name": "rollback",
  "created": "2023-09-18T15:05:09.098812Z",
  "finished": "2023-09-18T15:05:20.039114Z",
  "last_updated": "2023-09-18T15:05:20.039121Z",
  "input_params": {
    "files": {
      "cpg_file": "/usr/src/geonode/geonode/uploaded/tmpjtvfx4cg/wgwb_wuchsbezirke_2011.cpg",
      "dbf_file": "/usr/src/geonode/geonode/uploaded/tmpjtvfx4cg/wgwb_wuchsbezirke_2011.dbf",
      "prj_file": "/usr/src/geonode/geonode/uploaded/tmpjtvfx4cg/wgwb_wuchsbezirke_2011.prj",
      "shp_file": "/usr/src/geonode/geonode/uploaded/tmpjtvfx4cg/wgwb_wuchsbezirke_2011.shp",
      "shx_file": "/usr/src/geonode/geonode/uploaded/tmpjtvfx4cg/wgwb_wuchsbezirke_2011.shx",
      "base_file": "/usr/src/geonode/geonode/uploaded/tmpjtvfx4cg/wgwb_wuchsbezirke_2011.shp"
    },
    "total_layers": 1,
    "store_spatial_file": true,
    "handler_module_path": "importer.handlers.shapefile.handler.ShapeFileHandler",
    "skip_existing_layers": false,
    "overwrite_existing_layer": false
  },
  "output_params": {},
  "step": "importer.rollback",
  "log": "tuple.index(x): x not in tuple. Request: 75122b9b-2610-410e-bdaa-e638ba42b0e6",
  "name": "wuchsbezirke_2011.zip",
  "action": "import",
  "source": "upload",
  "user": 1000,
  "geonode_resource": null,
  "link": "http://172.17.0.1/api/v2/executionrequest/75122b9b-2610-410e-bdaa-e638ba42b0e6"
}

Geoserver error log:

18 Sep 15:05:19 WARN   [org.geoserver] - Error decode epsg code: EPSG:6258
org.opengis.referencing.NoSuchIdentifierException: No transform for classification "Colombia Urban".
        at org.geotools.referencing.operation.DefaultMathTransformFactory.lambda$getProvider$1(DefaultMathTransformFactory.java:267)
        at java.base/java.util.Optional.orElseThrow(Optional.java:408)
        at org.geotools.referencing.operation.DefaultMathTransformFactory.getProvider(DefaultMathTransformFactory.java:264)
        at org.geotools.referencing.operation.DefaultMathTransformFactory.getDefaultParameters(DefaultMathTransformFactory.java:298)
        at org.geotools.referencing.factory.epsg.DirectEpsgFactory.createCoordinateOperation(DirectEpsgFactory.java:2845)

[cut for brevity]

EDIT: EPSG:6258 seems a bit weird for Wuchsbezirke dataset which is located in Germany. Note the difference between EPSG:6258 and EPSG:6258 (Datum)!

However, I am sure I downloaded the dataset directly from Thuenen Atlas instance last year. Will have to further investigate tomorrow.

[ridoo]

Ok, here are my findings so far:

• Upload using unzipped zip file fails because neither cpg nor cst extension is a supported co-file for shapefile import handler and the export was not default-encoded properly (here is a bit more history context on this). This error is not raised to the UI and I had to debug to find out.
• I added cpg to the allowed optional extensions of the shapefile handler and then I get the error from geoserver as noted above

The whole stacktrace:

19 Sep 07:37:48 WARN   [org.geoserver] - Error decode epsg code: EPSG:6258
org.opengis.referencing.NoSuchIdentifierException: No transform for classification "Colombia Urban".
    at org.geotools.referencing.operation.DefaultMathTransformFactory.lambda$getProvider$1(DefaultMathTransformFactory.java:267)
    at java.base/java.util.Optional.orElseThrow(Optional.java:408)

... [cut for brevity]

Caused by: org.opengis.referencing.NoSuchIdentifierException: No transform for classification "Colombia Urban".
    at org.geotools.referencing.operation.DefaultMathTransformFactory.lambda$getProvider$1(DefaultMathTransformFactory.java:267)
    at java.base/java.util.Optional.orElseThrow(Optional.java:408)

It seems that GeoServer gets confused with EPSG:6258 and EPSG:6258 (Datum)!

@gannebamm Could you please check the data if the mis-configuration comes directly from wrong metadata. I would have to understand this a bit better before filing a GeoServer issue

[gannebamm]

Some more info for this particular dataset. The PRJ states:

GEOGCS["ETRS89",
    DATUM["European Terrestrial Reference System 1989",
        SPHEROID["GRS 1980",
        6378137.0,
        298.257222101,
        AUTHORITY["EPSG","7019"]],
        TOWGS84[0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0],
        AUTHORITY["EPSG","6258"]],
    PRIMEM["Greenwich",
        0.0,
        AUTHORITY["EPSG","8901"]],
    UNIT["degree",
        0.017453292519943295],
    AXIS["Geodetic longitude",
        EAST],
    AXIS["Geodetic latitude",
        NORTH],
    AUTHORITY["EPSG","4258"]]

So, the dataset itself is in EPSG:4258, and the other EPSGS are for the DATUM and PRIMEM, respectively. If downloaded as gpkg, this information is read adequately in QGIS. For some reason, our dev instance is not accepting it, though.

After reprojecting it to EPSG:25832, it will load correctly: wuchsbezirke_new_crs_25832.zip

[ridoo]

I really wonder who came to the idea to point EPSG:6258 either to ETRS89 Datum and MAGNA-SIRGAS / Mitu urban grid.

However, to gain more knowledge on this, I asked the GeoServer community. Maybe I overlook something here.

[ridoo]

Regarding to cpg/cst-files I created an issue at the geonode-importer and offered a PR.

[ridoo]

After reprojecting it to EPSG:25832, it will load correctly: wuchsbezirke_new_crs_25832.zip

The reprojected zip gets uploaded without error. Thanks for testing.

[ridoo]

@gannebamm Andrea Aime had a look at the issue I created, but says that GeoServer never resolves Datums. For now, I have no idea why the wrong CRS is picked from. The prj-file looks ok to me. However, as re-projecting the dataset worked out, I suggest to close this issue for now. If there are any more updates on the GeoServer issue, I will ping into here again

[ridoo]

I tried to upload Wuchbezirke to GeoServer only (skipping GeoNode importer) and it worked out without any issue. Seems that I would have to dig deeper into the uploader code to isolate the actual problem.

@gannebamm @matthesrieke Feel free to re-open this issue if you would like me to investigate further on this.