search

nlextract / NLExtract

Convert (ETL) and visualize free Dutch geo-datasets.
https://nlextract.nl
GNU General Public License v3.0
149 stars 84 forks source link

ogrinfo and ogr2ogr fail for BGT when srsDimension missing #378

Closed justb4 closed 3 months ago

justb4 commented 3 months ago

Problem

When using the latest NLextract Docker Image based on latest GDAL, suddenly getting errors from GfsPreparationFilter: Did not get at least 3 values or invalid number of set of coordinates. This shows up in other BGT-cases like on GeoForum

Notes:

So the error is triggered from a recent GDAL, though only when invoked via Python bindings. When directly on commandline son e.g. bgt_begroeidterreindeel.gml it seems to work (check again!).

Trigger the Error

To trigger:

./build-docker.sh
docker run --rm  nlextract/nlextract:latest bgt/etl/etl.sh > bgt.log 2>&1

In bgt.log (ignore DB connection erros, not related). Output for bgt_begroeidterreindeel

2024-03-26 02:12:00,255 fileinput INFO Pop file record: {'file_path': 'test/input/otterlo.zip', 'name': 'bgt_begroeidterreindeel.gml'}
2024-03-26 02:12:00,275 subfeaturehandler INFO invoke() for imgeo:positie
2024-03-26 02:12:00,277 subfeaturehandler INFO skipping GML file for subfeat=imgeo:positie
2024-03-26 02:12:00,277 subfeaturehandler INFO invoke() for imgeo:nummeraanduidingreeks[imgeo:Nummeraanduidingreeks]
2024-03-26 02:12:00,278 subfeaturehandler INFO skipping GML file for subfeat=imgeo:nummeraanduidingreeks[imgeo:Nummeraanduidingreeks]
2024-03-26 02:12:00,279 gfspreparationfilter INFO start preparing GFS
2024-03-26 02:12:00,279 gfspreparationfilter INFO calling ogrinfo
ERROR 1: Did not get at least 3 values or invalid number of set of coordinates <gml:posList>181932.439 457529.443 181933.382 457457.423 181934.736 457454.018 181936.377 457452.418 181938.552 457451.884 182083.905 457453.566 182086.326 457454.346 182087.639 457456.561 182088.377 457459.187 182086.859 457660.582 182086.367 457665.710 182085.095 457667.884 182083.167 457668.705 182002.223 457677.156 181929.588 457677.115 181923.700 457675.023 181922.346 457672.807 181922.121 457669.525 181923.413 457557.669</gml:posList>
ERROR 1: Did not get at least 3 values or invalid number of set of coordinates <gml:posList>181923.413 457557.669 181922.121 457669.525 181922.346 457672.807 181923.700 457675.023 181929.588 457677.115 182002.223 457677.156 182083.167 457668.705 182085.095 457667.884 182086.367 457665.710 182086.859 457660.582 182088.377 457459.187 182087.639 457456.561 182086.326 457454.346 182083.905 457453.566 181938.552 457451.884 181936.377 457452.418 181934.736 457454.018 181933.382 457457.423 181932.439 457529.443</gml:posList>
ERROR 1: Did not get at least 3 values or invalid number of set of coordinates <gml:posList>181933.990 457533.847 181932.007 457529.934 181932.951 457459.082 181933.292 457457.101 181934.178 457454.883 181935.806 457453.225 181938.136 457452.718 181992.212 457453.257 182081.154 457453.993 182083.603 457454.364 182085.681 457455.313 182087.083 457457.146 182087.628 457459.477 182087.796 457461.839 182087.119 457553.528 182086.377 457661.108 182086.292 457663.115 182086.067 457665.088 182085.613 457666.900 182084.429 457668.446 182082.835 457669.154 182001.394 457677.471 181927.609 457676.616 181925.537 457676.373 181923.267 457675.344 181922.017 457673.197 181921.646 457670.210 181922.946 457558.078 181922.787 457553.667</gml:posList>
ERROR 1: Did not get at least 3 values or invalid number of set of coordinates <gml:posList>181922.787 457553.667 181922.946 457558.078 181921.646 457670.210 181922.017 457673.197 181923.267 457675.344 181925.537 457676.373 181927.609 457676.616 182001.394 457677.471 182082.835 457669.154 182084.429 457668.446 182085.613 457666.900 182086.067 457665.088 182086.292 457663.115 182086.377 457661.108 182087.119 457553.528 182087.796 457461.839 182087.628 457459.477 182087.083 457457.146 182085.681 457455.313 182083.603 457454.364 182081.154 457453.993 181992.212 457453.257 181938.136 457452.718 181935.806 457453.225 181934.178 457454.883 181933.292 457457.101 181932.951 457459.082 181932.007 457529.934 181933.990 457533.847</gml:posList>
2024-03-26 02:12:00,384 gfspreparationfilter INFO parsing ogrinfo output
2024-03-26 02:12:00,384 gfspreparationfilter INFO 
{b'PlantCover': 1168}

Analysis and Solution

The problem appears to stem from this line in gfspreparationfiler.py

This statement: ogrinfo_cmd = 'ogrinfo -ro -al -so %s' % gml_file

Somewhat hidden, there seems to be a GDAL configuration option for this case in CityGML: GML_SRS_DIMENSION_IF_MISSING.

When changing the above line by adding --config GML_SRS_DIMENSION_IF_MISSING 2

ogrinfo_cmd = 'ogrinfo --config GML_SRS_DIMENSION_IF_MISSING 2 -ro -al -so %s' % gml_file

then the error Did not get at least 3 values or invalid number of set of coordinates does NOT appear and the GfsPreparationFilter works as expected. Output:

2024-03-26 02:31:38,247 fileinput INFO Pop file record: {'file_path': 'test/input/otterlo.zip', 'name': 'bgt_begroeidterreindeel.gml'}
2024-03-26 02:31:38,280 subfeaturehandler INFO invoke() for imgeo:positie
2024-03-26 02:31:38,282 subfeaturehandler INFO skipping GML file for subfeat=imgeo:positie
2024-03-26 02:31:38,283 subfeaturehandler INFO invoke() for imgeo:nummeraanduidingreeks[imgeo:Nummeraanduidingreeks]
2024-03-26 02:31:38,285 subfeaturehandler INFO skipping GML file for subfeat=imgeo:nummeraanduidingreeks[imgeo:Nummeraanduidingreeks]
2024-03-26 02:31:38,285 gfspreparationfilter INFO start preparing GFS
2024-03-26 02:31:38,285 gfspreparationfilter INFO calling ogrinfo
2024-03-26 02:31:38,384 gfspreparationfilter INFO parsing ogrinfo output
2024-03-26 02:31:38,384 gfspreparationfilter INFO 
{b'PlantCover': 1168}

Note: the same goes for the ogr2ogr command in the BGT Stetl config

There we need to also add GML_SRS_DIMENSION_IF_MISSING: options = -append -gt 65536 --config GML_SRS_DIMENSION_IF_MISSING 2 {multi_opts}

Conclusion

Two changes needed:

justb4 commented 3 months ago

Solved.