osmcoastline generates invalid ESRI shapes

[ka2ddo]

When running osmcoastline on the current planet-NNNNNN.osm.pbf files to produce water polygons, the resulting data has many invalid ESRI polygons of type zero (instead of the type 3 or 5 that seem to be valid). It doesn't matter whether I have osmcoastline generate the SQLite database and then convert the DB to ESRI shapefile with ogr2ogr, or have osmcoastline generate ESRI shapefiles directly. This results in large swaths of ocean not being "wet" (in obvious rectangular stripes) upon rendering the remaining valid polygons; such missing polygons appear in the Sulu Sea and the Bering Sea (among other places).

Platform is Intel CPU running Fedora Core 34 (64-bit version), latest git pull of osmcoastline and libosmium as of 2022-Nov-28, gdal 3.2.2-3. However, this problem has been apparent for the last year (may have been longer, but not previously logged by the application that subsequently reads the Shapefile).

[ka2ddo]

Here's the terminal output from the latest run. Note the warnings at the end.

[x@y osm-downloads]$ ~/git/osmcoastline/build/src/osmcoastline_filter -o planet-221121-coast.osm.pbf planet-221121.osm.pbf [x@y osm-downloads]$ osmcoastline -o coastline.db -f -g "ESRI Shapefile" -m 30000 -v --bbox-overlap 0 --output-polygons=water planet-221121-coast.osm.pbf [ 0:00] Started osmcoastline version 2.3.1 (v2.3.1-6-g5652a7b-changed) / libosmium version 2.18.0 [ 0:00] Using SRS 4326 for output. (Change with the --srs/s option.) [ 0:00] Writing to output database 'coastline.db'. (Was set with the --output-database/-o option.) [ 0:00] Removing database output file (if it exists) (because you told me to with --overwrite/-f). [ 0:00] Will create geometry index. (If you do not want an index use --no-index/-i.) [ 0:01] Reading from file 'planet-221121-coast.osm.pbf'. [ 0:01] Reading ways (1st pass through input file)... [ 0:14] There are 2 nodes where the coastline is not closed. [ 0:14] There are 722705 coastline rings (696749 from a single closed way and 25956 others). [ 0:14] Memory used: current: 3238 MBytes, peak: 3341 MBytes [ 0:14] Reading nodes (2nd pass through input file)... [ 1:08] Checking for missing locations... [ 1:08] All locations are there. [ 1:12] Memory used: current: 6740 MBytes, peak: 6740 MBytes [ 1:12] Check line segments for intersections and overlaps... [ 2:52] Trying to close Antarctica ring... [ 2:52] Closed Antarctica ring. [ 2:52] Close broken rings... (Use --close-distance/-c 0 if you do not want this.) [ 2:52] Closing if distance between nodes smaller than 1. (Set this with --close-distance/-c.) [ 2:52] Closed 0 rings. This left 0 nodes where the coastline could not be closed. [ 2:52] Not writing out rings. (Use option --output-rings/-r if you want the rings.) [ 2:52] Create polygons... [ 6:39] Fixing coastlines going the wrong way... [ 6:39] Turned 3 polygons around. [ 6:39] Not writing coastlines as lines (Use --output-lines/-l if you want this). [ 6:39] Checking for questionable input data... [ 6:40] Found 6 rings in input data. [ 6:40] Split polygons with more than 30000 points... (Use --max-points/-m to change this. Set to 0 not to split at all.) [ 6:40] Using overlap of 0 (Set this with --bbox-overlap/-b). Polygon split at depth 1 was not successful. Keeping un-split polygon. Polygon split at depth 3 was not successful. Keeping un-split polygon. [24:58] Checking and making polygons valid... [26:20] Writing out water polygons... [186:48] Memory used: current: 6740 MBytes, peak: 8788 MBytes [186:48] Committing database transactions... [186:48] All done. [186:48] Memory used: current: 6740 MBytes, peak: 8788 MBytes There were 1128 warnings. There were 0 errors. [x@y osm-downloads]$

[joto]

There was a bug that created some empty water polygons which might show up as invalid in your system. I have just fixed that in 4d96ad5a3324d3d99d31cb57f75e4c4643d0f691. But that doesn't explain your problems with ocean tiles missing. And I can not reproduce that here.

I don't know anything about ESRI polygon types. Maybe you can check the current master with the fix and see whether you still see those invalid Polygons. If yes, please check with SELECT count(*) FROM water_polygons WHERE NOT ST_IsValid(geometry); in the spatialite database created whether GEOS sees any invalid polygons.

[ka2ddo]

Thank you. I pulled and compiled your latest changes and are running them now. I'll let you know how it comes out, but osmcoastline still reported 3 backwards polygons and 6 questionable rings (it's not finished running yet).

[joto]

Osmcoastline reporting backwards polygons and all that is part of normal operation and not something to worry about. It detects bad OSM data and fixes it as far as possible. If the input data is broken beyond repair, it should stop and not generate broken data.

[ka2ddo]

Well, whatever your fixes were, they worked. I didn't get a single bad polygon out of the mix this time. I'll have to try it again specifying that osmcoastline generate the ESRI Shapefile output directly instead of going through the spatialite database and then the ogr2ogr command to see if that also produces clean output.

[ka2ddo]

I did have 169 empty polygons in the output, but none with bad types that the openmap Java library couldn't read like I was getting before.

[joto]

Were you testing this with the same input planet every time?

[ka2ddo]

Yes, the planet-221121.osm.pbf file as run through the osmcoastline-filter program.

[ka2ddo]

Note that I had seen the problem for many planet files before this one, but was trying to fix it in my application that reads the water_polygons ESRI shapefile. But I made sure to retest with your changes using the same input file so the numbers would be comparable.

[joto]

I tried with the planet file you tried with and using the same parameters as you and get all valid polygons when generating the Spatialite file as well as when generating shape files. Also I can not see any problems in the Barents Sea or the Sulu Sea.

I have no idea what software you are using or why it is complaining about bad data, but at this point I have to assume the problem is somewhere on your side.

[ka2ddo]

Yes, I do too, with the new code commit you provided (not with the older version).

[joto]

Okay, I am closing this issue here. If you have any further problem please open a new one.