Problem with Relation/Way polygon mapping during import (a.k.a. New Zealand is shrinking)

glenharris commented 7 years ago

Context

I am trying to import polygons corresponding to regions of interest. In general, it is working really well, so thanks for some great software.

Expected Behavior

I am importing admin_level=2 for New Zealand, and would expect to find a row with osm_id=556706 and name=New Zealand/Aotearoa https://www.openstreetmap.org/relation/556706

Actual Behavior

Instead, I get a row with osm_id=196244129, which corresponds to a (random?) way that is just one part of the overall New Zealand relation. https://www.openstreetmap.org/way/196244129

Steps to Reproduce

In the attached file, there are a few shell scripts that will reproduce nicely. issue.zip

Run setup.sh (will download the dataset and create a subset with just the admin_level=2 items). Note that this also runs 'osmium check-refs' to make sure the datasets are OK.
Run import.admin_2.sh. This will import the full new-zealand-latest dataset, but use an imposm3 filter to only bring in admin_level=2 items. It then does a query showing the unexpected result.
Run import.subset.sh. This will import the small new-zealand-latest.subset dataset, but uses the reference 'example-mapping.json' file. Again, a query is performed that shows the unexpected result.
Run import.all.sh. This will import the full new-zealand-latest dataset, using example-mapping.json. Again, a query is performed showing that the sole admin_level=2 item is not what was expected.

Context

Something is wrong here. It could be:

My understanding of OpenStreetMap data relationships
A fault in the source OpenStreetMap dataset
My incorrect usage of the imposm3 tool - in particular maybe I need to use relation_member, however I cannot get this to work correctly (and the current docs indicate this should not be necessary - https://imposm.org/docs/imposm3/latest/mapping.html - see the section on relation_types)
A bug/feature in imposm3 related to the handling of multipolygon import

Either way, if someone who knows a bit more about this can take a look and give me some suggestions/workarounds, that would be greatly appreciated.

Your Environment

PostgreSQL 9.5.6 on x86_64-pc-linux-gnu, compiled by gcc (Alpine 6.2.1) 6.2.1 20160822, 64-bit imposm3-0.4.0dev-20170519-3f00374-linux-x86-64.tar.gz

Regards,

Glen.

ImreSamu commented 7 years ago

@glenharris :

Something is wrong here. It could be:

A fault in the source OpenStreetMap dataset

see osmium check-refs -r result: Ways in relations missing: 140

$ osmium check-refs -r -i new-zealand-latest.subset.osm.pbf
w195461062 in r556706                                                         
w195461053 in r556706
w195461065 in r556706
w195461046 in r556706
w195461066 in r556706
w195461067 in r556706
w195461072 in r556706
w195461074 in r556706
w195461071 in r556706
w195461057 in r556706
w195461069 in r556706
w195461059 in r556706
w195461055 in r556706
w195461050 in r556706
w195461058 in r556706
w195461045 in r556706
w195461056 in r556706
w196197784 in r556706
w196197761 in r556706
w196197795 in r556706
w196197778 in r556706
w196197777 in r556706
w196197790 in r556706
w196197788 in r556706
w196197764 in r556706
w196197789 in r556706
w196197780 in r556706
w196197769 in r556706
w196197767 in r556706
w196197770 in r556706
w196197773 in r556706
w196197792 in r556706
w196197763 in r556706
w196197786 in r556706
w196197768 in r556706
w196197776 in r556706
w196197762 in r556706
w196197794 in r556706
w196197775 in r556706
w196197793 in r556706
w196197782 in r556706
w196197779 in r556706
w196197783 in r556706
w196197771 in r556706
w196197766 in r556706
w196197781 in r556706
w196197787 in r556706
w196197785 in r556706
w196197765 in r556706
w196197774 in r556706
w196197772 in r556706
w196215225 in r556706
w196215211 in r556706
w196215216 in r556706
w196215223 in r556706
w196215212 in r556706
w196215221 in r556706
w196215214 in r556706
w196215226 in r556706
w196215224 in r556706
w196215220 in r556706
w196215217 in r556706
w196215215 in r556706
w196215219 in r556706
w196215222 in r556706
w196215213 in r556706
w196215218 in r556706
w197085365 in r556706
w197085367 in r556706
w197085390 in r556706
w197085415 in r556706
w197085348 in r556706
w197085398 in r556706
w197085360 in r556706
w197085437 in r556706
w197085372 in r556706
w197085382 in r556706
w197085418 in r556706
w197085358 in r556706
w197085445 in r556706
w197085412 in r556706
w197085428 in r556706
w197085425 in r556706
w197085403 in r556706
w197085387 in r556706
w197085400 in r556706
w197085379 in r556706
w197085406 in r556706
w197085461 in r556706
w197085476 in r556706
w197085430 in r556706
w197085464 in r556706
w197085480 in r556706
w197085384 in r556706
w197085377 in r556706
w197085422 in r556706
w197085434 in r556706
w197085440 in r556706
w197085369 in r556706
w197085374 in r556706
w197085451 in r556706
w197085362 in r556706
w197085458 in r556706
w197085466 in r556706
w197085488 in r556706
w197085393 in r556706
w197085395 in r556706
w197085409 in r556706
w197085353 in r556706
w197085467 in r556706
w197085481 in r556706
w197085472 in r556706
w197091406 in r556706
w197091385 in r556706
w197091431 in r556706
w197091420 in r556706
w197091364 in r556706
w197091315 in r556706
w197091434 in r556706
w197091331 in r556706
w197091368 in r556706
w197091324 in r556706
w197091350 in r556706
w197091425 in r556706
w197091339 in r556706
w197091366 in r556706
w197091328 in r556706
w197091375 in r556706
w197091416 in r556706
w197091393 in r556706
w197091361 in r556706
w197091371 in r556706
w197091342 in r556706
w197091383 in r556706
w197091402 in r556706
w197091380 in r556706
w197091318 in r556706
w197091389 in r556706
w197091399 in r556706
w197091411 in r556706
[======================================================================] 100% 
There are 13792 nodes, 687 ways, and 1 relations in this file.
Nodes     in ways      missing: 0
Nodes     in relations missing: 0
Ways      in relations missing: 140
Relations in relations missing: 0

glenharris commented 7 years ago

Ah, thanks for that. For anyone browsing this thread, I ran 'osmium check-refs .osm.pbf', which gave no errors. However, it appears as though this command does not check the relations (just the ways), and so the correct command should have been 'osmium check-refs --check-relations .osm.pf' (or osmium check-refs -r .osm.pbf).

I am just about to go through the process again using the planet dataset, so I will update this issue once that has completed.

omniscale / imposm3