Missing admin entries - Githubissues

Seba95 commented 7 years ago

Hi,

i am using this file to test imposm3 : http://download.geofabrik.de/africa/egypt-latest.osm.pbf with the example-mapping.json.

the example-mapping fills the table with polygons but osm2pgsql uses linestrings. Is it an error?
if i change the mapping file to use linestrings then i can see that a lot of items are missing:
- osm2pgsql admin items: gis=> select count() from planet_osm_roads WHERE boundary = 'administrative'; count 615 (this request is similar to admin--zoom layers request in the openstreetmap-carto style)

Imposm3 admin items: gis=> select count(*) from osm_admin; count 146

Could you confirm the issue?

Regards, Sébastien

ImreSamu commented 7 years ago

Hello Sébastien,

imho: you have to check: unique osm_id vs. unique osm_id

Because osm2pqsl is splitting long ways : see https://help.openstreetmap.org/questions/40888/osm2pgsql-import-splits-ways-in-table-planet_osm_line

imho : only the select count(*) is meaningless info ; Can you give a missing - real osm_id examples, with a links ? ( like this relation/3726175 )

and you have to check a relation and a member ids
like :

relation id ( 1 x relation id="3726175")
member id ( 5 x member type="way" )

<osm version="0.6" generator="CGImap 0.5.8 (15710 thorn-02.openstreetmap.org)" copyright="OpenStreetMap and contributors" attribution="http://www.openstreetmap.org/copyright" license="http://opendatacommons.org/licenses/odbl/1-0/">
<relation id="3726175" visible="true" version="5" changeset="41692003" timestamp="2016-08-25T15:57:02Z" user="Malvolapukulo" uid="4128818">
<member type="way" ref="228633464" role="outer"/>
<member type="way" ref="228633444" role="outer"/>
<member type="way" ref="228633442" role="outer"/>
<member type="way" ref="228633447" role="outer"/>
<member type="way" ref="228633451" role="outer"/>
<member type="node" ref="768032316" role="admin_centre"/>
<tag k="admin_level" v="4"/>
<tag k="boundary" v="administrative"/>
<tag k="ISO3166-2" v="EG-MN"/>
<tag k="name" v="محافظة الفيوم"/>
<tag k="name:de" v="Minya"/>
<tag k="name:en" v="Minya Governate"/>
<tag k="name:eo" v="Provinco Fajum"/>
<tag k="name:hu" v="el-Minja kormányzóság"/>
<tag k="name:pl" v="Muhafaza Fajum"/>
<tag k="name:ru" v="мухафаза Эль-Минья"/>
<tag k="type" v="boundary"/>
</relation>
</osm>

( disclaimer: I am only a power user )

Seba95 commented 7 years ago

Hi ImreSamu,

i really think that items are missing because the negative osm ids used to build the admin-*-zoom layers are missing. I only see positive osm ids in the table built with imposm3. In the osmcarto style, admin layers are built with this request: (SELECT way, admin_level FROM planet_osm_roads WHERE boundary = 'administrative'
AND admin_level IN ('0', '1', '2', '3', '4')
AND osm_id < 0 <====== NEGATIVE ID ORDER BY admin_level DESC) AS admin_low_zoom"

Regards, Sébastien

ImreSamu commented 7 years ago

imho: this is not so simple ...

my advice, choose a very simple test case ...

like: “admin_level=2 and boundary=administrative in Egypt” ( http://overpass-turbo.eu/s/k5U )
- overpass info:
- "Loaded – nodes: 1941, ways: 44, relations: 1"
- "Displayed – pois: 2, lines: 1, polygons: 1"
load the osm data with :
- osm2pgsql
- imposm3
- with polygon
- and with linestring mapping
check the contents of tables
- check the length of the linestrings ( ST_Length ; sum( ST_length ) )
- check the name key
check the visual differences with QGIS ( or with other tools .. )
if it is OK - check the next admin_levels
- ( “admin_level=4 and boundary=administrative in Egypt” )
- ( “admin_level=7 and boundary=administrative in Egypt” http://overpass-turbo.eu/s/k5V )
- ( “admin_level=8 and boundary=administrative in Egypt” )

for me sometimes helps in debugging - if I pre-filter the data ( avoiding noise )

imho:

I am using imposm3, in multiple projects and I don't know any similar problems ..
There are lot of differences inside the imposm3 and osm2pgsql

ImreSamu commented 7 years ago

check ..

I made a little test .. with /egypt-latest.osm.pbf ( I have added a wikipedia and a 'name:en'key to the admin mapping )

map=# select osm_id, name, name_en, wikipedia,  admin_level, type  from osm_admin order by admin_level asc ;
  osm_id   |              name               |          name_en           |        wikipedia         | admin_level |      type      
-----------+---------------------------------+----------------------------+--------------------------+-------------+----------------
 317308949 |                                 |                            |                          |           2 | administrative
  -3061757 | محافظة أسوان         | Aswan Governorate          |                          |           4 | administrative
  -3061846 | الإسكندرية            | Alexandria Governorate     |                          |           4 | administrative
  -3062185 | الغربية                  | Suez Governorate           |                          |           4 | administrative
  -3062184 | الإسماعيلية          | Ismailia Governorate       |                          |           4 | administrative
  -3061826 | مطروح                      | Matrouh Governorate        |                          |           4 | administrative
  -3061827 | الوادي الجديد       | New Valley Governorate     |                          |           4 | administrative
  -3726124 | محافظة الفيوم       | Faiyum Governorate         |                          |           4 | administrative
  -3726170 | ‏محافظة بنى سويف | Beni Suef Governorate      |                          |           4 | administrative
  -3824206 | محافظة الجيزة‎    | Giza Governorate           |                          |           4 | administrative
  -3824513 | ‏البحيرة‎            | Beheira Governorate        |                          |           4 | administrative
  -3824207 | محافظة المنوفية   | Monufia Governorate        |                          |           4 | administrative
  -4103336 | القاهرة                  | Cairo Governorate          |                          |           4 | administrative
  -4103404 | دمياط‎‎                | Damietta Governorate       |                          |           4 | administrative
  -4103405 | كفر الشيخ               | Kafr el-Sheikh Governorate |                          |           4 | administrative
  -4103403 | الدقهلية                | Dakahlia Governorate       |                          |           4 | administrative
  -4103337 | القليوبية              | Qalyubia Governorate       |                          |           4 | administrative
  -4103406 | ‏محافظة بورسعيد  | Port Said Governorate      |                          |           4 | administrative
  -4103407 | الشرقية‎‎            | Al Sharqia Governorate     |                          |           4 | administrative
  -3584607 | ‏محافظة الغربي    | Gharbiyya Governorate      |                          |           4 | administrative
  -3060793 | جنوب سيناء             | South Sinai Governorate    |                          |           4 | administrative
  -3726175 | محافظة الفيوم       | Minya Governate            |                          |           4 | administrative
  -3726184 | أسيوط                      | Asyut Governorate          |                          |           4 | administrative
  -3726186 | سوهاج‎                   | Sohag Governorate          |                          |           4 | administrative
  -3726189 | محافظة قنا             | Qena Governorate           |                          |           4 | administrative
  -3726211 | الأقصر / Luxor            | Luxor Governate            | de:Gouvernement al-Uqsur |           4 | administrative
  -3586570 | محافظة الغربية     |                            |                          |           7 | administrative
  -3585231 | مديرية الغربية     |                            |                          |           7 | administrative
  -3603506 | ‏طنطا                     | Tanta                      |                          |           8 | administrative
  -3608214 | برما                        | Birma                      |                          |           8 | administrative
  -3608407 | كفر شعبان               | Kafr Shaban                |                          |           9 | administrative
  -3587035 | ميت السودان           |                            |                          |           9 | administrative
  -3605484 | شوبر                        | Shobar                     |                          |           9 | administrative
  -3605393 | كفر عصام                 | Kafr Isam                  |                          |           9 | administrative
  -3588686 | بريك الحجر             | Bureik al Hagar            |                          |           9 | administrative
  -3578930 | سبرباي                    |                            |                          |           9 | administrative
  -4081799 | وادي حلفا               | Wadi Halfa Salient         |                          |             | administrative
(37 rows)

As I see the country border is missing :(

analyze

imho: As I see the Geofabric extract poly for Egypt is smaller than the real border (admine_level=2 ) https://gist.github.com/anonymous/b2de797a46446cd2be29bd388ec47935

So the extract only contains - part of the country border / part of Egypt osm data ; so this is a reason why not included the admin_level=2 in the impom3 type: polygon mappings .. :disappointed:

imho : We have one more problems.. There is a data problem here ..

ImreSamu commented 7 years ago

As I remember - If you want to compare the osm2pgsql with the imposm3 it is better to add a--multi-geometry (-G) flag to the osm2pgsql ;

see : https://github.com/openstreetmap/osm2pgsql/blob/master/docs/analysis.md

Seba95 commented 7 years ago

Hi ImreSamu,

thx for the info about multi-geometry it has indeed the same pb. Are there other kind of objects with that issue? It would be nice to have a non multi-geometry option in imposm3 to be able to render things like osm2pgsql even if the data is not perfect.

Regards, Sébastien

ImreSamu commented 7 years ago

What is your use case ?
Why it is important for you ?

imho:

If you want compare, than you have to modify the osm2pgsql line splitting : https://github.com/openstreetmap/osm2pgsql/search?utf8=%E2%9C%93&q=Split+long+ways+after+around+1+degree+or+100km And on the imposm3 side: you can use PostGIS post-processing - to split or merge lines ...

But the PostGIS post-processing always works ..

you want to split polygons ? ( you have to use: http://postgis.net/docs/ST_Subdivide.html )
you want to merge objects? ( PostGIS - ST_LineMerge , ST_Collect, ... )
- see http://postgis.net/docs/reference.html

omniscale / imposm3

Missing admin entries #126

check ..

analyze