search

omniscale / imposm3

Imposm imports OpenStreetMap data into PostGIS
http://imposm.org/docs/imposm3/latest/
Apache License 2.0
719 stars 157 forks source link

[writer] not found #100

Closed tobwen closed 8 years ago

tobwen commented 8 years ago

I'm getting about 20-30 warnings about [writer] not found.

How can I help you to fix this problem?

ImreSamu commented 8 years ago

my2c : probably some error in your mapping file, but I am not sure

suggestion:

ImreSamu commented 8 years ago

more questions:

tobwen commented 8 years ago

I'm on Debian Jessie stable with PostgreSQL 9.4, PostGIS 2.2, libgeosc 3.4.2-6. I've just realized, it's imposm3 0.1 only. I've used this to download it, seems like this won't get and build latest source:

go get github.com/omniscale/imposm3
go install github.com/omniscale/imposm3

The error appeared on importing with this mapping, which I didn't change: https://github.com/omniscale/imposm3/blob/master/test/route_relation_mapping.yml

Cache directory is fine (my 3rd SSD) and I will post a logfile later.

ImreSamu commented 8 years ago

thank you for the information!

strange error .. 

+ /go/src/github.com/omniscale/imposm3/imposm3 import -mapping ./test/route_relation_mapping.yml -read ./parser/pbf/monaco-20150428.osm.pbf -diff -write -optimize -overwritecache -deployproduction -connection postgis://localhost/imposm3dev
[Apr 26 23:09:43] [INFO] removing existing cache /tmp/imposm3
[Apr 26 23:09:43] [INFO] [reader] reading ./parser/pbf/monaco-20150428.osm.pbf with data till 2015-04-27 22:21:02 +0200 CEST
[Apr 26 23:09:43] [INFO] [     0] C:       0/s (17233) N:       0/s (931) W:       0/s (2398) R:      0/s (108)
[Apr 26 23:09:43] [INFO] Reading OSM data took: 412.675762ms
[Apr 26 23:09:45] [INFO] [    1s] C:       0/s ( 0.0%) N:       0/s (100.0%) W:   54100/s (100.0%) R:   1490/s (100.0%)
[Apr 26 23:09:45] [INFO] Writing OSM data took: 1.134386422s
[Apr 26 23:09:45] [INFO] [PostGIS] Creating generalized tables took: 237.215µs
[Apr 26 23:09:45] [INFO] [PostGIS] Creating OSM id index on osm_master_routes took: 37.29078ms
[Apr 26 23:09:45] [INFO] [PostGIS] Creating geometry index on osm_master_routes took: 7.487349ms
[Apr 26 23:09:45] [INFO] [PostGIS] Creating OSM id index on osm_routes took: 48.505409ms
[Apr 26 23:09:45] [INFO] [PostGIS] Creating OSM id index on osm_route_members took: 48.787496ms
[Apr 26 23:09:45] [INFO] [PostGIS] Creating geometry index on osm_route_members took: 28.896392ms
[Apr 26 23:09:45] [INFO] [PostGIS] Creating geometry indices took: 78.013882ms
[Apr 26 23:09:45] [INFO] Importing OSM data took: 1.213244215s
[Apr 26 23:09:45] [INFO] [PostGIS] Analysing osm_routes took: 15.891221ms
[Apr 26 23:09:45] [INFO] [PostGIS] Indexing osm_route_members on geohash took: 64.376139ms
[Apr 26 23:09:45] [INFO] [PostGIS] Indexing osm_master_routes on geohash took: 71.157501ms
[Apr 26 23:09:45] [INFO] [PostGIS] Clustering osm_master_routes on geohash took: 191.479352ms
[Apr 26 23:09:45] [INFO] [PostGIS] Clustering osm_route_members on geohash took: 221.443837ms
[Apr 26 23:09:45] [INFO] [PostGIS] Analysing osm_master_routes took: 29.67392ms
[Apr 26 23:09:45] [INFO] [PostGIS] Analysing osm_route_members took: 12.591437ms
[Apr 26 23:09:45] [INFO] [PostGIS] Clustering on geometry took: 298.853849ms
[Apr 26 23:09:45] [INFO] [PostGIS] Rotating osm_master_routes from import -> public -> backup
[Apr 26 23:09:45] [INFO] [PostGIS] backup of osm_master_routes, to backup
[Apr 26 23:09:45] [INFO] [PostGIS] Rotating osm_route_members from import -> public -> backup
[Apr 26 23:09:45] [INFO] [PostGIS] backup of osm_route_members, to backup
[Apr 26 23:09:45] [INFO] [PostGIS] Rotating osm_routes from import -> public -> backup
[Apr 26 23:09:45] [INFO] [PostGIS] backup of osm_routes, to backup
[Apr 26 23:09:45] [INFO] [PostGIS] Rotating tables took: 20.970619ms
[Apr 26 23:09:45] [INFO] Imposm took: 2.06303398s
ImreSamu commented 8 years ago

I have tried other input data .. and found something similar :) can you confirm ? @TobWen - your error was similar ?

[Apr 26 23:20:19] [INFO] Reading OSM data took: 26.03967484s [Apr 26 23:20:46] [WARN] [writer] not found [Apr 26 23:20:46] [WARN] [writer] not found [Apr 26 23:20:46] [WARN] [writer] not found

my full log: input data: ../hungary-160226.osm.pbf

+ /go/src/github.com/omniscale/imposm3/imposm3 import -mapping ./test/route_relation_mapping.yml -read ../hungary-160226.osm.pbf -diff -write -optimize -overwritecache -deployproduction -connection postgis://localhost/imposm3dev
[Apr 26 23:19:53] [INFO] removing existing cache /tmp/imposm3
[Apr 26 23:19:54] [INFO] [reader] reading ../hungary-160226.osm.pbf with data till 2016-02-26 21:14:02 +0100 CET
[Apr 26 23:20:19] [INFO] [   25s] C:  845000/s (10960848) N:   26400/s (343993) W:  114500/s (1354730) R:  94100/s (52412)
[Apr 26 23:20:19] [INFO] Reading OSM data took: 26.03967484s
[Apr 26 23:20:46] [WARN] [writer] not found
[Apr 26 23:20:46] [WARN] [writer] not found
[Apr 26 23:20:46] [WARN] [writer] not found
[Apr 26 23:21:19] [INFO] [  1m0s] C:       0/s ( 0.0%) N:       0/s ( 0.0%) W:   26600/s (55.7%) R:   1810/s (100.0%)
[Apr 26 23:21:39] [INFO] [ 1m19s] C:       0/s ( 0.0%) N:   97500/s (100.0%) W:   30900/s (100.0%) R:   1810/s (100.0%)
[Apr 26 23:21:41] [INFO] Writing OSM data took: 1m21.469911895s
[Apr 26 23:21:41] [INFO] [PostGIS] Creating generalized tables took: 37.353µs
[Apr 26 23:21:41] [INFO] [PostGIS] Creating OSM id index on osm_master_routes took: 84.115078ms
[Apr 26 23:21:41] [INFO] [PostGIS] Creating OSM id index on osm_routes took: 109.183922ms
[Apr 26 23:21:41] [INFO] [PostGIS] Creating geometry index on osm_master_routes took: 48.398253ms
[Apr 26 23:21:41] [INFO] [PostGIS] Creating OSM id index on osm_route_members took: 299.59858ms
[Apr 26 23:21:44] [INFO] [PostGIS] Creating geometry index on osm_route_members took: 2.792209323s
[Apr 26 23:21:44] [INFO] [PostGIS] Creating geometry indices took: 3.092138446s
[Apr 26 23:21:44] [INFO] Importing OSM data took: 1m24.562240215s
[Apr 26 23:21:44] [INFO] [PostGIS] Analysing osm_routes took: 30.76086ms
[Apr 26 23:21:44] [INFO] [PostGIS] Indexing osm_master_routes on geohash took: 34.62636ms
[Apr 26 23:21:44] [INFO] [PostGIS] Clustering osm_master_routes on geohash took: 156.431004ms
[Apr 26 23:21:44] [INFO] [PostGIS] Analysing osm_master_routes took: 25.6634ms
[Apr 26 23:21:45] [INFO] [PostGIS] Indexing osm_route_members on geohash took: 1.43668858s
[Apr 26 23:21:52] [INFO] [PostGIS] Clustering osm_route_members on geohash took: 6.261467688s
[Apr 26 23:21:53] [INFO] [PostGIS] Analysing osm_route_members took: 1.518301131s
[Apr 26 23:21:53] [INFO] [PostGIS] Clustering on geometry took: 9.216807298s
[Apr 26 23:21:53] [INFO] [PostGIS] Rotating osm_routes from import -> public -> backup
[Apr 26 23:21:53] [INFO] [PostGIS] backup of osm_routes, to backup
[Apr 26 23:21:53] [INFO] [PostGIS] Rotating osm_master_routes from import -> public -> backup
[Apr 26 23:21:53] [INFO] [PostGIS] backup of osm_master_routes, to backup
[Apr 26 23:21:53] [INFO] [PostGIS] Rotating osm_route_members from import -> public -> backup
[Apr 26 23:21:53] [INFO] [PostGIS] backup of osm_route_members, to backup
[Apr 26 23:21:53] [INFO] [PostGIS] Rotating tables took: 33.598678ms
[Apr 26 23:21:53] [INFO] Imposm took: 2m0.057781314s
tobwen commented 8 years ago

Yep, confirmed. Maybe it's because of running multiple processing? I didn't look into the internals, but maybe one process is done already, but it's still monitored by the main application?

ImreSamu commented 8 years ago

Yep, confirmed.

Good news ! ( half work done :) - the hardest part is replicating ..)

Maybe it's because of running multiple processing

I don't think. But I will try to find ...

my current tip :

you can check your input data with the osmium command

the smallest test file - for replicating this warning is luxembourg-160101.osm.pbf

and ...

$  osmium check-refs -r luxembourg-160101.osm.pbf
There are 1552951 nodes, 197343 ways, and 3611 relations in this file.
Nodes     in ways      missing: 0
Nodes     in relations missing: 992
Ways      in relations missing: 35960
Relations in relations missing: 1547
tobwen commented 8 years ago

Good work. Yeah, the result of my testfile is looking equal. Could you add some verbose information like node xxxxx in way yyyy not found? Also see https://github.com/omniscale/imposm3/issues/99 for this.

ImreSamu commented 8 years ago

I am 80% sure, that the main cause is data integrity , but I am not know what is the best solution to solve this on the imposm3 side.

the warning is defined here and adding a Warning about what is missing - is a very - long list 

+ /go/src/github.com/omniscale/imposm3/imposm3 import -mapping ./test/route_relation_mapping.yml -read ../luxembourg-160101.osm.pbf -diff -write -optimize -overwritecache -deployproduction -connection postgis://localhost/imposm3dev
[Apr 27 00:26:31] [INFO] removing existing cache /tmp/imposm3
[Apr 27 00:26:32] [INFO] [reader] reading ../luxembourg-160101.osm.pbf with data till 2016-01-01 21:14:02 +0100 CET
[Apr 27 00:26:36] [INFO] [    4s] C:  641000/s (1552951) N:   19400/s (47694) W:  138800/s (197343) R:      0/s (3600)
[Apr 27 00:26:36] [INFO] Reading OSM data took: 4.648754571s
2016/04/27 00:26:38 ways.go:62: GetWay- missing way : 316780230    0/s       0/s ( 0.0%) W:       0/s       0/s ( 0.0%) R:      0/s      0/s ( 0.0%)
2016/04/27 00:26:38 ways.go:62: GetWay- missing way : 29721127
2016/04/27 00:26:38 ways.go:62: GetWay- missing way : 24047311
2016/04/27 00:26:38 ways.go:62: GetWay- missing way : 351325208
2016/04/27 00:26:38 ways.go:62: GetWay- missing way : 22954132
2016/04/27 00:26:38 ways.go:62: GetWay- missing way : 35200525     0/s       0/s ( 0.0%) W:       0/s       0/s ( 0.0%) R:      0/s      0/s ( 0.4%)
2016/04/27 00:26:38 ways.go:62: GetWay- missing way : 133500126
2016/04/27 00:26:38 ways.go:62: GetWay- missing way : 51830344
2016/04/27 00:26:38 ways.go:62: GetWay- missing way : 378332009
2016/04/27 00:26:38 ways.go:62: GetWay- missing way : 29412354
2016/04/27 00:26:38 ways.go:62: GetWay- missing way : 183946929
2016/04/27 00:26:38 ways.go:62: GetWay- missing way : 28988617
2016/04/27 00:26:38 ways.go:62: GetWay- missing way : 98508001
2016/04/27 00:26:38 ways.go:62: GetWay- missing way : 126402028
2016/04/27 00:26:38 ways.go:62: GetWay- missing way : 364810396
2016/04/27 00:26:38 ways.go:62: GetWay- missing way : 262754842
2016/04/27 00:26:38 ways.go:62: GetWay- missing way : 369647602
2016/04/27 00:26:38 ways.go:62: GetWay- missing way : 371523259
...

// I just added for the program code - a 'log.Println - for every cases of "return nil, NotFound", but this is too huge list ...

    if data == nil {
                log.Println("GetWay- missing way :", id)
                return nil, NotFound
    }
ImreSamu commented 8 years ago

Could you add some verbose information like node xxxxx in way yyyy not found?

I don't know what is the optimal solutions ..

But you can check & list all the integrity problems with the osmium tool - just add the '-i' option

-i, --show-ids
       Print all missing IDs to stdout. If you don't give this option, only a summary is shown.

A little test from the previous log:

2016/04/27 00:26:38 ways.go:62: GetWay- missing way : 316780230

and grepping for osmium output for 316780230 shows the correct context

full command

root@2c50e1d63d3f:/go/src/github.com/omniscale# osmium check-refs -r -i luxembourg-160101.osm.pbf  | grep 316780230
w316780230 in r4281
tobwen commented 8 years ago

Okay, you're right. imposm3 shouldn't be a tool for debugging OSM files. But perhaps you could make the warning "not found" a bit more informative that other users don't get confused ;)

For example: integry problem: node, way or relation not found... skipping item

ImreSamu commented 8 years ago

But perhaps you could make the warning "not found" a bit more informative that other users don't get confused ;)

agree .. and thank you for your suggestions!

note: I just helping pre-debugging the interesting issues, like this problem. :) I am not an official maintainer - so I don't know what will be the final solutions.

tobwen commented 8 years ago

Thanks for your great work, f.e. on your fork, especially "advanced filtering".

olt commented 8 years ago

NotFound errors are normally discarded. This was a stray warning when collecting relations for relation members (e.g. a master route references a sub route, but this sub route is not cached). These errors are normal with smaller extracts.

It's fixed with 3c271279b211c89aabe2bfd3825c9e8f548b651a Thanks for the report.