search

pelias / pbf2json

An OpenStreetMap pbf parser which exports json, allows you to cherry-pick tags and handles denormalizing ways and relations. Available as a standalone binary and comes with a convenient npm wrapper.
https://pelias.io
MIT License
143 stars 36 forks source link

[pbf2json]: denormalize failed for relation #76

Closed woiza closed 5 years ago

woiza commented 5 years ago

I am building europe with your docker files. Everything is preparing and importing nicely, except that I have thousands of the following lines in my log:

[pbf2json]: 2019/04/24 07:10:46 denormalize failed for relation: 3607248 no ways found [pbf2json]: 2019/04/24 07:10:46 denormalize failed for way: 244292338 node not found: 2516251096 [pbf2json]: 2019/04/24 07:10:46 denormalize failed for relation: 3607301 no ways found [pbf2json]: 2019/04/24 07:10:46 denormalize failed for way: 32919502 node not found: 302613551 [pbf2json]: 2019/04/24 07:06:46 denormalize failed for relation: 364100 no ways found

I use the following pbf file: https://download.geofabrik.de/europe-latest.osm.pbf

Here is my complete pelias.json config: https://github.com/dataBikeHsUlm/docker/blob/master/projects/europe/pelias.json

And docker compose: https://github.com/dataBikeHsUlm/docker/blob/master/projects/europe/docker-compose.yml

What's wrong?

missinglink commented 5 years ago

Hi @woiza, the data comes normalized, in the case of denormalize failed for way: 244292338 node not found: 2516251096 the error message is saying that the definition of https://www.openstreetmap.org/way/244292338 if found in the .osm.pbf file but https://www.openstreetmap.org/node/2516251096 is not found in the .osm.pbf file.

One of two things has happened

  1. the .osm.pbf file has become corrupted, or some elements have been 'clipped' from the file
  2. there is a bug in pbf2json

Would it be possible for you to confirm whether node 2516251096 is contained with the .osm.pbf file? If you're having trouble, please send me a link to the file or a shasum hash of it and a link to the download?

woiza commented 5 years ago

Hi @missinglink I downloaded the file last week from geofabrik.de and unfortunately my server is not accessible from the public internet. How can I check if the mentioned nodes are contained in it? Are there tools for this?

missinglink commented 5 years ago

Try a command such as this on your file:

curl -O http://download.geofabrik.de/antarctica-latest.osm.pbf
docker run --rm -it -v"$PWD/antarctica-latest.osm.pbf:/tmp/extract.osm.pbf" missinglink/osmium getid -f debug '/tmp/extract.osm.pbf' w329705189 n3366314117

docs: https://docs.osmcode.org/osmium/latest/osmium-getid.html

woiza commented 5 years ago

docker run --rm -it -v"/data/pelias-docker-compose/openstreetmap/europe-latest.osm.pbf:/tmp/extract.osm.pbf" missinglink/osmium getid -f debug '/tmp/extract.osm.pbf' w244292338 n2516251096

Here's the result:

node 2516251096 visible version: 1 changeset: 0 timestamp: 2013-11-01T15:01:04Z (1383318064) user: 0 "" lon/lat: 6.7207356,44.6197842

way 244292338 visible version: 8 changeset: 0 timestamp: 2016-03-01T10:08:13Z (1456826893) user: 0 "" nodes: 387 (closed) 000: 2516251096 001: 2516251333 002: 2516251159 003: 2516251360 004: 2516251291 005: 2516251125 006: 2516251329 007: 2516251223 008: 2516251356 009: 2516251232 010: 2516251303 011: 2516251364 012: 2516251245 013: 2516251103 014: 2516251169 015: 2516251369 016: 2516251314 017: 2516251348 018: 2516251280 019: 2516251224 020: 2516251307 021: 2516251336 022: 2516251267 023: 2516251114 024: 2516251133 025: 2516251178 026: 2516251308 027: 2516251302 028: 2516251203 029: 2516251269 030: 2516251100 031: 2516251161 032: 2516251335 033: 2516251104 034: 2516251285 035: 2516251391 036: 2516251185 037: 2516251120 038: 2516251330 039: 2516251288 040: 2516251173 041: 2516251140 042: 2516251206 043: 2516251339 044: 2516251105 045: 2516251294 046: 2516251352 047: 2516251374 048: 2516251313 049: 2516251320 050: 2516251117 051: 2516251164 052: 2516251217 053: 2516251148 054: 2516251219 055: 2516251175 056: 2516251289 057: 2516251250 058: 2516251231 059: 2516251191 060: 2516251334 061: 2516251196 062: 2516251188 063: 2516251151 064: 2516251375 065: 2516251386 066: 2516251239 067: 2516251379 068: 2516251299 069: 2516251357 070: 2516251202 071: 2516251305 072: 2516251197 073: 2516251237 074: 2516251216 075: 2516251368 076: 2516251354 077: 2516251390 078: 2516251090 079: 2516251213 080: 2516251118 081: 2516251115 082: 2516251087 083: 2511986772 084: 2511986878 085: 2511986666 086: 2511986611 087: 2511986683 088: 2511986932 089: 2511987004 090: 2511986689 091: 2511986891 092: 2516251195 093: 2516251130 094: 2516251325 095: 2516251287 096: 2516251290 097: 2516251276 098: 2516251241 099: 2516251366 100: 2516251281 101: 2516251388 102: 2516251332 103: 2516251372 104: 2516251189 105: 2516251170 106: 2516251304 107: 2516251268 108: 2516251318 109: 2516251367 110: 2516251097 111: 2516251085 112: 2516251323 113: 2516442179 114: 2516251236 115: 2516251310 116: 2516251256 117: 2516251186 118: 2516251244 119: 2516251261 120: 2516251092 121: 2516251153 122: 2516251205 123: 2516251121 124: 2516251387 125: 2516251362 126: 2516251107 127: 2516251238 128: 2516251371 129: 2516251139 130: 2516251377 131: 2516251111 132: 2516251220 133: 2516251319 134: 2516251084 135: 2516251218 136: 2516251260 137: 2516251187 138: 2516251306 139: 2516251198 140: 2516251134 141: 2516251204 142: 2516251384 143: 2516251207 144: 2516251160 145: 2516251150 146: 2516251155 147: 2516251176 148: 2516251324 149: 2516251292 150: 2516251210 151: 2516251248 152: 2516251312 153: 2516251225 154: 2516251242 155: 2516251293 156: 2516251358 157: 2516251365 158: 2516251275 159: 2516251351 160: 2516251147 161: 2516251271 162: 2516251322 163: 2516251376 164: 2516251378 165: 526697910 166: 526697912 167: 526697914 168: 2516251131 169: 2516251279 170: 2516251328 171: 2516251112 172: 2516251158 173: 2516251089 174: 2516251383 175: 2516251373 176: 2516251154 177: 2516251284 178: 2516251341 179: 2516251342 180: 2516251346 181: 2516251385 182: 2516251337 183: 2516251142 184: 2516251201 185: 2516251345 186: 2516251340 187: 2516251349 188: 526697918 189: 526697920 190: 2516251192 191: 2516251157 192: 2516251235 193: 2516251226 194: 2516251234 195: 2516251243 196: 2516251251 197: 2516251149 198: 2516251254 199: 2516251183 200: 2516251311 201: 2516251249 202: 526697938 203: 526697939 204: 526697940 205: 2516251286 206: 2516251246 207: 2516251093 208: 526697948 209: 526697950 210: 526697952 211: 526697954 212: 526697956 213: 2516251221 214: 526697960 215: 526697962 216: 526697964 217: 526697966 218: 526697968 219: 526697970 220: 526697971 221: 526697972 222: 526697973 223: 526697974 224: 526697976 225: 526697978 226: 526697979 227: 526697981 228: 526697984 229: 526697986 230: 526697987 231: 526697988 232: 526697989 233: 526697990 234: 526697991 235: 526697992 236: 526697993 237: 526697994 238: 526697995 239: 526697996 240: 526697997 241: 526697998 242: 526697999 243: 526698000 244: 526698001 245: 526698002 246: 526698003 247: 526698004 248: 526698005 249: 526698006 250: 526698007 251: 527121948 252: 527121949 253: 527121950 254: 3955577686 255: 3955577684 256: 3955577722 257: 3955577741 258: 3955577699 259: 3955577733 260: 3955577749 261: 3955577734 262: 3955577745 263: 3955577738 264: 3955577720 265: 3955577726 266: 3955577756 267: 3955577690 268: 527121951 269: 527121952 270: 3106058814 271: 527121953 272: 527121954 273: 527121046 274: 527121047 275: 527121048 276: 527121049 277: 527121050 278: 527121051 279: 2516251208 280: 2516251212 281: 2516297921 282: 2516251316 283: 2516297940 284: 2516297975 285: 2516251181 286: 2516297932 287: 2516297938 288: 2516251270 289: 2516251177 290: 2516251272 291: 2516251137 292: 2516251126 293: 2516251146 294: 2516251129 295: 2516251182 296: 2516251355 297: 2516251168 298: 2516251163 299: 2516251174 300: 2516251262 301: 2516251214 302: 2516251167 303: 2516251215 304: 2516251229 305: 2516251259 306: 2516251295 307: 2516251265 308: 2516251282 309: 2516251253 310: 2516251255 311: 2516251138 312: 2516251144 313: 2516251094 314: 2516251227 315: 2516251257 316: 2516251091 317: 2516251136 318: 2516251277 319: 2516251193 320: 2516251088 321: 2516251102 322: 2516251278 323: 2516251095 324: 2516251309 325: 2516251263 326: 527121061 327: 3469235663 328: 527121062 329: 527121063 330: 3469235662 331: 3469235619 332: 527121064 333: 527121065 334: 527121066 335: 527121067 336: 527121068 337: 527121069 338: 527121070 339: 527121034 340: 527121651 341: 527121652 342: 527121653 343: 527121654 344: 527121655 345: 527121656 346: 2516251194 347: 2516251128 348: 2516251166 349: 2516251180 350: 2516251382 351: 2516251326 352: 2516251359 353: 2516251350 354: 2516251380 355: 2516251199 356: 2516251297 357: 2516251344 358: 2516251156 359: 2516251165 360: 2516251266 361: 2516251135 362: 2516251127 363: 2516251099 364: 2516251184 365: 2516251145 366: 2516251338 367: 2516251123 368: 2516251389 369: 2516251298 370: 2516251258 371: 2516252729 372: 2516251101 373: 2516251171 374: 2516251247 375: 2516251108 376: 2516251301 377: 2516251274 378: 2516251233 379: 2516251317 380: 2516251209 381: 2516251347 382: 2516251327 383: 2516251228 384: 2516251141 385: 2516251361 386: 2516251096

woiza commented 5 years ago

I used Valhalla to extract polylines since my server does not have enough RAM for "pelias prepare polylines". Could this be related to my issue?

woiza commented 5 years ago

I got more RAM for my server, used pelias to extract polylines and started the openstreetmaps import again. Same result: "denormalize failed for way: 244292338 node not found: 251625109" and thousands more...

missinglink commented 5 years ago

Interesting, I will have a look in to this and see if there is a bug.

The denormalize failed message is fairly common for extracts where the bounding box has clipped some of the elements near the edge of the boundary.

For now I wouldn't worry too much about it, it's likely only affecting a very small percentage of the ways/relations in the file.

Please follow this issue and I'll give you an update when I know more.

missinglink commented 5 years ago

Regarding the RAM.. the denormalize step in pbf2json does not require significant amount of RAM.

But you're correct in saying that the other step in missinglink/pbf which creates the polyline extract is limited by available RAM, but that's unrelated to this issue.

woiza commented 5 years ago

Thanks, will do.

... grep denormalize | wc -l --> 62793 (way and relation) The pbf file has 20GB. I don't know what the percentage is but the number seems high.

missinglink commented 5 years ago

This should now be resolved by https://github.com/pelias/pbf2json/pull/77 Thanks for the bug report @woiza, there was an error denormalizing relations which should be fixed now.

It will take a bit of time to merge and release the related Pelias packages such as pelias/openstreetmap

missinglink commented 5 years ago

@woiza you can test this out by changing the openstreetmap.image setting in your docker-compose file to:

pelias/openstreetmap:relations_bugfix-2019-04-25-cc778095371c142147e31249947a3b43fb57d46d

just remember to change that back to master once https://github.com/pelias/openstreetmap/pull/480 is merged to master.

woiza commented 5 years ago

@missinglink did that, started the import again and got no errors. Thanks!