Core dump when building North America AND Europe

[brad-4k]

I get a core dump when trying to build with both the North America AND Europe PBF files. It runs fine with only Europe OR North America. It doesn't appear to be running out of memory or storage. Any ideas?

System information: AWS EC2 8 CPUs 64G Mem $ uname -a Linux ip-172-31-4-115 4.4.0-1087-aws #98-Ubuntu SMP Wed Jun 26 05:50:53 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Log: 5:43.293925 [INFO] Start stage = initialize End stage = cleanup 2022/01/05 19:15:43.295663 [WARN] Non-empty /valhalla/scripts/valhalla_tiles/0 will be purged of tiles 2022/01/05 19:15:43.398855 [WARN] Non-empty /valhalla/scripts/valhalla_tiles/1 will be purged of tiles 2022/01/05 19:15:43.712235 [WARN] Non-empty /valhalla/scripts/valhalla_tiles/2 will be purged of tiles 2022/01/05 19:15:46.275906 [INFO] Parsing files for ways: /custom_files/europe-latest.osm.pbf, /custom_files/north-america-latest.osm.pbf 2022/01/05 19:15:46.276063 [INFO] Parsing ways... 2022/01/05 19:23:07.563335 [INFO] invalid_argument thrown for condition (2021 Aug 27- 2022 Jan 31) 2022/01/05 19:23:13.506014 [INFO] invalid_argument thrown for condition (2021 Oct 07- 2022 Feb 28) 2022/01/05 19:24:20.102276 [WARN] Exceeded max speed for way id: 25469744 2022/01/05 19:24:20.102455 [WARN] Exceeded max speed for way id: 25469747 2022/01/05 19:25:59.469598 [WARN] Error during parsing of layer tag on the way 30492014: stoi 2022/01/05 19:28:30.881110 [WARN] Error during parsing of layer tag on the way 41749349: stoi 2022/01/05 19:28:45.162944 [WARN] Exceeded max speed for way id: 43053918 2022/01/05 19:29:25.661362 [WARN] Error during parsing of layer tag on the way 48462727: stoi 2022/01/05 19:30:53.082021 [INFO] invalid_argument thrown for condition Nov-Feb 8:00-dusk || dusk-00:00 off) 2022/01/05 19:30:53.084063 [INFO] invalid_argument thrown for condition Nov-Feb 8:00-dusk || dusk-00:00 off) 2022/01/05 19:31:16.318393 [INFO] invalid_argument thrown for condition Nov-Feb 8:00-dusk || dusk-00:00 off) 2022/01/05 19:31:23.129362 [INFO] invalid_argument thrown for condition Nov-Feb 8:00-dusk || dusk-00:00 off) 2022/01/05 19:31:23.131080 [INFO] invalid_argument thrown for condition Nov-Feb 8:00-dusk || dusk-00:00 off) 2022/01/05 19:31:23.132744 [INFO] invalid_argument thrown for condition Nov-Feb 8:00-dusk || dusk-00:00 off) 2022/01/05 19:31:23.134480 [INFO] invalid_argument thrown for condition Nov-Feb 8:00-dusk || dusk-00:00 off) 2022/01/05 19:31:23.136125 [INFO] invalid_argument thrown for condition Nov-Feb 8:00-dusk || dusk-00:00 off) 2022/01/05 19:31:24.747101 [INFO] invalid_argument thrown for condition Nov-Feb 8:00-dusk || dusk-00:00 off) 2022/01/05 19:31:24.748836 [INFO] invalid_argument thrown for condition Nov-Feb 8:00-dusk || dusk-00:00 off) 2022/01/05 19:31:24.750544 [INFO] invalid_argument thrown for condition Nov-Feb 8:00-dusk || dusk-00:00 off) 2022/01/05 19:31:24.752233 [INFO] invalid_argument thrown for condition Nov-Feb 8:00-dusk || dusk-00:00 off) 2022/01/05 19:31:24.753930 [INFO] invalid_argument thrown for condition Nov-Feb 8:00-dusk || dusk-00:00 off) 2022/01/05 19:31:24.755621 [INFO] invalid_argument thrown for condition Nov-Feb 8:00-dusk || dusk-00:00 off) 2022/01/05 19:31:24.757332 [INFO] invalid_argument thrown for condition Nov-Feb 8:00-dusk || dusk-00:00 off) 2022/01/05 19:31:25.140022 [INFO] invalid_argument thrown for condition Nov-Feb 8:00-dusk || dusk-00:00 off) ... 2022/01/05 21:13:45.911629 [WARN] Error during parsing of layer tag on the way 987800239: stoi 2022/01/05 21:23:59.492248 [INFO] Added 283697 culdesac roundabouts from 412603 candidates. 2022/01/05 21:23:59.492293 [INFO] Finished with 116500255 routable ways containing 1188615188 nodes 2022/01/05 21:24:00.609484 [INFO] Sorting osm access tags by way id... 2022/01/05 21:25:28.084936 [INFO] Finished 2022/01/05 21:25:29.854086 [INFO] Parsing files for relations: /custom_files/europe-latest.osm.pbf, /custom_files/north-america-latest.osm.pbf 2022/01/05 21:25:29.860840 [INFO] Parsing relations... 2022/01/05 21:52:46.363129 [INFO] invalid_argument thrown for condition Jul 1- Aug 31 off 2022/01/05 21:52:50.764123 [INFO] Finished with 1041013 simple restrictions 2022/01/05 21:52:50.764156 [INFO] Finished with 0 lane connections 2022/01/05 21:52:50.791066 [INFO] Sorting complex restrictions by from id... 2022/01/05 21:52:50.839908 [INFO] Sorting complex restrictions by to id... 2022/01/05 21:52:50.886808 [INFO] Finished 2022/01/05 21:52:50.889770 [INFO] Parsing files for nodes: /custom_files/europe-latest.osm.pbf, /custom_files/north-america-latest.osm.pbf 2022/01/05 21:52:50.889819 [INFO] Sorting osm way node references by node id... terminate called after throwing an instance of 'std::runtime_error' what(): sequence: /valhalla/scripts/valhalla_tiles/way_nodes.bin has an incorrect size for type /valhalla/scripts/configure_valhalla.sh: line 272: 24 Aborted (core dumped) valhalla_build_tiles -c ${config_file} ${files}

[nilsnolde]

The last few lines are usually enough;)

Well might be a bug. What’s the version you’re using?

You can try to merge the PBFs, then try building again.

[brad-4k]

The last few lines are usually enough;)

Well might be a bug. What’s the version you’re using?

You can try to merge the PBFs, then try building again.

Thank you for the quick reply. I tried a merged PBF with the same results.

Version is:

REPOSITORY TAG IMAGE ID CREATED SIZE docker.pkg.github.com/gis-ops/docker-valhalla/valhalla 3.1.4 27606002b3af 2 months ago 387MB

[nilsnolde]

hm, ok.. can you try the current latest image?

I had a quick look what that error represents and doesn't make much sense to me. think I'll file upstream, maybe kevin has an idea.

do you still have access to the temporary files the pbf(s) produced in valhalla_tiles?

(btw, sorry, your log output is perfect.. looked at it first on the phone..)

[nilsnolde]

so, in case you're not following the valhalla issue: how much disk space you got on your machine?

[brad-4k]

@nilsnolde I'll run one more time to see if I'm running out of storage before I try latest. Storage is roughly 150Gb with only 38% used.

Thank you for listing this on valhalla/valhalla. I should have tried there first, my apologies.

[nilsnolde]

uh ja, 100 GB could be too low. the temp files valhalla generates are HUGE (around 300 GB for the planet). that's very likely the problem.

no worries, I'm also a valhalla maintainer so doesn't matter really.

[brad-4k]

uh ja, 100 GB could be too low. the temp files valhalla generates are HUGE (around 300 GB for the planet). that's very likely the problem.

no worries, I'm also a valhalla maintainer so doesn't matter really.

It looks like it doesn't get over 75% used, the crash is when it's 72%. I'm checking once per second so there is a chance that it jumps up real fast and then back down, but doubt it.

I'll try using latest next.

[nilsnolde]

Hmpf, that’s a pity, would’ve been an easy fix.. thanks for investing the time to get to the bottom of this!

[brad-4k]

Ok, with latest, it crashes... but the filesystem is 100% used. Doh!

22:58:42.506280 [WARN] Error during parsing of layer tag on the way 862092167: stoi /valhalla/scripts/configure_valhalla.sh: line 172: 38 Bus error (core dumped) valhalla_build_tiles -c ${CONFIG_FILE} -e build ${files}

[nilsnolde]

Uff finally;) hope it’ll work when you add disk space.

[brad-4k]

Uff finally;) hope it’ll work when you add disk space.

Ok, So I believe it build but I also believe the is a problem with latest. I just performed a query in North America and I get this: Query: http://localhost:8002/route?json={%22locations%22:[{%22lat%22:47.66919095071193,%22lon%22:-122.35056209730568},{%22lat%22:32.26991977170772,%22lon%22:-110.98902358319154}],%22costing%22:%22auto%22,%22directions_options%22:{%22units%22:%22miles%22}}

Error response: [libprotobuf ERROR google/protobuf/wire_format_lite.cc:626] String field 'valhalla.TaggedValue.value' contains invalid UTF-8 data when serializing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes. 2022/01/09 18:20:00.813579 [INFO] Got Odin Request 0 2022/01/09 18:20:00.813878 [ERROR] Failed parsing pbf in Odin::Worker [libprotobuf ERROR google/protobuf/wire_format_lite.cc:626] String field 'valhalla.TaggedValue.value' contains invalid UTF-8 data when parsing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes. 0 2022/01/09 18:20:00.814158 400 283

[nilsnolde]

that'd be an old latest image then, you'll need to pull a fresh one, see #51

[brad-4k]

that'd be an old latest image then, you'll need to pull a fresh one, see #51

Ok. Looks like mine was 3 weeks old. Trying "latest" latest which appears to be 3 days old.

That said, my tile generation phase got me down to 9G free out of the 300G... I was sweating a little :). It freed up its temporary files and landed on 177G Used, 120G free.

[nilsnolde]

177 GB used for two continents? that'd be ridiculously too high, whole planet has 70 GB.. or you're telling me about your entire hard drive for some reason?

[brad-4k]

177 GB used for two continents? that'd be ridiculously too high, whole planet has 70 GB.. or you're telling me about your entire hard drive for some reason?

That was total system, yes sorry. Total results are around 81G for Europe and North America. I do not know why it's larger than the planet database.