Closed ITninja04 closed 2 years ago
What OS are you on? Did you monitor your RAM usage while Valhalla is working? I routinely build Europe/the planet and don't have those issues on a 32 gb instance.
this is also a project wrapping another project, so it's often worth looking upstream as well, e.g. https://github.com/valhalla/valhalla/discussions/3288
I have the same issue with the latest docker image. Before it worked (4 cpu, 8 gb azure container instance, germany pbf file) and then kept crashing on startup both with building tiles and using pre-built tiles). Dropping down to 3.1.2 again fixed it for me.
8 gb might be too little by now, there's a couple of tile building stages that consume quite a lot of memory. even 16 gb might not be enough to process the planet, 32 gb should be. generally I'd recommend 16 gb, even for "smaller" extracts like germany.
did anyone here record an actual error message from valhalla_build_tiles
? so far I've only seen healthy logs
I can redeploy a bigger instance later and get the logs but the latest container also crashed when using prebuilt tiles (force_rebuild = false and use_tiles_ignore_pbf= true)
what does "crashed" mean exactly?
I am not quite sure how Azure determines that a container has stopped responding and last thing in the logs was "core dumped...". Unfortunately I wasn't paying too much attention to it at the time, since what I was doing worked on the previous latest version so I just dropped down to that. I've started a container with 16 gb, "new" latest version and my previous setup that is building tiles now and once that's done I'll share the logs here
ok cool thanks. if the host doesn't somehow destroy the failed containers, you can still run docker logs <container_name>
to see the logs.
It puts the container in a restart loop and I automatically deleted the deployment a few days ago. I am sure the information is somewhere but seemed easier to retry with one possible error source eliminated
So here is what's happening in the log (4 cpu, 16 gb ram container now):
2021/08/30 11:10:22.475249 [INFO] --Reverse restrictions added: 6332
2021/08/30 11:10:22.483888 [INFO] Finished
2021/08/30 11:10:22.483942 [INFO] Validating, finishing and binning tiles...
2021/08/30 11:11:56.684896 [INFO] Finished
2021/08/30 11:11:56.684931 [INFO] Binning inter-tile edges...
2021/08/30 11:12:55.041802 [INFO] Finished
2021/08/30 11:12:55.041867 [WARN] Possible duplicates at level: 0 = 20
2021/08/30 11:12:55.041879 [WARN] Possible duplicates at level: 1 = 58
2021/08/30 11:12:55.041887 [WARN] Possible duplicates at level: 2 = 0
2021/08/30 11:12:55.082453 [INFO] Cleaning up temporary *.bin files within /valhalla/scripts/valhalla_tiles/
tar: Removing leading `/' from member names
Successfully built files: /custom_files/germany-latest.osm.pbf
Hashing files: /custom_files/germany-latest.osm.pbf
Hashing file: /custom_files/germany-latest.osm.pbf
2d171e8b75f3f4afb77671f8ebd572c794c55d87ec2a9852e535c328908826ea
Found config file. Starting valhalla service!
2021/08/30 11:14:15.473757 [INFO] Tile extract successfully loaded with tile count: 1127
2021/08/30 11:14:15.475480 [WARN] (stat): No such file or directory
2021/08/30 11:14:15.475498 [WARN] Traffic tile extract could not be loaded
terminate called after throwing an instance of 'boost::wrapexcept<boost::property_tree::ptree_bad_path>'
what(): No such node (service_limits.max_avoid_locations.max_matrix_distance)
/valhalla/scripts/run.sh: line 54: 59 Aborted (core dumped) valhalla_service ${CUSTOM_CONFIG} ${server_threads}
terminate called after throwing an instance of 'boost::wrapexcept<boost::property_tree::ptree_bad_path>'
what(): No such node (service_limits.max_avoid_locations.max_matrix_distance)
you'll need to remove your valhalla.json
.. you can ask in the upstream repository to come up with a more explicit error telling you that your valhalla.json is not up-to-date (it does happen very frequently that new mandatory keys are introduced..)
note, that has nothing to do with the OP's problem. and the lesson being, always check logs!
What OS are you on? Did you monitor your RAM usage while Valhalla is working? I routinely build Europe/the planet and don't have those issues on a 32 gb instance.
this is also a project wrapping another project, so it's often worth looking upstream as well, e.g. valhalla/valhalla#3288
CentOS 8 and MacOS 11.5 on the CentOS box I did watch and the RAM would creep to the 120 GB limit then it'd crash.
When I say crashing I mean the container terminates because it runs out of RAM, however the logs show this at the end:
2021/08/30 03:27:35.189607 [INFO] Class: 7 exit count = 0
/valhalla/scripts/configure_valhalla.sh: line 272: 19 Killed valhalla_build_tiles -c ${config_file} ${files}
There are tiles created in /valhalla/scripts/valhalla_tiles
-rw-r--r--. 1 root root 28864336 Aug 29 20:52 access.bin
-rw-r--r--. 1 root root 12599712 Aug 29 20:52 complex_from_restrictions.bin
-rw-r--r--. 1 root root 11892960 Aug 29 20:52 complex_to_restrictions.bin
-rw-r--r--. 1 root root 2359268520 Aug 29 20:52 edges.bin
-rw-r--r--. 1 root root 7946331400 Aug 29 20:52 nodes.bin
-rw-r--r--. 1 root root 2291891 Aug 29 20:52 tile_manifest.json
-rw-r--r--. 1 root root 22273174272 Aug 29 20:52 way_nodes.bin
-rw-r--r--. 1 root root 5580275456 Aug 29 20:52 ways.bin
But they are never copied to the custom_files directory, so when the container crashes that is usually lost.
The image above is htop output from the currently running docker container on my mac. As it progresses I can update with more screenshots.
the RAM would creep to the 120 GB
and you're actually talking about RAM used, not the virtual RAM reserved right? not sure if docker doesn't get confused with the -m
option maybe? can you try without?
There are tiles created in /valhalla/scripts/valhalla_tiles
there are no tiles yet. in fact, where it exits is the part it needs to do just before building actual tiles. I'm not super familiar with how the .bin
files are loaded or scanned, but they are always mmap'd, meaning your virtual RAM can get > 120GB, but physical RAM never exceeds a few 10s of GBs. you're not working with some custom OSM PBF are you? (added/deleted/edited nodes/edges?)
lastly, I can only encourage you to build valhalla from source on centos (8 builds totally fine) and if you see the same issue there we know it's an upstream problem.
and you're actually talking about RAM used, not the virtual RAM reserved right? not sure if docker doesn't get confused with the
-m
option maybe? can you try without?
The -m was there for podman (docker replacement for Centos) and that allows the container to use up to 120GB of physical RAM. I checked the logs and indeed the box was using that much physical RAM.
you're not working with some custom OSM PBF are you? (added/deleted/edited nodes/edges?)
correct, no custom PBF, just using the latest download for North America.
lastly, I can only encourage you to build valhalla from source on centos (8 builds totally fine) and if you see the same issue there we know it's an upstream problem.
That is what I am working on now, will keep you all posted.
That is what I am working on now, will keep you all posted.
great thanks! hope that'll work.
somewhat sounds like a problem with podman to me.. never heard of those issues using docker so far
somewhat sounds like a problem with podman to me.. never heard of those issues using docker so far
The same behavior happens on my mac running docker desktop with everything set as follows
The docker run command I am using on my mac is:
docker run -i -d -e use_tiles_ignore_pbf=True -e force_rebuild=False -e force_rebuild_elevation=False -e build_elevation=True -e build_admins=True -e build_time_zones=True -v /Users/altiview/Desktop/valhalla_files:/custom_files --privileged --name valhallabuilder -m 47g -p 8002:8002 gisops/valhalla:latest
The issue is still occurring when using north-america-latest.osm.pbf but works when using the united states only pbf.
The issue occurs while using the docker image you all have uploaded, along with building Valhalla from valhalla/valhalla so I can only assume it's something to do with older code OR the pbf is corrupt.
thanks for the info!
The issue occurs while using the docker image you all have uploaded, along with building Valhalla from valhalla/valhalla
are you saying that both, our image and the current valhalla master, are showing the same behavior of OOM killed? what you describe makes me wonder if it has smth to do with either:
I think if you take care to not include edges/nodes that are crossing the antimeridian (like the US extract apparently) you should be fine. until further notice that is, I'm sure someone will very soon squash that bug!
I'll leave this open until it's resolved upstream
@ITninja04 did you get this resolved?
closing as stale, feel free to reopen
Sorry, of course that’s probably still an issue, it’s not resolved upstream yet..
according to #53 north america seems to build fine again
I'm having the same issue trying to build the tiles using the latest container image.
It's been on the last line above for around an hour and will eventually crash due to running out of RAM on the host.
My server has 128 GB of RAM and 8 cores, the container is allowed to use almost all of the RAM.
Here is my docker run command:
I pulled the latest docker image before starting this run, so everything should be up to date.
Any idea on how to fix this issue?