search

gis-ops / docker-valhalla

This is our flexible Docker repository for the Valhalla routing engine
MIT License
240 stars 70 forks source link

Rebuilding tiles on each start. #13

Closed azhi closed 3 years ago

azhi commented 3 years ago

I'm having trouble starting valhalla without rebuild.

Log from first start that builds everything:

Build log 
TEST: /custom_files           
Valhalla tiles not found!
Couldn't find .file_hashes.txt
Hash not found for: california-latest.osm.pbf!
Hash counter couldn't find hash file!
Either valhalla_tiles.tar couldn't be found or new files or file constellations were detected. Rebuilding files:  /custom_files/california-latest.osm.pbf
PBF hashes: 0
PBF file Counter: 1

=========================================================================
= No valid bounding box or elevation parameter set. Skipping elevation! =
=========================================================================

==========================
= Skipping admin regions =
==========================

==========================
= Skipping timezone data =
==========================

==========================
= Using existing config file =
==========================

=========================
= Build the tile files. =
=========================
Running build tiles with: /custom_files/valhalla.json  /custom_files/california-latest.osm.pbf
2021/01/12 13:33:39.236728 [INFO] Start stage = initialize End stage = cleanup
2021/01/12 13:33:39.238619 [INFO] Parsing files for ways: /custom_files/california-latest.osm.pbf
2021/01/12 13:33:39.238809 [INFO] Parsing ways...
2021/01/12 13:35:04.952612 [INFO] invalid_argument thrown for condition (Jun 15-Labor Day)
2021/01/12 13:35:14.911536 [INFO] invalid_argument thrown for condition (Jun 15-Labor Day)
2021/01/12 13:36:33.560932 [INFO] Finished with 2579025 routable ways containing 28228675 nodes
2021/01/12 13:36:33.594439 [INFO] Sorting osm access tags by way id...
2021/01/12 13:36:33.613080 [INFO] Finished
2021/01/12 13:36:33.615281 [INFO] Parsing files for relations: /custom_files/california-latest.osm.pbf
2021/01/12 13:36:33.615562 [INFO] Parsing relations...
2021/01/12 13:37:21.124575 [INFO] Finished with 43520 simple restrictions
2021/01/12 13:37:21.124605 [INFO] Finished with 0 lane connections
2021/01/12 13:37:21.128371 [INFO] Sorting complex restrictions by from id...
2021/01/12 13:37:21.135937 [INFO] Sorting complex restrictions by to id...
2021/01/12 13:37:21.141034 [INFO] Finished
2021/01/12 13:37:21.142950 [INFO] Parsing files for nodes: /custom_files/california-latest.osm.pbf
2021/01/12 13:37:21.142991 [INFO] Sorting osm way node references by node id...
2021/01/12 13:37:37.644936 [INFO] Parsing nodes...
2021/01/12 13:38:53.890431 [INFO] Finished with 24354202 nodes contained in routable ways
2021/01/12 13:38:53.890461 [INFO] Sorting osm way node references by way index and node shape index...
2021/01/12 13:39:07.370631 [INFO] Finished: max_osm_id 8309434941
2021/01/12 13:39:07.370665 [INFO] Number of nodes with refs (exits) = 6151
2021/01/12 13:39:07.370668 [INFO] Number of nodes with exit_to = 1
2021/01/12 13:39:07.370670 [INFO] Number of nodes with names = 37
2021/01/12 13:39:07.370673 [INFO] Number of way refs = 14575
2021/01/12 13:39:07.370675 [INFO] Number of reverse way refs = 1221
2021/01/12 13:39:07.370677 [INFO] Unique Node Strings (names, refs, etc.) = 1173
2021/01/12 13:39:07.370680 [INFO] Unique Strings (names, refs, etc.) = 271421
2021/01/12 13:39:07.371667 [INFO] Creating graph edges from ways...
2021/01/12 13:39:10.814600 [INFO] Finished with 5682821 graph edges
2021/01/12 13:39:10.873489 [INFO] Sorting graph...
2021/01/12 13:39:18.564381 [INFO] Finished with 4387384 graph nodes
2021/01/12 13:39:18.599975 [INFO] Writing tile manifest to /valhalla/scripts/valhalla_tiles/tile_manifest.json
2021/01/12 13:39:18.606243 [INFO] Reclassifying link graph edges...
2021/01/12 13:39:19.786516 [INFO] Class: 0 exit count = 6166
2021/01/12 13:39:19.786625 [INFO] Class: 1 exit count = 1595
2021/01/12 13:39:19.786642 [INFO] Class: 2 exit count = 7342
2021/01/12 13:39:19.786654 [INFO] Class: 3 exit count = 9359
2021/01/12 13:39:19.786665 [INFO] Class: 4 exit count = 4047
2021/01/12 13:39:19.786676 [INFO] Class: 5 exit count = 287
2021/01/12 13:39:19.786704 [INFO] Class: 6 exit count = 770
2021/01/12 13:39:19.786724 [INFO] Class: 7 exit count = 0
2021/01/12 13:39:20.262367 [INFO] Finished with 35845 reclassified.  Turn channel count = 13832
2021/01/12 13:39:20.311558 [INFO] Reclassifying ferry connection graph edges...
2021/01/12 13:39:21.567595 [INFO] Finished ReclassifyFerryEdges: ferry_endpoint_count = 5, 143 edges reclassified.
2021/01/12 13:39:21.607705 [INFO] Building 793 tiles with 2 threads...
2021/01/12 13:39:21.609276 [WARN] Admin db /data/valhalla/admin.sqlite not found.  Not saving admin information.
2021/01/12 13:39:21.609306 [WARN] Time zone db /data/valhalla/tz_world.sqlite not found.  Not saving time zone information.
2021/01/12 13:39:21.609286 [WARN] Admin db /data/valhalla/admin.sqlite not found.  Not saving admin information.
2021/01/12 13:39:21.609366 [WARN] Time zone db /data/valhalla/tz_world.sqlite not found.  Not saving time zone information.
2021/01/12 13:39:48.784921 [INFO] Finished
2021/01/12 13:39:48.785402 [INFO] Node Count = 4387384
2021/01/12 13:39:48.785542 [INFO] Directed Edge Count = 11365642
2021/01/12 13:39:48.785620 [INFO] EdgeInfo Count = 5720896
2021/01/12 13:39:48.785760 [INFO] Enhancing local graph...
2021/01/12 13:39:48.791274 [WARN] (stat): /data/valhalla/traffic.tar No such file or directory
2021/01/12 13:39:48.791455 [WARN] Traffic tile extract could not be loaded
2021/01/12 13:39:48.799375 [WARN] Admin db /data/valhalla/admin.sqlite not found.  Not saving admin information.
2021/01/12 13:39:48.799375 [WARN] Admin db /data/valhalla/admin.sqlite not found.  Not saving admin information.
2021/01/12 13:48:36.985474 [WARN] Exceeding maximum.  Average speed: 141
2021/01/12 13:48:36.990364 [WARN] Exceeding maximum.  Average speed: 141
2021/01/12 13:54:08.084753 [WARN] Exceeding maximum.  Average speed: 141
2021/01/12 13:54:08.099428 [WARN] Exceeding maximum.  Average speed: 141
2021/01/12 13:54:40.876317 [WARN] Exceeding maximum.  Average speed: 141
2021/01/12 13:54:41.430014 [WARN] Exceeding maximum.  Average speed: 141
2021/01/12 13:54:41.431064 [WARN] Exceeding maximum.  Average speed: 141
2021/01/12 13:54:41.611614 [WARN] Exceeding maximum.  Average speed: 141
2021/01/12 13:55:38.681067 [WARN] Exceeding maximum.  Average speed: 141
2021/01/12 13:55:39.605283 [WARN] Exceeding maximum.  Average speed: 141
2021/01/12 13:55:39.605882 [WARN] Exceeding maximum.  Average speed: 141
2021/01/12 13:55:39.711413 [WARN] Exceeding maximum.  Average speed: 141
2021/01/12 13:55:39.712256 [WARN] Exceeding maximum.  Average speed: 141
2021/01/12 13:55:40.284352 [WARN] Exceeding maximum.  Average speed: 141
2021/01/12 13:55:40.285086 [WARN] Exceeding maximum.  Average speed: 141
2021/01/12 13:55:40.285194 [WARN] Exceeding maximum.  Average speed: 141
2021/01/12 13:56:49.921459 [INFO] Finished with max_density 25.746178
2021/01/12 13:56:49.921489 [INFO] internal intersection = 131333
2021/01/12 13:56:49.921712 [INFO] GraphFilter - nothing to filter. Skipping...
2021/01/12 13:56:49.921763 [INFO] Transit directory not found. Transit will not be added.
2021/01/12 13:56:49.921790 [INFO] HierarchyBuilder                                    
2021/01/12 13:57:23.792875 [INFO] Done HierarchyBuilder
2021/01/12 13:57:23.795101 [INFO] Creating shortcuts on level 1
2021/01/12 13:57:25.523076 [WARN] Exceeding max shortcut edges from a node at LL = 34.146071,-118.805791
2021/01/12 13:57:29.077628 [INFO] Finished with 162402 shortcuts
2021/01/12 13:57:29.077668 [INFO] Creating shortcuts on level 0
2021/01/12 13:57:30.980213 [INFO] Finished with 56338 shortcuts
2021/01/12 13:57:30.980331 [INFO] ElevationBuilder: no elevation data, skipping
2021/01/12 13:57:30.983969 [INFO] Adding Restrictions at level 2
2021/01/12 13:57:40.927549 [INFO] --Forward restrictions added: 1543
2021/01/12 13:57:40.927591 [INFO] --Reverse restrictions added: 777
2021/01/12 13:57:40.928567 [INFO] Adding Restrictions at level 1
2021/01/12 13:57:43.500230 [INFO] --Forward restrictions added: 4090
2021/01/12 13:57:43.500265 [INFO] --Reverse restrictions added: 4545
2021/01/12 13:57:43.500644 [INFO] Adding Restrictions at level 0
2021/01/12 13:57:44.767593 [INFO] --Forward restrictions added: 3376
2021/01/12 13:57:44.767625 [INFO] --Reverse restrictions added: 3648
2021/01/12 13:57:44.767635 [INFO] Finished
2021/01/12 13:57:44.767681 [INFO] Validating, finishing and binning tiles...
2021/01/12 13:57:59.254758 [INFO] Finished
2021/01/12 13:57:59.254856 [INFO] Binning inter-tile edges...
2021/01/12 13:58:02.669194 [INFO] Finished
2021/01/12 13:58:02.669438 [WARN] Possible duplicates at level: 0 = 26
2021/01/12 13:58:02.669481 [WARN] Possible duplicates at level: 1 = 22
2021/01/12 13:58:02.669497 [WARN] Possible duplicates at level: 2 = 4
2021/01/12 13:58:02.685625 [INFO] Cleaning up temporary *.bin files within /valhalla/scripts/valhalla_tiles/
Successfully build files:  /custom_files/california-latest.osm.pbf
Hashing files:  /custom_files/california-latest.osm.pbf
Hashing file: /custom_files/california-latest.osm.pbf
cb3db4d629aa9f12fc45fc20ac39665401960f0f7e1391b8402cf8c7d49f7acc
Found config file. Starting valhalla service!
2021/01/12 13:58:07.594812 [INFO] Tile extract successfully loaded with tile count: 2054
2021/01/12 13:58:07.597138 [WARN] (stat): /data/valhalla/traffic.tar No such file or directory
2021/01/12 13:58:07.597164 [WARN] Traffic tile extract could not be loaded
2021/01/12 13:58:07.606151 [WARN] /data/valhalla/elevation/ currently has no elevation tiles

As you can see, it created a tar file and added hash of /custom_files/california-latest.osm.pbf.

Log on second start 
TEST: /custom_files           
Valid valhalla_tiles.tar found with use_tiles_ignore_pbf: !
Build new valhalla_tiles.tar from available PBF(s).
cb3db4d629aa9f12fc45fc20ac39665401960f0f7e1391b8402cf8c7d49f7acc
Found valid hash for /custom_files/california-latest.osm.pbf!
Scanning old hash cb3db4d629aa9f12fc45fc20ac39665401960f0f7e1391b8402cf8c7d49f7acc
Either valhalla_tiles.tar couldn't be found or new files or file constellations were detected. Rebuilding files:  /custom_files/california-latest.osm.pbf
PBF hashes: 0
PBF file Counter: 1

=========================================================================
= No valid bounding box or elevation parameter set. Skipping elevation! =
=========================================================================

==========================
= Skipping admin regions =
==========================

==========================
= Skipping timezone data =
==========================

==========================
= Using existing config file =
==========================

=========================
= Build the tile files. =
=========================
Running build tiles with: /custom_files/valhalla.json  /custom_files/california-latest.osm.pbf
2021/01/12 14:12:51.725154 [INFO] Start stage = initialize End stage = cleanup
2021/01/12 14:12:51.730031 [INFO] Parsing files for ways: /custom_files/california-latest.osm.pbf
2021/01/12 14:12:51.730388 [INFO] Parsing ways...
2021/01/12 14:14:15.263492 [INFO] invalid_argument thrown for condition (Jun 15-Labor Day)
2021/01/12 14:14:24.139132 [INFO] invalid_argument thrown for condition (Jun 15-Labor Day)

It looks like it decides to build the files because:

PBF hashes: 0
PBF file Counter: 1

, and 0 != 1.

PBF hashes (result of hash_counter()) is 0 because it starts counting from -1.

Unless I'm missing something, it really should start counting from 0. I'll make a PR if that's the case.

nilsnolde commented 3 years ago

Puuh, good question! That's been done by one of our past colleagues and I never really bothered to fully understand that logic.. Did you test a few cases (with & without forcing rebuild) with your changes and it now behaves properly? Then I'd trust it enough to just merge it.

azhi commented 3 years ago

Nope, didn't check other cases. I'm trying to check my own case right now to confirm fix at least works there. If you have a list of cases you want me to check I can run a few builds.

nilsnolde commented 3 years ago

That'd be cool, thanks! (redundant, not sure right now what to test there..) Just tried looking into it, that script needs some decrypting reaaally bad! I'll try your PR myself one of the next days and maybe even come up with smth less cryptic. Hope you can make it work for yourself until then!

azhi commented 3 years ago

UPD: can confirm my fix doesn't work either :).

TEST: /custom_files
Valid valhalla_tiles.tar found with use_tiles_ignore_pbf: !
Build new valhalla_tiles.tar from available PBF(s).
6d83e2e04b33e2ee3d2297803680e225637f79cf6adcbf2eb727b6a326641df3
Found valid hash for /custom_files/arkansas-latest.osm.pbf!
Scanning old hash 6d83e2e04b33e2ee3d2297803680e225637f79cf6adcbf2eb727b6a326641df3
Either valhalla_tiles.tar couldn't be found or new files or file constellations were detected. Rebuilding files:  /custom_files/arkansas-latest.osm.pbf
PBF hashes: 1
PBF file Counter: 1

=========================================================================
= No valid bounding box or elevation parameter set. Skipping elevation! =
=========================================================================

==========================
= Skipping admin regions =
==========================

==========================
= Skipping timezone data =
==========================

==========================
= Using existing config file =
==========================

=========================
= Build the tile files. =
=========================
Running build tiles with: /custom_files/valhalla.json  /custom_files/arkansas-latest.osm.pbf
2021/01/12 15:44:59.591490 [INFO] Start stage = initialize End stage = cleanup
2021/01/12 15:44:59.593129 [INFO] Parsing files for ways: /custom_files/arkansas-latest.osm.pbf
2021/01/12 15:44:59.593230 [INFO] Parsing ways...
PBF hashes: 1
PBF file Counter: 1

, but it still decides to rebuild.

I'll close PR in that case.

nilsnolde commented 3 years ago

can you try use_tiles_ignore_pbf=True in the docker-compose?

nilsnolde commented 3 years ago

I actually always do that with existing valhalla_tiles.tar and didn't have a problem so far when using that env var.. but yeah, can't say how it behaves when it's not set

azhi commented 3 years ago

Yeah, already tried use_tiles_ignore_pbf, it works, thanks. Was just playing with my fix to see if it worked :).

nilsnolde commented 3 years ago

Ok great! I'll leave this open as a reminder to finally clean up a little:)

nilsnolde commented 3 years ago

took ages, but I finally had some time to clean up and this works now too!