Merge the generated filtered.o5m.pbf and filtered_names.o5m.pbf to one pbf, to increase performance

alfh commented 10 months ago

Is your feature request related to a problem? Please describe.

Looking at the performance for generating the map for countries with hundreds or thousands of tiles, the "split_filtered_country_files_to_tiles" function takes a really long time.

One reason is that for each tile, the pbf file needs to be read three times by the osmium extract tool. And when the file is gigabytes in size this takes some times.

And since currently, for each tile, the osmium extract command is launched for two large files, the filtered.o5m.pbf and filtered_names.o5m.pbf, this really adds up to the time for each tile.

Describe the solution you'd like

I suggest that the two filtered files generated by the "osmium tags-filter" command, is merged to one file by using osmium merge, and that this merged file is then really used for the subsequent steps. So some more time might actually been saved where the merge of tile files are currently being done, since there will be one less tile file to merge in "merge_splitted_tiles_with_land_and_sea".

At least I wonder if that has been considered ?

I also wonder if it is really necessary to generate two different files with the "tags-filter" or if this could be done in one command. I have just tested the performance for doing the osmium extract on one merged file versus two separate files.

So I have actually not generated the map and verified that the map ends up being correct. I wanted input if you think this approach is useful, before spending more time on it. And looking for input how to verify that the generated tiles are identical.

Of course, I appreciate that it takes some time to merge the two files, (9 seconds for Norway), but for any contries with more than 10 tiles, I think this would really improve the overall performance.

Describe alternatives you've considered

I am also looking at some other radical change, which is to create one large json file, and pass to "osmium extract", so extracting all tiles with one command, instead of one tile at the time. The memory usage for that is currently large, but I'm looking at potential changes in osmium to really help there.

I hope to raise another feature request to discuss that shortly. (Doing Canada in less than one hour, instead of 38 hours). But note that even that case really would enjoy a big saving for just needing to process one file (1111 seconds for all of Norway, instead of 1521)

Additional context

File sizes for Norway, which I have used for testing 748207171 alf_merged.o5m.pbf 670306916 filtered_names.o5m.pbf 418104255 filtered.o5m.pbf

One performance measurement on linux, using the "linux perf" tool. Look at the "seconds elapsed", that is the "wall time", the other are CPU times (multi core cpu). So it is 3.5 seconds for the merged file, while the sum of (2.4+3.1) the two files are 5.5. With over 1100 tiles for Norway, that saving really adds up. And for Canada, with 4900 tiles, where each tile takes 26-28 seconds when doing the files, the potential is also large there.

For the merged file 153 939 313 085 cpu-cycles
283 315 190 337 instructions

   3,547681733 seconds time elapsed

  28,106475000 seconds user
   4,872589000 seconds sys

For the filtered file 85 521 022 047 cpu-cycles
157 321 282 847 instructions

   2,386457164 seconds time elapsed

  15,310888000 seconds user
   3,216435000 seconds sys

For the filtered_names file 140 564 335 570 cpu-cycles
256 643 420 752 instructions

   3,128506510 seconds time elapsed

  25,682144000 seconds user
   4,122075000 seconds sys

Looking at the number of CPU instructions is mainly for "fun", amazing how fast the CPUs are nowadays.

treee111 commented 10 months ago

Thanks for your thoughts! Haven't read through it fully but some coments on this.

At least I wonder if that has been considered ?

I haven't thought about this yet since I had no issue with the time it needs until now. But for larger countries there might be a significant impact. Adding your PR #207 ith printing out the taken time should be done before to be able to measure the performance gains.

I also wonder if it is really necessary to generate two different files with the "tags-filter" or if this could be done in one command. I have just tested the performance for doing the osmium extract on one merged file versus two separate files.

I had the same question, see #109 but in the end relied on @Ebe66 because he somehow is the creator of the base structure in generating the tiles.

So I have actually not generated the map and verified that the map ends up being correct. I wanted input if you think this approach is useful, before spending more time on it. And looking for input how to verify that the generated tiles are identical.

I have some unittests to validate some parts of the generated files with static input files and against static generated files, see: https://github.com/treee111/wahooMapsCreator/blob/develop/tests/test_generated_files.py. Due to recent changes the unittests are not green. In addition what I wrote at the beginning, this should also be corrected before implementing your new stuff.

As I have limited time at the moment and - as written - there might be some stuff to do I would appreciate some help on this.

Ebe66 commented 4 months ago

Hi, this sounded like a good idea so I tried this with my version of the wahoomapscreator, on windows with osmconvert and osmfilter today with Germany including all sub-regions. It is faster, but just barely unfortunately. My old version with the non merged files took exactly 1 hour, the version that merges the filtered files before splitting took 57 minutes.

The need for two filter stages is described in the issue Treee111 refers to. Is it absolutely needed? No, but it keeps the generated map sizes a lot smaller. There is an indication in your test files, I assume you changed the filters (or there is something else going on with treee111's version on linux c.q. osmium) but your filtered_names.o5m.pbf file is significantly larger than your filtered.o5m.pbf file. The names file should be very much smaller than the no names file. I guess it contains all streetnames for example which would end up in the map file while the Wahoo device does not use them at all. If you compared your map files with mine on wahoomaps.higli.de, you will likely see my maps are smaller.

treee111 / wahooMapsCreator