alfh
commented
9 months ago Thanks for the initial feedback, @treee111. I've read your comments, and will consider them thoroughly.
I've now also gotten the contour/elevation generation to work with asyncio (using pyhgtmap).
I've also briefly tested on Windows (not with contour generation), and only ran into trouble when running the lzma/compression steps, I think it might that compression executable that is using some temp files that collide with other invocation. I'll look more into that.
So now I have a working solution, with nice performance improvements. For Canada, the number looks like this
INFO:--------------------------------------------------------------------------------
INFO:# Filter tags from country osm.pbf files
INFO:+ Filtering unwanted map objects out of map of canada
INFO:+ Filter tags from country osm.pbf files: OK, took 81.311 s
INFO:--------------------------------------------------------------------------------
INFO:# Generate land for each coordinate
INFO:start land sea
INFO:start land1
INFO:+ Generate land for each coordinate: OK, took 115.640 s
INFO:--------------------------------------------------------------------------------
INFO:# Generate sea for each coordinate
INFO:+ Generate sea for each coordinate: OK, took 0.173 s
INFO:--------------------------------------------------------------------------------
INFO:# Split filtered country files to tiles
INFO:+ Split filtered country files to tiles: OK, took 34182.288 s
INFO:--------------------------------------------------------------------------------
INFO:# Merge splitted tiles with land, elevation, and sea
INFO:+ Sorted: OK, took 76.932 s
INFO:+ Merge splitted tiles with land, elevation, and sea: OK, took 389.623 s
INFO:--------------------------------------------------------------------------------
INFO:# Creating .map files for tiles
INFO:+ Created .map files for tiles: OK, took 2406.952 s
INFO:+ Compressed .map files for tiles: OK, took 2423.778 s
INFO:+ Creating .map files for tiles: OK, took 2423.880 s
INFO:--------------------------------------------------------------------------------
INFO:# Create: .map.lzma files
INFO:+ Country: canada
INFO:+ Copying .map.lzma tiles to output folders
INFO:+ Create .map.lzma files: OK, took 1.255 s
INFO:--------------------------------------------------------------------------------(No countour, ran out of disk space when testing that, need to look more into it)
For France, the numbers look like this
INFO:+ Filter tags from country osm.pbf files: OK, took 100.156 s
INFO:--------------------------------------------------------------------------------
INFO:# Generate land for each coordinate
INFO:start land sea
INFO:start land1
INFO:+ Generate land for each coordinate: OK, took 2.508 s
INFO:--------------------------------------------------------------------------------
INFO:# Generate sea for each coordinate
INFO:+ Generate sea for each coordinate: OK, took 0.005 s
INFO:--------------------------------------------------------------------------------
INFO:# Generate contour lines for each coordinate
INFO:+ Generate contour lines for each coordinate: OK, took 111.715 s
INFO:--------------------------------------------------------------------------------
INFO:# Split filtered country files to tiles
INFO:+ Split filtered country files to tiles: OK, took 283.197 s
INFO:--------------------------------------------------------------------------------
INFO:# Merge splitted tiles with land, elevation, and sea
INFO:+ Sorted: OK, took 3.409 s
INFO:+ Merge splitted tiles with land, elevation, and sea: OK, took 90.307 s
INFO:--------------------------------------------------------------------------------
INFO:# Creating .map files for tiles
INFO:+ Created .map files for tiles: OK, took 151.222 s
INFO:+ Compressed .map files for tiles: OK, took 173.164 s
INFO:+ Creating .map files for tiles: OK, took 173.167 s
INFO:--------------------------------------------------------------------------------
INFO:# Create: .map.lzma files
INFO:+ Country: france
INFO:+ Copying .map.lzma tiles to output folders
INFO:+ Create .map.lzma files: OK, took 2.018 s
INFO:--------------------------------------------------------------------------------
INFO:# Total time 769.618 sI will now think about how many PRs I will split this into, I think some PRs with "just" code reformatting / restructuring, and then 1-2 final PRs, with introducing the asyncio. I think I will first have all commits in one PR, but with a sequence that makes it easy to split them out into separate PRs.
This PR…
Use Python asyncio library to launch external programs in parallell where possible, to speed up the generation on multi CPU core machines.
Considerations and implementations
This is just a proof of concept as of now, with several hacks, and also not considering Windows to any extent as of now.
So mainly to get some early feedback if possible, on the general approach.
I will add a method to calculate the number of Semaphores, based on "max CPU core used by external program invocation, min memory required by each external program invocation".
Will probably also take parts of this pull request, and make a general "reduce code duplication", and have that PR be merged first.
Also unsure if I should have one method for linux and one for windows for setting up the parameters for each external program invocation case, or one method with if tests for linux and windows. Currently, I have a mix of these, and I think one place I just removed the windows part when I was hacking the very first attempts, will readd the Windows support there in this PR.
I have also commented out the logging for each tile, since unsure how much sense that makes now. Will think about it, it would be useful if something is hanging or causing error.
Also note that is really pays of to have the _tiles/germany/ directory, which contains the two files
put on a RAM disk (dev/shm for example). So I think a PR to add some config options somehow to control where those two files are put, would be really useful. If they are on disk or SSD, you will get a lot of iowait in the filtering step, since each tile is reading those files then.
I see a 10 fold increase in performance on some steps when looking at Norway, when running on a 16 core CPU (32 threads), 64 GB RAM.
Here is the output for Germany on my machine with asyncio
Here is when running the upstream develop code, not using asyncio (just showing the main parts of the log)
So a 3x increase in performance for Germany.
{...}
How to test
Invoke as normal, but as of now manually adjust the Semphore settings, which control how many parallel external programs are running.
Pull Request Checklist