OTM Docker Image

[lukey78]

Hi, I've just finished my Docker implementation of OpenTopoMap. It's based on the config files in this repository, fixes and updates some stuff that changed in the last years, and it includes an easy to following step-by-step setup guide that is almost fully automated.

Using a small country like Luxembourg as an example, you can setup the server on your local machine in 30 minutes.

https://github.com/lukey78/otm-docker

Please test the image and the steps described in the README. If someone's able to extend the image to allow minutely updates to the database: Go on!

Thanks for your feedback and suggestions.

[sletuffe]

Hi, Nice to read that you've manage to reach your goal at creating a docker image. On my side I've decided to set up an additional instance of OTM as well, but not going the docker way. Still, I'll be happy to share feedbacks and improvements :

• Is there a live instance (Possibly viewable ?) of you OTM docker image covering Luxembourg, or, even better covering the world ?
• Have you kept all procedures of the README as is, or have you tweaked things in case of re-import (for the lowzoom db or the main tables ) ?
• Have you generated successfully raster images ?

[lukey78]

For testing I could setup a small country like Luxembourg in a short amount of time. To make everything work - especially permissions - I had to tweak some of the files. There are also some merge requests open for Opentopomap that must be used to make it work (e.g. the new water polygon files or using "bicubic" instead of "bilinear" for creating the hillshades).

The real problem lies in creating a server for the whole planet. The most problematic issue is generating the contour lines, because phyghtmap crashes with big bad errors for larger files.

Currently I don't have a demo server running, but I managed to set this thing up locally with docker on a normal MacBook Pro for a small country including hillshade and contours. There is a small Leaflet demo server included to verify that all is running.

[sletuffe]

Thanks Jens and Lukey for you answers. I also ran into issues with the gdal process (one is a North-South shift of 10km or so due to a gdal version) I also simplified things by reducing the number of tif files without significant increase in generation time.

All those little improvements or alternatives could probably be pushed as PR for stefan to update or simplify the setup process and documentation. The contours line generation is also a long process (I dropped phyghtmap in favor or gdal_contours, but ultimately asked fro @yvecai to send me ~350GB of allready processed shapefile, that saved me a lot of time)

I also got stuck on peak and volcano isolation : https://github.com/der-stefan/OpenTopoMap/blob/master/mapnik/README.md#edit-update_saddlessh There are ~1M of such poi in OSM, and on rotating disks it takes me around ~1s / peak wich mean it would take around 10 days ! Don't know if my raw DEM file is responsible or disks...

Last but not least, the opportunity to minutely update the rendering with fresh OSM datas seams to be out of reach in my setup. The lowzoom processing is ~10 hours, full import 15 days. Even with SSD drives I don't know how the lowzoom could be kept up to date.

A yearly update seams more doable, but the OSM data upgrade isn't documented beside redoing the import that whould need to offline the rendering for several hours if not days...

[max-dn]

Since a few months we have been trying a regular update on the Opentopomap server. Every week a differential update is applied, twice a month a lowzoom update.

This means that in the time between diff and lowzoom, tiles are rendered that are only half up-to-date. A lake could have been deleted and still the name could be displayed.

On the server, a differential update takes about 2 days for a weekly diff, lowzoom alone takes 7 hours if there is not much else going on on the server. (peaks takes 30min)

For lowzoom it was important to give the postgres a lot of memory (shared_buffers = 2GB, temp_buffers = 8GB, work_mem = 12GB), so that less cache is swapped out to disk. I think everything could go much faster if we didn't have tilecache and database indexes on the same disk. Unfortunately, this cannot be changed.

The whole thing is still in development, only half-documented, but I could summarize it and create a PR...

[sletuffe]

Thanks max for the insight,

It is interesting to know what type of config OTM is currently running on, I guess the : https://github.com/der-stefan/OpenTopoMap/blob/master/mapnik/README.md about server ressources isn't up to date anymore ? Could you provide a bit more info on drives, memory or CPU ?

Also, even if half-baked, scripts or documentation about your current update process would be nice to compare strategies. What's your dirty metatiles expiry method in your setup ? To conquer your half up-to-date 7 hours of lowzoom update+2 days long update have you considered a full re-import and table moves ? How long is a full osm2pgsql import ?

When you say peak dominance calculation of 30mn is it an update or a full calculation ? The script having a where otm_isolation IS NULL or otm_isolation NOT SIMILAR TO '[0-9]+' An update should run much faster than a full computation.

[max-dn]

Quick and dirty, at your own risk ;) https://github.com/max-dn/OpenTopoMap/tree/server-stuff/mapnik/server-stuff

at the moment expire_tiles reads the output of osm2pgsql and set the date of tiles in Z15-Z17 to "very old" and all others to "3 weeks old". In the backround we render all tiles older than 3 weeks, alls files older than 4 weeks are rendered fresh when they are read by a browser. So expired Z0-14 will be rendered next in backround, Z15-Z17 are rendered at once.

I'm not very happy with expiry. It's a lot of work for the server and most of the changes trigger a new rendering while nobody sees the changes (eg a new restaurant is not shown in OTM). All tiles are rendered after some weeks so we don't have to prefer any newly changed tiles. Z16 and Z17 seldom get older than some weeks, because they are deleted when the disk space for the cache gets full.

I dont know how the last full import in september 2019 was. Think it was 6-8 days.

Afrt differential updates only peaks with "otm_isolation IS NULL" are calculated (2 min), after lowzoom all peaks are updated (30 min). That's necessary from time to time, because we trust mappers more than SRTM: A wrongly mapped peak with ele=5000 next to the Mont Blanc will reduce the isolation of the Mont Blanc to the distance between these two peaks. When the wrong peak is corrected, you have to recalculate all his neighbors, not only the newly changed peak.