opentopomap.org frequently down

[megies]

I've recently switched some of our services to use opentopomap.org for map backgrounds. Unfortunately, it turns out that opentopomap.org is quite frequently down. Are there any problems with the web hosting that the website maintainers are not aware of? Is there a chance to fix the website availability percentage?

EDIT: Stability has been real good from August 2019 as of January 2020

CC @jwassermann @egdorf

[lukey78]

I also use Opentopomap for a project (in development) and try to conquer the outages with excessive caching on my side (with Mapproxy & Nginx). Are there any mirrors or other public Opentopomap server available?

How complicated/easy is it to setup a server? Is there a docker image / buildfile available yet? How much HD space is required?

I would love to run an additional server if the installation is not that much of a pain. For 15-30 € a month I could rent a server with a lot HD space and enough power. Maybe if 2 or 3 people can contribute 5 to 10 € a month for a reliable service that would be great.

Anyone interested?

[sletuffe]

Hi,

Are there any mirrors or other public Opentopomap server available?

None that I know of (yet !)

How complicated/easy is it to setup a server?

Solid linux knowleges are required The howto is here : https://github.com/der-stefan/OpenTopoMap/tree/master/mapnik

Is there a docker image / buildfile available yet?

Don't think so

How much HD space is required?

1 TB SSD for planet wide is a minimum

For 15-30 € a month I could rent a server with a lot HD space and enough power.

I doubt that'd be enough (unless you know some really low cost provider !) if you are interested in planet wide, regular (less than an hour) updates and are going for a bit of tiles served, 4 cores+32GB RAM is needed

[lukey78]

The server just depends on how many people are willing to contribute a small amount. There are server providers in Germany where you can get a good enough server. But of course I don't know about the load, and load balancing could be done with the official Opentopo servers. Currently I think the a/b/c servers are all the same, at least it's the same IP, and if one is down, all seem to be down. I too don't think that so many people use Opentopomap yet.

I think I can install this thing and create a docker image. At least I'd try to if some people are interested.

When there's a docker image available, even more people would be able to just spin up a server. I did that recently with my own mirror of the OSM Overpass API.

[megies]

Maybe whatever consortium/construct is hosting openstreetmap could be asked?

[der-stefan]

The consortium/construct is just me. The server is a dedicated machine at the University of Erlangen-Nuremberg and is quite powerful (64 GB RAM, 1.1TB SSD for postgresql database, 3 TB HDD for rendered tiles). In the last month 11 TB tiles were served, so the load is quite heavy. Since I currently don't have the free time due to job/family, the server sometimes is unsupervised for weeks and I am happy that it still runs ok. Running an own OTM render server is not that simple (and requires a respective hardware). But if somebody wants to run a tile mirror server (synchronized with our main renderer), this would be great and reduce downtimes, if clients try a|b|c.tile.opentopomap.org accordingly.

[lukey78]

@der-stefan Do I understand you correctly, is it possible to "just" mirror the generated tiles from your server (maybe through an rsync job) and serve them through some service? Maybe that'd be the simpler solution, and would need much less CPU and memory, but enough HD space.

Are all the tiles already rendered, or are they rendered on request?

[megies]

The consortium/construct is just me.

Yeah, that's what I guessed already. I was on about the OSM construct and if it might be worth a thought to let them do the hosting, since it's quite a strain, but I have no idea about how that's organised and all..

[der-stefan]

@lukey78 The tiles are served from the cache and rendered, if they are missing on HDD. A few years ago, the database was synchronized with OpenStreetMap every minute ("minutely diffs") - old tiles were re-rendered and if the rendering took too long, the old cached tile was delivered. Since the OSM planet file consumes too much space nowadays, minutely diffs are not enabled anymore. I occasionally re-import the full planet file (requires 3-5 days).

Tirex has a built in feature to synchronize tile mirrors. I never tried it out, but am willing to, if someone organizes and operates a tile mirror server. :-)

[lukey78]

@der-stefan Is "Tirex" this one: https://t-rex.tileserver.ch/ ? Do you think it's possible to try a relatively quick proof of concept, maybe by mirroring only a specific region/country? Then I'd be in to try. If the POC proofs that it's working, we can think about getting some people to invest in a server.

[hottrails]

For tirex see https://github.com/openstreetmap/tirex and https://wiki.openstreetmap.org/wiki/Tirex

[lukey78]

Another very easy solution would be to setup a public caching proxy server (e.g. nginx or squid) with the current opentopomap server as a backend. Caching time could be defined as well as max hd usage, there's automatic fallback to the upstream if a tile is not available in the cache, and setup is very easy.

It'd of course take some time to fill the cache, but maybe this can be automated by some scripts.

A caching server like this just needs a big HD, but not so much memory and CPU, and you can setup a lot of them in a short time.

What do you think? Am I completely wrong? Tiles are served by Opentopomap with specific URIs and a simple HTTP GET request. So this should work.

[lukey78]

3 caching servers for a/b/c in front of the current backend server would reduce the load significantly and could act as fallbacks for an outage of the backend.

[der-stefan]

The a.tile server could be the render server itself, only b.tile and c.tile could be external tile cache servers. For worldwide coverage (zoom 1...17), 3TB, >100 MBit/s uplink and unlimited traffic (~1...10TB/month) should be available.

[lukey78]

Ok. A caching proxy server (no rendering by itself) with 3-4 TB HD and good uplink would start at about 60 € / month or 720 € / year.

@der-stefan Would it be OK for you if I'd start a crowdfunding campaign in Germany and international to raise some money for server operation for 1/2 years? If enough people contribute, I'd operate, install and monitor the server(s). I'd also invest a good amount by myself if I'd make some money with my websites -- unfortunately that's not the case yet :-)

[sletuffe]

All that fallow is my humble opinion with the few experience I have operating a tile server https://wiki.openstreetmap.org/wiki/Hiking/mri

You'r obviously free to choose the cache/CDN path if you like,

And it's sexy and all because any squid or varnish with 1 or 2 days work will do along with a rather cheap server, but it won't solve your problem imho. Worst, it will add another point of failure + consume time that might much better be invested elsewhere : Like, in a rendering failover/load balancer ;-)

Any problem related to load on a osm2pgsql/postgres/mapnik/renderd/mod_tile rendering infrastructure is not at the tile delivery step but at the tile generation step with renderd+postgresql eating up all the ressources. The mod_tile reading and http sending process is almost negligible in this setup. At best you will have a caching backup failover with missing 404 tiles at random making user experience crappy. But you will unlikely offload the tile server.

Given the tile.openstreetmap.org history, even if I can only guess the reasons, they first choose a caching cdn strategy, but my guess is it was to reduce bandwidth at first. And when they discovered the load and queue length did not budge, they added new rendering nodes.

That beeing said, I don't have a solution to offer now to build/operate that failover worldwide rendering server. But it's been a few month that I've been looking at the opentopomap style to replace my own with a few custom additions. But my target is not and cannot be (lack of SSD) planet wide+minute updates but only Europe.

[lukey78]

The biggest problem at the moment is the single point of failure, and actually the server isn't available at some times. Of course you can offload that risk by adding caching proxies. In the worst case they can't deliver some tiles because they are not cached. And no, there's not another point of failure, because you as a user should also implement your own load balancing to the a/b/c servers if you want to benefit from the failover capabilities. At the moment you don't have that option because a, b, and c are the same server.

No one said that the server load is a problem.

[lukey78]

Ah... no one? I really like to have a reliable mapping service with free topographic maps. Opentopomap is great for that, and I'm willing to contribute even money for this, but I can't afford the whole server.

Actually, I also tried some seeding to fill my own proxy with one country, but of course the official server seems to be rate limited. So, without the help of the server operator it's not possible to setup a separate caching server.

Maybe we can find a solution for that. I won't give up on this.

[der-stefan]

@lukey78 That's quite normal, we won't find too many people who invest the necessary time and money... Fortunately, voluntary service is still widespread in the real world (not the internet). I had to accept that after starting the OTM - but sometimes the right person steps in after years! :-)

[lukey78]

Sadly, I didn't receive any reply, even by personal mails, from the OpenTopomap team. At the moment the server seems to be quite loaded, I often run into timeouts with my application. I try to cache the tiles on my server but the implemented rate limiting is blocking me all the time because all requests then come from the same IP.

It seems I have no other chance than trying to setup my own Opentopo server because I really need a reliable service and I think the generated maps from this project are the best topographic maps from free sources that you can get - all the other commercial alternatives are not that great.

I'll start soon and try go the docker way. Once we have a docker file to setup a full server it'd be MUCH easier for other contributors to setup their own server, or even setup new public servers.

For failover, I'd still offer to create a caching tile mirror, because that'd be really simple and not that expensive. But that'd be only possible, too, if the server operator of opentopomap.org removes the rate limiting for the mirror.

If someone wants to help or contribute in any way: Please contact me. Thanks.

[der-stefan]

Hi lukey78,

if we find out how the parameter sync_to_host in tirex.conf works, we could easily add a tile mirror. See https://wiki.openstreetmap.org/wiki/Tirex/Config.

If you give me a SSH login for scp-ing the currently rendered tiles, I would copy all of them to your server right now.

[lukey78]

Thanks, coming back to you.

The problem at the moment is the large HDD requirement (3 TB) for a full mirror - and I guess most of the tiles are never requested because they belong to parts of the world where no one is going to, or are just water :-).

A server with such a large disk is about 100 € / month, and I can't afford that alone :-(. With another 8-9 users willing to pay each 10€/month for the mirror, I'd happily operate the server.

[febs]

I also use Opentopomap for a project (in development) and try to conquer the outages with excessive caching on my side (with Mapproxy & Nginx).

I'm afraid this is not about contributing to your server, but would you mind sharing your mapproxy configuration file? I'd also like to set my local proxy, so to spare as much bandwidth as possible from the official server. Many thanks if you won't mind sharing. Cheers!

[mboeringa]

A server with such a large disk is about 100 € / month, and I can't afford that alone :-(. With another 8-9 users willing to pay each 10€/month for the mirror, I'd happily operate the server.

@lukey78 Well, don't know if they are given yet, but the OpenStreetMap Operations team recently had 4 retired dual six core servers on offer:

https://twitter.com/OSM_Tech/status/1125676280840630273

No 3TB disks, but at least you'd have some hardware to work and start with for free (at least, that is what I am guessing from the post). OTM might be a nice second life ;-)

[lukey78]

No 3TB disks, but at least you'd have some hardware to work and start with for free

@mboeringa Thanks for the link, but that wouldn't help much. Even if you have servers you still need a datacenter to operate them, and then you're fully responsible for the hardware and you need to have it near you to be able to actually go there for maintenance. I'd never ever again operate my own hardware in a world where hardware becomes less important with each day passing :-)

[crispe]

Firstly a big thanks to Stefan for such a great tile set. The renderer has been down for several days again, not sure if Stefan is aware of it. Also, I've been trying to accumulate as many cached tiles as possible however my server keeps getting throttled so can only get 1 tile per second.

[yvecai]

There is a modtile config for not throttling proxies: ModTileEnableTileThrottlingXForward

0 - don't use X-Forward-For and allways use the IP that apache sees 1 - use the client IP address, i.e. the first entry in the X-Forwarded-For list. This works through a cascade of proxies. However, as the X-Forwarded-For is written by the client this is open to manipulation and can be used to circumvent the throttling 2 - use the last specified IP in the X-Forwarded-For list. If you know all requests come through a reverse proxy that adds an X-Forwarded-For header, you can trust this IP to be the IP the reverse proxy saw for the request

[sletuffe]

Hi there,

As said up there, I'm also a big fan of the OTM rendering and using it on a project as an alternative rendering. (Main one being https://wiki.openstreetmap.org/wiki/Hiking/mri )

Having a backup service to fail over to would be nice for me... so I've done it ! Here it is : https://maps.refuges.info/?layers=0B

But it's far from perfect and unlikely as is to fit for a load balancer of the main OTM server. The main reason being that the server is not beefy enough. (I'm lacking SSD/NVME drives which almost are a "must have" for such a setup.) But I'll be glad to move to a more powerful server if a solution of availability arise (who knows, there might be an happy donator to fund a public OTM load balancing server in the room !).

As of now, it can be used as is with this usage policy https://wiki.openstreetmap.org/wiki/Hiking/mri#Usage_policy but more likely in a slow, last resort "fail over" to the more powerful OTM server.

Beside the server, the current solution for updating the OSM datas is also of concern because it is a long process and is to be improved in the scripts to avoid downtime (using temporary tables for both low and high zoom tables ?)

In the long term, I'll be happy to add a few improvements to the style as well.

For devs, here is the repo of my changes : https://github.com/RefugesInfo/OpenTopoMap/tree/sly-version Note that I'm using Tirex instead of renderd shown in the README

And I'll add a big thanks to @yvecai who sent me a copy of Digital Elevation Model files + hillshaded terrain + contours lines for several Gigabytes

I'll also be happy to share tools, ideas and tips to improve performances or availability with others operating an OTM rendering service. Like how to compute peak dominance or saddle orientation in less than 30 hours... How to improve holes in the DEM : https://opentopomap.org/#map=13/63.04862/11.92223

[megies]

As a side note, I noticed that the main opentopomap.org has very much improved in stability since last August. Since then, downtimes have been very few. Thanks @der-stefan!

[gjedeer]

If mirrors are needed, I've made a quick Docker image with a fork of Cyclestreets Tilecache that allows hosting an OpenTopoMap mirror with maps.refuges.info fallback with just two commands.

If there's a definite statement that such a mirror would help, I'll look into hosting one myself and we'll see if I can sustain it financially.

[sletuffe]

Hi, Looks like tile downloading from OTM is very slow those last days. Is there some temporary problem ?

And among all those docker rendering project, new installation, is there some publicly active mirror ?

[gjedeer]

http://maps.refuges.info/ seems to work pretty well.

[sletuffe]

In the mean time, things have improved and response time from the OTM server is good once again.

http://maps.refuges.info/ seems to work pretty well.

It is indeed working, but lack ressources (no SSD, no NVMe) to act as a proper mirror/backup for OTM. It is mainly a demonstrator or last resort backup. And without SSD/NVMe drives I cannot keep the database up to date (Too I/O intensive).

[gjedeer]

Regarding your previous question. I never heard form anyone in the project (opentopomap) about my offer to run a tile cache mirror. So I haven't ran one.

Regarding my answer about your own mirror. That was a silly answer I thought you're a different person. Check your email please.

[crispe]

Hi Stefan I must have overlooked you previous email, so I went to check it out, however the colours seem to be a little different, see example bellow: 6119 PNG Image · 27 KB

6119 PNG Image · 25 KB

Since over time downloaded over 88 million tiles I would have to invalidate them all.

Once again, thank you very much for such a fantastic tile set.

Best regards Cristiano

On 15 Feb 2023, at 18:59, GDR! @.***> wrote:

Regarding your previous question. I never heard form anyone in the project (opentopomap) about my offer to run a tile cache mirror. So I haven't ran one.

Regarding my answer about your own mirror. That was a silly answer I thought you're a different person. Check your email please.

— Reply to this email directly, view it on GitHub https://github.com/der-stefan/OpenTopoMap/issues/217#issuecomment-1431860514 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQU2CQZUP3LVOTJW63FEVLWXURPVANCNFSM4GZFIDHA . You are receiving this because you commented. https://github.com/notifications/beacon/ABQU2CUGEQAJS2SIPYGZQBTWXURPVA5CNFSM4GZFIDHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOKVMHKIQ.gif Message ID: @.***>

[hyperknot]

Hi,

I've just wanted to contribute the proxy solution I've been using on MapHub for 6 years now. It's an nginx config and is blazing fast.

You can tune the values as you like, but basically it makes sure that if a tile is accessed at least twice, then it's served from nginx's static file cache. A simple 100 GB cache can cover the full world in my case, as it only caches titles which people actually view somewhat frequently. Nginx is really good in this.

Here is the full config adapted for OpenTopoMap upstream servers.

server {
    listen 80;

    server_name opentopomap-proxy.maphub.net;

    access_log off;
    error_log /data/opentopomap_proxy/logs/nginx-error.log;

    location / {
        proxy_cache_key "$request_uri";

        proxy_cache opentopomap_proxy_cache;
        proxy_cache_lock on;
        proxy_cache_min_uses 2;

        proxy_ignore_headers Cache-Control expires;
        add_header Cache-Control public;
        add_header Access-Control-Allow-Origin *;

        expires 1M;
        proxy_cache_valid any 1M;

        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_redirect off;

        proxy_pass http://opentopomap_proxy_servers;
    }
}

I think you could solve the load on your server by adding this config in front of it. If run on a separate computer this really doesn't need a lot of CPU or memory, just a decent SSD drive.

[hyperknot]

Also, using this config you can allow other members of the community to run their own proxies, thus distributing the load even more.

[IPSTECH2020]

Hi Stefan, I sent you one email today to offer VM (compute) with the load and traffic. Please check my email. Thx, Serginho.

[Rosaweb]

Hello,

sorry to interrupt the technical conversation. As an outdoor enthusiast, thru hiker, bushcrafter, canoeist and small outdoor youtuber this conversation looks unreal to me. Open Topo Map is the only free and decent topographic map option worldwide available online. It has a huge impact on my outdoor practice as for most of guys like me, and I do believe that if you would emphasize more on a social and community dimension, not with developers but with actual users, raising 10 or 20000 euros would be doable.

For exemple I made a donation to gpx.studio which integrates open topo map and has 1800 followers on ko-fi donation plateform but one can't find anything for open topo map on the internet.

I'm using open topo map almost every week and I'm desperate seeing it working properly and not lagging. It is so unreliable that I use to create offline map for preventing any problem on the terrain despite I'm in Germany with 5G coverage. If you would create an Open topo map web page with an official crowdfunding campaign or donation supporting program I would definitely speak about it on my (very small) outdoor youtube channel and I have no doubt that many other outdoor/hiking youtubers would be more than happy to do the same.

As users we need and wait for a big technical jump with OpenTopoMap in order that it becomes THE outdoor map service we are all dreaming about. It has the potential for but the conversation here doesn't seem to aim for it.

Thanks for reading this, best regards. youtube@baroudeuraventure

[pierrepoulpe]

Hello @der-stefan

3 TB HDD for rendered tiles). In the last month 11 TB tiles were served, Are these numbers still valid? I imagine monthly traffic increased. 3TB, does that include all zoom level? or the last one are rendered on the fly?

Like many, I love openstreetmap, I actually depend on it! So I was thinking how to make a mirror server. At first for my own usage, but also to support a growing community of enthusiasts.

1/ I'm working on mounting a VM according to the README, to become familiar with all the processes. I started with the most recent ubuntu version, and struggle with phyghtmap. I'll probably try to stick with the documented version 16.04 or 18.04.

2/ While I'm absolutely not a specialist, I look into current offers of the cloud/hosting market, to find the sweet spot cost/performance/easy. On the serving side, I would like to explore the object storage way. More traffic would be transparent. On the processing side, I have an offer that sounds difficult to beat : 4core/32GB/2TB disk for 17€/month (ovh kimsufi)