search

maptiler / tileserver-gl

Vector and raster maps with GL styles. Server side rendering by MapLibre GL Native. Map tile server for MapLibre GL JS, Android, iOS, Leaflet, OpenLayers, GIS via WMTS, etc.
https://maptiler-tileserver.readthedocs.io/
Other
2.11k stars 614 forks source link

Serve robots.txt file #947

Open Empty2k12 opened 10 months ago

Empty2k12 commented 10 months ago

I am running a tileserver-gl instance on the internet and have recently noticed that Google has started indexing multiple files from my tileserver. This is not my intention, and it seems like it's not super easy to configure a robots.txt with this project. I would have to run a second Docker container which runs nginx or some other file-server and configure my reverse proxy to serve it when tileserver.my.tld/robots.txt is requested. This is quite cumbersome.

The simplest fix would be to serve a robots.txt when it's available somewhere (configurable) in the file tree, and return a 404 on the robots.txt route if it's not found. This way, you could easily mount a read-only robots.txt into the container.

acalcutt commented 10 months ago

I wonder if anyone considers google indexing their tile server a good thing, should we just put a default robots.txt in "public/resources" which gets served as root of the site right now?

As for a configurable option, i'm not really sure the best way to approach it.

Empty2k12 commented 10 months ago

@acalcutt Completely blocking indexing is also fine in my opinion, but I am not sure if there are cases where people would want their tile server to be indexed. But we could just serve a noindex robots.txt and have people override that using their reverse proxy setup should indexing be wanted.

yetzt commented 1 month ago

i always liked googling for intitle:"TileServer GL - Server for vector and raster maps with GL styles" and seeing what maps people make. ;)