search

go-spatial / tegola

Tegola is a Mapbox Vector Tile server written in Go
http://tegola.io/
MIT License
1.29k stars 196 forks source link

S3 cache - Improving purge speed #722

Open carronol opened 4 years ago

carronol commented 4 years ago

Hello! First of all, congrats for the good work! Tegola is really easy to set up and use and does the expected job 👍

The only drawback I find in our usage is the speed of the cache purge command. It seems that it purges the tiles one by one in the specified bounds and zoom levels and that it could be optimised. This begins to be a problem for us because part of our data can change every day so we tend to use the purge feature a lot.

In our case, we are using a S3 Bucket for our cache storage and I found that the aws s3api command line enables to delete batches of files (tiles in this case) as documented here : https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/delete-objects.html

So I was wondering if the implementation of the purge command for the S3 cache could take advantage of this to speed up the process, by batching the tiles deletion.

My workaround would be to use the aws command myself but I would need the list of tiles to purge and I can't obtain it in a simple way.

I hope my explanation is clear and I'm happy to provide more details if needed.

ARolek commented 4 years ago

@carronol I'm glad you're enjoying tegola!

That's an interesting idea around the purge command, and I'm sure the purge interface could be upgraded to a BulkPurger though I would need to give it some thought. Would you want to tackle this issue and send in a pull request?

My workaround would be to use the aws command myself but I would need the list of tiles to purge and I can't obtain it in a simple way.

This is an interesting idea too. The purge command could potentially be enhanced to generate a tile list as well. Are you generally working with bounds + zome ranges, or purging entire zooms (i.e. all of zoom 13 and 14)?

carronol commented 4 years ago

Thank you for the quick answer @ARolek

I have never coded in Go and I would require some time to get into the project in order to create a pull request, but we will give it some thought at work. If we decide to tackle the issue it won't be in the next weeks, but maybe in a sprint to come.

Concerning your second question, we usually modify a few geometries at a time in a map so we purge and seed the cache locally (with bounds and zoom ranges to purge all zooms and seed only lower zooms). But because of the way tiles are purged, I sometimes decide to "manually" purge the entire map (the directory when I was using the file system and now with a recursive rm command with the path of the map to clean in our S3 bucket).