go-spatial / tegola

Tegola is a Mapbox Vector Tile server written in Go
http://tegola.io/
MIT License
1.25k stars 193 forks source link

Preventing errors when reseeding tiles while server is running #888

Closed daawaan4U closed 1 year ago

daawaan4U commented 1 year ago

We currently store property boundaries in our database and when a row gets added, it immediately triggers a seed --overwrite on the bounding box of the new geometry. Our concern however is are there any mechanisms present in tegola to handle serving files while they are currently being overwritten (e.g. obtaining a lock or writing to disk on a temporary folder then performing a mv to atomically and instantly overwrite the existing file) ?

Edit: Checked the source code and it appears that tegola actually writes in a temporary path first then later performs a rename. Keeping this issue open in case there are other edge cases to consider when reseeding tiles while server is running

ARolek commented 1 year ago

@daawaan4U looks like you found the implementation for our file cache. For our [S3 cache], the operation does not appear to be atomic. This could be refactored to write the object to S3 with a UUID then issue a rename. That would be a nice durability improvement.

iwpnd commented 1 year ago

Updates to a single key are atomic. For example, if you make a PUT request to an existing key from one thread and perform a GET request on the same key from a second thread concurrently, you will get either the old data or the new data, but never partial or corrupt data.

see Amazon S3 data consistency model

ARolek commented 1 year ago

@iwpnd thanks for that reference! closing this issue ;-)