search

tilezen / go-tilepacks

A Go-based tile downloader that saves to deduplicated mbtiles files.
MIT License
14 stars 3 forks source link

Try requesting gzipped tiles. #2

Closed zerebubuth closed 5 years ago

zerebubuth commented 5 years ago

NOTE: not sure this is ready for merge - issue below about CloudFront returning identity encoded responses needs more thought than I've given it.

This should have a couple of benefits:

  1. Lower CPU usage in tilepacks, as we don't need to decompress the response.
  2. Smaller mbtiles using compressed tiles.

However, there's a small issue; it seems that if CloudFront has a non-compressed tile in cache then it will return the tile without compressing it rather than fetch the compressed version from the origin. This means that we get back some uncompressed bytes and might need to compress them, but we don't know what the compression level is to match the settings on the origin, which might mean we end up not de-duplicating the tile when we could.

iandees commented 5 years ago

I merged this in 9edf43f.

I did some debugging on the compression from CloudFront. It looks like CloudFront doesn't compress application/x-protobuf Content-Type and we weren't passing the Accept-Encoding header from the request to the backend, so nothing was getting compressed. Since the gzipping was giving a really good storage gain, I added in a client-side compression step if the server didn't do it for us.