Releasing 0.12.1 with S3 cache support

[pka]

@arbakker I'm wondering whether we could release 0.12.1 with the S3 cache support you contributed.

I would like to have it tested (and ideally also documented) with Amazon AWS S3 storage and then publish a t-rex release ASAP. When using t_rex generate to seed an S3 cache, we should be able to use it directly with the AWS S3 URLs. Maybe that's already the case and the generated TileJSON file contains the public S3 URL when baseurl is properly configured?

[arbakker]

I did not consider this usecase of using the direct S3 urls when implementing it, but it seems like a no-brainer.

I do not think it will work just yet, at least the following points needs to be addressed:

• setting Content-Encoding: gzip headers on the mvt files. My proposal is to make this optional, but enabled by default. We experienced failing backups with Minio/S3 with the Content-Encoding headers, so would like the option to diasble them.
• adding the key_prefix to the baseurl to advertise the right url in TileJSON document

I will take a look this week to fix these issues and run a test.

[arbakker]

Finally had some time to work on it today, see this commit.

Added the following options to the s3 cache config:

• public_read_enabled (defaults to false)
• gzip_header_enabled (default to true) [maybe rename to ce_header_enabled]

When running with the following config:

baseurl = "https://trex-s3-test.ams3.digitaloceanspaces.com/trex"
endpoint = "https://trex-s3-test.ams3.digitaloceanspaces.com"
bucket = "trex"
access_key = "{{env.S3_ACCESS_KEY}}"
secret_key = "{{env.S3_SECRET_KEY}}"
region = "my-region"
gzip_header_enabled = true
public_read_enabled = true

The vector tiles are generated with the correct permission and content-encoding headers. See the following code sandbox (need to open the app in a new window, does not seem to work in the editor environment for reasons unknown to me).

Still want to add some tests to check if the headers are set on the S3 objects (depending on the config) before creating a PR.

[pka]

Had a look at the sandbox application. When opening https://q2d5d.csb.app/, I get

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://trex-s3-test.ams3.digitaloceanspaces.com/trex/gemeenten/6/33/21.pbf. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing).

Looks like missing CORS headers?

[arbakker]

Oops, I did some testing and forgot to ensure the sandbox application was still working. It is now working again.

[pka]

Looks good!

I also tried to open the tileset in Maputnik with https://maputnik.github.io/editor/?style=https://trex-s3-test.ams3.digitaloceanspaces.com/ne_110m_admin_0_countries.style.json but the source url in https://trex-s3-test.ams3.digitaloceanspaces.com/ne_110m_admin_0_countries.style.json has a double slash, which prevents loading.

[arbakker]

I regenerated the tiles with the correct config, the base_url had a trailing slash. Should be working now.

[pka]

Great! mapbox-gl.js is now happy, too.

[lomereiter]

Hello, sorry to hijack the thread, I have a small ask regarding releases.

The Linux binaries for 0.12.0 depend on glibc 2.29+ (check with objdump -T t_rex | grep GLIBC) whereas Debian stable is currently on 2.28. Could you please build the releases inside a Docker container? Simply use the official Rust docker image:

FROM rust:1.49-buster
WORKDIR /code
COPY . .
RUN apt-get update && apt-get install -y libgdal-dev
RUN cargo install --path .
CMD ["t_rex"]

Thanks!

[pka]

So it looks like it's appropriate to release it as 0.13! I'll wait a few days to estimate when #237 can be merged. If it takes too long, I will release without updating actix-web. To update all Tokio dependencies we also had to update rusoto, which leads into the next async adventure...