Implement PMTiles S3 bucket support with authentication

[nyurik]

pmtiles crate will soon support s3 buckets directly. Martin needs to enable support for S3-hosted pmtiles

TODO

• [ ] create a new PmtS3Source similar to PmtHttpSource
• [ ] establish config settings for pmtiles on s3, including authentication
• [ ] establish CLI parameters for pmtiles on s3
• [ ] update martin book with usage examples

Developers can already use by adding this at the end of the root Cargo.toml:

[patch.crates-io]
pmtiles = { git = "https://github.com/stadiamaps/pmtiles-rs", branch = "add-s3-backend" }

See also https://github.com/maplibre/martin/discussions/1125 CC: @alexjball @lseelenbinder

[nyurik]

for configuration, we currently have this pattern:

pmtiles:
  paths:
    # scan this whole dir, matching all *.pmtiles files
    - /dir-path
    # specific pmtiles file will be published as a pmt source (filename without extension)
    - /path/to/pmt.pmtiles
    # A web server with a PMTiles file that supports range requests
    - https://example.org/path/tiles.pmtiles
  sources:
    # named source matching source name to a single file
    pm-src1: /path/to/pmt.pmtiles
    # A named source to a web server with a PMTiles file that supports range requests
    pm-web2: https://example.org/path/tiles.pmtiles

We can add s3 support as either an s3 subsection under pmtiles, or by adding a new top level pmtiles-s3. I think the first option is better, but it might be trickier to implement.

pmtiles:
  s3:
    [...something...]

pmtiles-s3:
    [...something...]

Config should cover the following cases:

• one s3 bucket with a single pmtiles file
• one s3 bucket with multiple pmtiles files
  • file names are listed in the config
  • files are auto-discovered (using s3 enumeration - requires additional permission)
• multiple s3 buckets

So it seems the usage is similar to postgres configuration, where we can have one or more pg connection (i.e. s3 authentication), and each connection can have one or more table (i.e. pmtiles files), and the table name can be declared in the config (with the corresponding source name), or they can be auto-discovered, in which case the source name is auto-generated, e.g. {filename} or {bucket}_{filename}.

[jleedev]

The protomaps/PMTiles supports specifying a bucket (and a path prefix), and allows fetching any pmtiles object by name, rather than having to specify them all ahead of time, or listing the objects. Would require some refactoring here, but it's a sensible way to avoid expensive reconfigurations.

[jtmiclat]

Hi! I would be interested in implementing this if noone has started on it yet!

[nyurik]

we have some s3 support in place, but nothing with auth as far as i know, and I don't know anyone working on this at the moment. @jleedev was the only person who also worked on it in the merged #1127