feat(s3): allow virtual-host-style buckets

[cmdoret]

Context: depending on the S3 provider, buckets may be specified in path-style (domain/bucket/key), or virtual-host-style (bucket.domain/key). Modos uses botocore, which should autodetect which format is supported by the S3 server, however this detection fails depending on the provider.

Change summary: Keep autodetect by default, but allow setting an environment variable (S3_ADDRESSING_STYLE) to force a specific bucket addressing style to "path" or "virtual".

Minor changes:

• template htsget config with environment variables (using envsubst)
• pass path_style to htsget based on S3_ADDRESSING_STYLE
• consistent style in modos dockerfile
• Set correct region variable in htsget service, and rm uneeded awscli client

Limitations

virtual-host style does not work behind the reverse proxy, only with an external s3 server. This is because I could not find a way to setup nginx to propertly handle both subdomains and /s3 I propose that we stick for virtual host support for dedicated servers

Trying it

1. Setup "external S3 server"

$ echo "127.0.0.1   modos modos-demo.modos >> /etc/hosts
$ docker run -e 'MINIO_DOMAIN=modos' --rm -p 9000:9000 -p 9001:9001 docker.io/bitnami/minio:2024

2. Create modos-demo public bucket in mino console at http://modos:9000

3. Start compose without minio:

   • Comment-out minio service
   • Add S3_LOCAL_URL=modos:9000 to .env file docker compose up --build

4. Create modo in forced virtual host mode

   export S3_ADDRESSING_STYLE='virtual'
   modos create --s3-endpoint http://modos:9000 modos-demo/abc

[cmdoret]

Indeed, I had to build the client/server images locally for this to work, forgot to specify, sorry!