Access buckets by subdomain or so?

lf- commented 2 months ago

Is your feature request related to a problem? Please describe.

I want to be able to host the releases bucket at releases.example.com. In principle I could just mess with it at the reverse proxy, but s3-proxy will generate incorrect links, and may also have bad interactions with redirectWithTrailingSlashForNotFoundFile among other things.

It feels like I am hacking this tool too hard, and maybe it needs to be able to do the thing natively.

Describe the solution you'd like

Ideally I would like to be able to ask for the mount point or some other string to be stripped off the start of all paths sent to the client.

Describe alternatives you've considered

Reverse proxy hacks are not ideal and feel like I could be screwing up an edge case.

Additional context Add any other context or screenshots about the feature request here.

oxyno-zeta commented 2 months ago

Hello @lf- ,

I must admit that I don't understand very well the issue you are referring to. I'm pretty sure I can help you with this.

Can you give me some examples of what you are looking for in terms of URLs ? Can you provide me the configuration you are using and a bucket structure example please ?

Best regards,

Oxyno-zeta

lf- commented 2 months ago

Yes, so I have some buckets, say, s3://releases, which I want to host on releases.example.com. I also have several other such buckets that I want to do the same thing for, without running separate instances of s3-proxy. So I use a reverse proxy that forwards releases.example.com to localhost:10652/releases/. Here is the configuration of s3-proxy being used for this:

server:
  compress:
    enabled: false
  cors:
    allowMethods:
    - GET
    allowOrigins:
    - '*'
    enabled: true
  listenAddr: 127.0.0.1
  port: 10652
targets:
  docs:
    actions:
      GET:
        config:
          indexDocument: index.html
          redirectWithTrailingSlashForNotFoundFile: true
        enabled: true
    bucket:
      credentials:
        accessKey:
          env: AWS_ACCESS_KEY_ID
        secretKey:
          env: AWS_SECRET_KEY
      name: docs
      region: garage
      s3Endpoint: http://127.0.0.1:3900
    mount:
      path:
      - /docs/
  install:
    actions:
      GET:
        config:
          indexDocument: index.html
          redirectWithTrailingSlashForNotFoundFile: true
        enabled: true
    bucket:
      credentials:
        accessKey:
          env: AWS_ACCESS_KEY_ID
        secretKey:
          env: AWS_SECRET_KEY
      name: install
      region: garage
      s3Endpoint: http://127.0.0.1:3900
    mount:
      path:
      - /install/
  releases:
    actions:
      GET:
        config:
          indexDocument: index.html
          redirectWithTrailingSlashForNotFoundFile: true
        enabled: true
    bucket:
      credentials:
        accessKey:
          env: AWS_ACCESS_KEY_ID
        secretKey:
          env: AWS_SECRET_KEY
      name: releases
      region: garage
      s3Endpoint: http://127.0.0.1:3900
    mount:
      path:
      - /releases/

However, this has a problem! When generating the index page for releases.example.com aka localhost:10652/releases/, the default templates use .Entries[N].Path, which includes the /releases prefix in all the URLs appearing on the page. Additionally the same thing happens for redirectWithTrailingSlashForNotFoundFile so the reverse proxy needs to be configured to eat the first URL component of redirects (I have other reasons to be dissatisfied with this feature but I will raise them in another issue).

Bucket structure

I expect to be able to access releases.example.com/lix/lix-2.90-beta.0/ and get a directory listing and have this also work for other buckets on their respective similarly-configured subdomains.

~ » aws s3 ls --recursive s3://releases
2024-05-05 16:13:01   13064752 lix/lix-2.90-beta.0/nix-2.90.0-beta.0-aarch64-darwin.tar.xz
2024-05-05 16:13:01   20698772 lix/lix-2.90-beta.0/nix-2.90.0-beta.0-aarch64-linux.tar.xz
2024-05-05 16:13:01   22970916 lix/lix-2.90-beta.0/nix-2.90.0-beta.0-i686-linux.tar.xz
2024-05-05 16:13:01   15988324 lix/lix-2.90-beta.0/nix-2.90.0-beta.0-x86_64-darwin.tar.xz
2024-05-05 16:13:01   22392548 lix/lix-2.90-beta.0/nix-2.90.0-beta.0-x86_64-linux.tar.xz
2024-05-05 18:50:26   13060816 lix/lix-2.90-beta.1/lix-2.90.0-beta.1-aarch64-darwin.tar.xz
2024-05-05 18:50:26   20699144 lix/lix-2.90-beta.1/lix-2.90.0-beta.1-aarch64-linux.tar.xz
2024-05-05 18:50:26   22968472 lix/lix-2.90-beta.1/lix-2.90.0-beta.1-i686-linux.tar.xz
2024-05-05 18:50:26   15989924 lix/lix-2.90-beta.1/lix-2.90.0-beta.1-x86_64-darwin.tar.xz
2024-05-05 18:50:26   22408228 lix/lix-2.90-beta.1/lix-2.90.0-beta.1-x86_64-linux.tar.xz
2024-05-05 18:50:01   13060816 lix/lix-2.90-beta.1/nix-2.90.0-beta.1-aarch64-darwin.tar.xz
2024-05-05 18:50:01   20699144 lix/lix-2.90-beta.1/nix-2.90.0-beta.1-aarch64-linux.tar.xz
2024-05-05 18:50:01   22968472 lix/lix-2.90-beta.1/nix-2.90.0-beta.1-i686-linux.tar.xz
2024-05-05 18:50:01   15989924 lix/lix-2.90-beta.1/nix-2.90.0-beta.1-x86_64-darwin.tar.xz
2024-05-05 18:50:01   22408228 lix/lix-2.90-beta.1/nix-2.90.0-beta.1-x86_64-linux.tar.xz
2024-06-14 20:22:52   16284524 lix/lix-2.90.0-rc1/lix-2.90.0-rc1-aarch64-darwin.tar.xz
2024-06-14 20:22:53         65 lix/lix-2.90.0-rc1/lix-2.90.0-rc1-aarch64-darwin.tar.xz.sha256
2024-06-14 20:22:52   21149860 lix/lix-2.90.0-rc1/lix-2.90.0-rc1-aarch64-linux.tar.xz
2024-06-14 20:22:53         65 lix/lix-2.90.0-rc1/lix-2.90.0-rc1-aarch64-linux.tar.xz.sha256
2024-06-14 20:22:52  130966810 lix/lix-2.90.0-rc1/lix-2.90.0-rc1-docker-image-aarch64-linux.tar.gz
2024-06-14 20:22:53         65 lix/lix-2.90.0-rc1/lix-2.90.0-rc1-docker-image-aarch64-linux.tar.gz.sha256
2024-06-14 20:22:53  131838425 lix/lix-2.90.0-rc1/lix-2.90.0-rc1-docker-image-x86_64-linux.tar.gz
2024-06-14 20:22:53         65 lix/lix-2.90.0-rc1/lix-2.90.0-rc1-docker-image-x86_64-linux.tar.gz.sha256
2024-06-14 20:22:57    5350341 lix/lix-2.90.0-rc1/lix-2.90.0-rc1-manual.tar.gz
2024-06-14 20:22:53         65 lix/lix-2.90.0-rc1/lix-2.90.0-rc1-manual.tar.gz.sha256
2024-06-14 20:22:55   19327800 lix/lix-2.90.0-rc1/lix-2.90.0-rc1-x86_64-darwin.tar.xz
2024-06-14 20:22:57         65 lix/lix-2.90.0-rc1/lix-2.90.0-rc1-x86_64-darwin.tar.xz.sha256
2024-06-14 20:23:00   22967536 lix/lix-2.90.0-rc1/lix-2.90.0-rc1-x86_64-linux.tar.xz
2024-06-14 20:23:03         65 lix/lix-2.90.0-rc1/lix-2.90.0-rc1-x86_64-linux.tar.xz.sha256
2024-06-14 20:23:06    1407304 lix/lix-2.90.0-rc1/lix-2.90.0-rc1.tar.gz

Solutions

Support virtual hosts natively so you can just have bucket.example.com based on the Host header
Support mountpoint-relative paths in the default templates
- Also have some way to configure redirects to be mountpoint-relative
- To be honest I don't like this solution because it doesn't allow you to merge two buckets into the same namespace later on.

oxyno-zeta commented 2 months ago

Hello @lf- ,

Ok I think I got the issue here. There is a way to have virtual hosts in the app. Check this part of the configuration: https://oxyno-zeta.github.io/s3-proxy/configuration/structure/#mountconfiguration

I will just write an example of this based on your example:

server:
  compress:
    enabled: false
  cors:
    allowMethods:
    - GET
    allowOrigins:
    - '*'
    enabled: true
  listenAddr: 127.0.0.1
  port: 10652
targets:
  docs:
    actions:
      GET:
        config:
          indexDocument: index.html
          redirectWithTrailingSlashForNotFoundFile: true
        enabled: true
    bucket:
      credentials:
        accessKey:
          env: AWS_ACCESS_KEY_ID
        secretKey:
          env: AWS_SECRET_KEY
      name: docs
      region: garage
      s3Endpoint: http://127.0.0.1:3900
    mount:
      host: docs.example.com
  install:
    actions:
      GET:
        config:
          indexDocument: index.html
          redirectWithTrailingSlashForNotFoundFile: true
        enabled: true
    bucket:
      credentials:
        accessKey:
          env: AWS_ACCESS_KEY_ID
        secretKey:
          env: AWS_SECRET_KEY
      name: install
      region: garage
      s3Endpoint: http://127.0.0.1:3900
    mount:
      host: install.example.com
  releases:
    actions:
      GET:
        config:
          indexDocument: index.html
          redirectWithTrailingSlashForNotFoundFile: true
        enabled: true
    bucket:
      credentials:
        accessKey:
          env: AWS_ACCESS_KEY_ID
        secretKey:
          env: AWS_SECRET_KEY
      name: releases
      region: garage
      s3Endpoint: http://127.0.0.1:3900
    mount:
      host: releases.example.com

I removed the mount path in order to have only a hostname.

The other solution can be to change the list template by a different one created by you. Check this part of the configuration to change the template for all targets: https://oxyno-zeta.github.io/s3-proxy/configuration/structure/#main-structure (under the templates key) or this part for a custom per target: https://oxyno-zeta.github.io/s3-proxy/configuration/structure/#targetconfiguration (under the templates key)

Tell me if this is solving your issue or if need more help.

Oxyno-zeta

lf- commented 2 months ago

Oh wow I completely missed the virtual hosts support in the config file. Well that simply solves the problem, thank you!

It might be helpful to point out in the examples that it exists, but feels like a bit of failure of imagination on my part to not have looked for it.

oxyno-zeta commented 2 months ago

Oh I see now :) . Happy to see that it solved your issue !

Thanks for the ping. I will try to do something in the docs.

oxyno-zeta / s3-proxy

Access buckets by subdomain or so? #473