custom_domain insufficient for supporting MinIO

odigity commented 2 years ago

Goal

I want to use MinIO in our development stack so that our S3-dependent code will work the same in both dev and production. Our stack is a set of Docker containers - for Django, Postgres, and MinIO.

Relevant Facts

S3 simultaneously supports two URL styles:
- path-style: https://s3.amazonaws.com/<bucket>/path/to/file.ext
- virtual-host-style: https://<bucket>.s3.amazonaws.com/path/to/file.ext
MinIO also supports both styles.
Amazon desires to and is progressing towards deprecating path-style eventually.
The boto3 Client and ServiceResource classes default to virtual-host-style when generating URLs, but switch to path-style when endpoint_url is specified during instantiation.
```
client = session.client( 's3', endpoint_url='http://minio:9000/' )
 ->  http://minio:9000/<bucket>/path/to/file.ext
```
The S3Boto3Storage class uses the boto3 classes to generate URLs unless the custom_domain attribute is truthy, in which case URLs are assembled by the S3Boto3Storage class using path-style: https://<custom_domain>/path/to/file.ext
Linux desktops will resolve *.localhost domains out-of-the-box, but MacOS desktops won't.

Problem

Given that MinIO supports both, and MacOS developers would be inconvenienced by having to take extra steps to make use of virtual-host-style URLs in dev (http://<bucket>.localhost:9000/path/to/file.ext/), I'd prefer to use path-style URLs.

However...

...since I must specify endpoint_url to connect S3Boto3Storage to MinIO instead of AWS...
...and since the endpoint_url (http://minio:9000/) is only resolveable inside Docker...
...and since the one option for overriding the use of endpoint_url during URL generation is to set custom_domain...
...and since setting custom_domain results in URLs that are virtual-host-style (bucket_name not pre-pended to object key)

...I see no solution to using django-storages with MinIO in a Docker stack with path-style URLs without modifying django-storages or at least overriding the url() method with my own version, which is unappealing.

Fallback to Virtual-Host-Style

Assuming I'm willing to inconvenience the aforementioned MacOS developers, then a solution is possible as follows:

set settings.AWS_S3_ENDPOINT_URL = 'http://minio:9000/' (or from env var)
invent and set settings.AWS_S3_SERVING_BASE = 'localhost:9000'` (or from env var)

create base class to override custom_domain setting:

class StorageBase(S3Boto3Storage):
    @property
    def custom_domain(self):
        if settings.AWS_S3_SERVING_BASE:
            return f"{self.bucket_name}.{settings.AWS_S3_SERVING_BASE}"

set bucket_name in each subclass

class UploadStorage(StorageBase):
    bucket_name = settings.AWS_S3_UPLOAD_BUCKET

Proposal

I'm not confidant in recommending a specific change to django-storages to deal with this situation because I'm fairly new to the topic. However, clearly something is needed. Hopefully my betters will find their way here and contribute ideas.

odigity commented 2 years ago

UPDATE: I've just learned that while MacOS doesn't resolve subdomains of localhost, major browsers do, even when run on MacOS. So, I'm going to be using the "Fallback to Virtual-Host-Style" implementation described above.

odigity commented 2 years ago

UPDATE: I just noticed the addressing_style option which does improve the situation slightly.

However, I still think it's worth at least discussing the inclusion of an option like AWS_S3_SERVING_BASE to represent that concept.

jschneier / django-storages