search

theriverman / django-minio-backend

Minio Backend for Django
https://pypi.org/project/django-minio-backend/
MIT License
111 stars 22 forks source link

file url django admin in docker compose #23

Closed iforvard closed 2 years ago

iforvard commented 3 years ago

There is a config to replace url in admin if django and minio are running in docker compose. By default, there will be an internal docker link to minio.

theriverman commented 3 years ago

I see what your problem might be in a Docker environment. I think by changing the default behaviour in https://github.com/theriverman/django-minio-backend/blob/b201e7b6f0d98b736e56ae6a5c64da58f8eeacf3/django_minio_backend/models.py#L177-L196 should solve the problem for you, but I need to find a solution which is backward compatible too.

I'll take a look at this during the weekend. Thanks for reporting, @iforvard!

theriverman commented 3 years ago

Hey, @iforvard Please upgrade to 2.7.0, and add the following new parameters to your settings.py:

Let me know if this release solves your problem!

iforvard commented 3 years ago

Hey, @theriverman Here is my config. But in the admin panel, the url of the file has not changed.

URL django amdin file

http://minio:9000/s3-private/2021-6-7/terms.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=MINIO_ACCESS_KEY%2F20210607%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210607T082008Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=c70a6190d6e1e362c3bc329da23145b6e509d6d8a41ee38edd17e6bb5929a451

MINIO Config

MINIO_EXTERNAL_ENDPOINT = "127.0.0.1" # Default is same as MINIO_ENDPOINT MINIO_EXTERNAL_ENDPOINT_USE_HTTPS = False MINIO_ENDPOINT = 'minio:9000' MINIO_ACCESS_KEY = 'MINIO_ACCESS_KEY' MINIO_SECRET_KEY = 'MINIO_SECRET_KEY' MINIO_USE_HTTPS = False MINIO_URL_EXPIRY_HOURS = timedelta(days=7) # Default is 7 days (longest) if not defined MINIO_CONSISTENCY_CHECK_ON_START = True MINIO_PRIVATE_BUCKETS = [ 's3-private', ] MINIO_PUBLIC_BUCKETS = [ 's3-public', ] MINIO_POLICY_HOOKS: List[Tuple[str, dict]] = []

theriverman commented 3 years ago

hey, @iforvard Can you show me an example of what would be the expected URL, please? For example, http://minio.iforvard.com/s3-private/2021-6-7/terms.pdf (you can truncate the end of the URL starting from X-Amz-... )

iforvard commented 3 years ago

Hey, @theriverman For example , http://127.0.0.1:9000/s3-private/2021-6-7/terms.pdf If you go through the web interface and share the file, you get such links.

theriverman commented 3 years ago

Thanks for the example, @iforvard ! I see you've set MINIO_EXTERNAL_ENDPOINT = "127.0.0.1" instead of MINIO_EXTERNAL_ENDPOINT = "127.0.0.1:9000" which means the generated URL will not contain the port number part (:9000) where you're MinIO is actually served from.

Could you update your settings.py, restart Django and try again, please?

iforvard commented 3 years ago

I added a port to MINIO_EXTERNAL_ENDPOINT. Nothing has changed in the admin panel, the link is the same as without specifying the port. if you have the option to test docker here is the docker-compose.yaml file

version: "3.3"

services:
  minio:
    image: minio/minio
    ports:
      - "9000:9000"
    volumes:
      - ./data:/data
    environment:
      - "MINIO_ACCESS_KEY=minios3buckets"
      - "MINIO_SECRET_KEY=ApPi1GjEfaHIqI"
    command: server /data
  createbuckets:
    image: minio/mc
    depends_on:
      - minio
    entrypoint: >
      /bin/sh -c "
      /usr/bin/mc alias set myminio http://minio:9000 minios3buckets ApPi1GjEfaHIqI;
      /usr/bin/mc mb myminio/s3-public;
      /usr/bin/mc mb myminio/s3-private;
      /usr/bin/mc policy set public myminio/s3-public;
      exit 0;
      "
theriverman commented 3 years ago

Okay, I think I got the problem for good this time. I suppose you were receiving SignatureDoesNotMatch errors from MinIO. This happens because of a faulty implementation I've done in generating the URL. I have corrected this mistake in 2.7.1. Please upgrade!

By the way: I think you have an error in your docker-compose.yaml file in line: /usr/bin/mc alias set myminio http://minio:9000 minios3buckets ApPi1GjEfaHIqI; The ; in the end is not separated from the password with a whitespace so your alias operation fails.

iforvard commented 3 years ago

Hey, @theriverman

django-minio-backend==2.7.1

MINIO_ENDPOINT = 'minio:9000' if MINIO_EXTERNAL_ENDPOINT = "127.0.0.1:9000" or "localhost:9000" Error in the admin panel: Exception Value: | Couldn't connect to Minio. Check django_minio_backend parameters in Django-Settings if MINIO_EXTERNAL_ENDPOINT = 'minio:9000' There is no error. For some reason, he cannot connect to an external address. Although I can do it through the browser.

theriverman commented 3 years ago

This has to be a problem with your Docker configuration for sure, @iforvard . django-minio-backend is not altering anything on network-level. I have verified most scenarios, and I could access my MinIO instance both internally running via Docker on localhost and externally from another machine.

Tip: Try running both MinIO and your Django application on your localhost (without involving Docker) and see if it works

theriverman commented 3 years ago

I'm closing this issue now. Please feel free to reopen if needed.

ezawadzki commented 3 years ago

Hi @theriverman,

It seems I've got the same issue, even I can connect to Minio Browser at http://localhost:9153 :

backend_1   | urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=9153): Max retries exceeded with url: /django-backend-dev-private?location= (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fbea44290a0>: Failed to establish a new connection: [Errno 111] Connection refused'))
backend_1   |    File "/usr/local/lib/python3.9/site-packages/django_minio_backend/models.py", line 254, in url
backend_1   |     raise ConnectionError("Couldn't connect to Minio. Check django_minio_backend parameters in Django-Settings")
backend_1   | ConnectionError: Couldn't connect to Minio. Check django_minio_backend parameters in Django-Settings

My configuration :

settings.py

MINIO_ENDPOINT = 'minio1:9000'
MINIO_EXTERNAL_ENDPOINT = 'localhost:9153'
MINIO_EXTERNAL_ENDPOINT_USE_HTTPS = False

docker-compose.yml

  minio1:
    image: "minio/minio"
    command: server /data
    volumes:
      - ./data:/srv/data
    environment:
      - MINIO_ROOT_USER=....
      - MINIO_ROOT_PASSWORD=....
    ports:
      - "9153:9000"

Versions:

django-minio-backend : 3.0.0
minio  : 7.1.0
django : 3.1
python : 3.9.5

Thanks for your help !

theriverman commented 3 years ago

Hi @ezawadzki

Edit: I need to correct myself, I think I've found the bug's root cause, but it's not completely trivial to fix. I'll get back to you here when I have some progress.

ezawadzki commented 3 years ago

Hi @theriverman ,

Thanks for your quick answer ! Yes, it's strange that Django is trying to connect to minio service through localhost:9153 instead of docker host minio1:9000.

But I feel confused as I have to put my website in production next week.... do you have a visibility when you could fix it ? It would be a pleasure to help you and make you a PR but I have to finish the other features of the website.

All the best !

ezawadzki commented 3 years ago

@theriverman

At this line, even if MINIO_EXTERNAL_ENDPOINT is set, the endpoint still has to be internal , as Django is communicating with Minio ? Right?

theriverman commented 3 years ago

If you're moving to production with your publicly available MinIO instance , then you should be okay by setting both values ( MINIO_ENDPOINT, MINIO_EXTERNAL_ENDPOINT ) to your MinIO's public address (e.g.: my-minio.company.co.uk ).

That way all connection from Django to MinIO will be made through a public interface instead of Docker's internal network. Please, try verifying it in public facing staging environment.

I'll take a look at the code sometime during this weekend.

theriverman commented 3 years ago

@ezawadzki

I've made some internal changes, so Docker environments should now work normally both in production and development environments. Version 3.1.0b0 has been pushed to PyPI, please give it a try!

pip install django-minio-backend==3.1.0b0
chugunovyar commented 1 year ago

MINIO_EXTERNAL_ENDPOINT_USE_HTTPS = False MINIO_EXTERNAL_ENDPOINT = 'mysite.com:9000'

I set the variables but still the link in template is incorrect like https://mysite.com:9000/bucket-name/myfile.jpg but I expected that it will be http://mysite.com:9000/bucket-name/myfile.jpg

what I`m doing wrong ?

theriverman commented 1 year ago

@chugunovyar Could you verify your settings via Django Shell, please? See DjangoExampleApplication/tests.py for some clue on doing this.