sregistry:1.1.39 - uwsgi_1: minio.error.ResponseError: ResponseError: code: BadRequest, message: Bad Request, bucket_name: test

[ifelsefi]

Hi

uwsgi_1 cannot connect to minio though it's up:

[~/repos/sregistry]$ sudo docker-compose ps
        Name                       Command               State                                    Ports
----------------------------------------------------------------------------------------------------------------------------------------
sregistry_db_1          docker-entrypoint.sh postgres    Up       5432/tcp
sregistry_minio_1       /usr/bin/docker-entrypoint ...   Up       0.0.0.0:9000->9000/tcp,:::9000->9000/tcp
sregistry_nginx_1       nginx -g daemon off;             Up       0.0.0.0:443->443/tcp,:::443->443/tcp, 0.0.0.0:80->80/tcp,:::80->80/tcp
sregistry_redis_1       docker-entrypoint.sh redis ...   Up       6379/tcp
sregistry_scheduler_1   python /code/manage.py rqs ...   Exit 1
sregistry_uwsgi_1       /bin/sh -c /code/run_uwsgi.sh    Up       3031/tcp
sregistry_worker_1      python /code/manage.py rqw ...   Exit 1

Settings:

MINIO_EXTERNAL_SERVER = (
    "127.0.0.1:9000"  # minio server for Singularity to interact with
)
MINIO_BUCKET = "test"
MINIO_SSL = False  # use SSL for minio
MINIO_SIGNED_URL_EXPIRE_MINUTES = 5
MINIO_REGION = "us-east-1"
MINIO_MULTIPART_UPLOAD = True

Env:

MINIO_ACCESS_KEY=newminio
MINIO_SECRET_KEY=key
MINIO_ACCESS_KEY_OLD=minio
MINIO_SECRET_KEY_OLD=key
MINIO_BROWSER=off

Any idea why uwsgi can't connect to minio on the same local docker network?

I do see:

[~/repos/sregistry]$ curl https://127.0.0.1:9000 curl: (51) Unable to communicate securely with peer: requested domain name does not match the server's certificate

Unsure why I would need a certificate to connect with a local docker image?

Any way to disable SSL verification?

Thank you!

[vsoch]

You can look at the logs of your images?

[ifelsefi]

Somehow minio was deployed with a SSL cert and it's listening on https. I thought this was only going to happen with nginx.

So I think bad request was attempt to do http to https. When I set:

MINIO_SERVER = "domain.example.com:9000"  # Internal to sregistry
MINIO_EXTERNAL_SERVER = (
    "domain.example.com:9000"  # minio server for Singularity to interact with
)
MINIO_BUCKET = "sregistry"
MINIO_SSL = True  # use SSL for minio
MINIO_SIGNED_URL_EXPIRE_MINUTES = 5
MINIO_REGION = "us-east-1"
MINIO_MULTIPART_UPLOAD = True

This does make curl work and I see there's no TLS chain issues:

[~/repos/sregistry]$ curl -v domain.example.com:9000

About to connect() to domain.example.com port 9000 (#0)
*   Trying 10.1.54.76...
* Connected to domain.example.com (10.1.54.76) port 9000 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=domain.example.com,OU=foobar
*       start date: Mar 03 14:57:41 2022 GMT
*       expire date: Apr 04 14:57:41 2023 GMT
*       common name: domain.example.com,
*       issuer: CN=CAuthority
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: domain.example.com:9000
> Accept: */*
>
< HTTP/1.1 403 Forbidden
< Accept-Ranges: bytes
< Content-Length: 226
< Content-Security-Policy: block-all-mixed-content
< Content-Type: application/xml
< Server: MinIO
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Vary: Origin
< Vary: Accept-Encoding
< X-Amz-Request-Id: 16D8F14AFF6914A9
< X-Content-Type-Options: nosniff
< X-Xss-Protection: 1; mode=block
< Date: Thu, 03 Mar 2022 18:09:28 GMT
<
<?xml version="1.0" encoding="UTF-8"?>
* Connection #0 to host domain.example.com left intact

So minio does have the cert in place!

But logs show:

uwsgi_1      |   File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 786, in urlopen
uwsgi_1      |     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
uwsgi_1      |   File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 592, in increment
uwsgi_1      |     raise MaxRetryError(_pool, url, error or ResponseError(cause))
uwsgi_1      | urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='domain.example.com', port=9000): Max retries exceeded with url: /sregistry/ (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))
uwsgi_1      |
uwsgi_1      | 0 static files copied to '/var/www/static', 340 unmodified.

So I am unsure why uwsgi_1 can't verify the chain. I probably need to add /etc/ssl as a volume so it can read the CA bundle?

My docker-compose.yaml does not have that present:

uwsgi:
  restart: always

  # IMPORTANT: update the tag to coincide with release version
  image: quay.io/vanessa/sregistry:1.1.39
  env_file:
    - ./.minio-env
  volumes:
    - .:/code
    - ./static:/var/www/static
    - ./images:/var/www/images

So it can't verify the chain :)

[vsoch]

So first I'd recommend starting your first attempt without ssl anywhere and making sure that your updated credentials work, etc. Otherwise you might put yourself into a situation of debugging multiple layers at once. And once you are ready to turn on SSL, there are docs for how to do that - the short story is that there is a different set of deploy files in https folder you need to use. https://singularityhub.github.io/sregistry/docs/install/https. But you cannot just use the docker-compose.yml in the root with https - it isn't intended for that as you see.

[ifelsefi]

Yes, I copied the https specific docker-compose file. That's how nginx was able to start.

[vsoch]

And the nginx assets as well?

[vsoch]

So what I would do in your case is debug the error verbatim. Your uwsgi container is telling you that it cannot see domain.example.com so you probably should shell in to test that. With docker-compose the hostnames are typically container names so I would start by figuring out what the uwsgi container is seeing and go from there.

[ifelsefi]

All good now! I disabled TLS on minio but kept it working with nginx!

[vsoch]

Woohoo! That's what I like to hear! :partying_face: