SSL not working properly with built image (instead of only pulled)

[hashkeks]

Hello everyone, while trying to enable PAM and therefore rebuilding the image it seems like the rebuilt image is faulty in regards of SSL verfication for minio. When I try to push an image, on the client side I get the error message

FATAL:   Unable to push image to library: request did not succeed: http status code: 500

and on the server side I receive (which looks like communication with Minio failed):

uwsgi_1      | botocore.exceptions.SSLError: SSL validation failed for https://my.domain.com:9000/sregistry
/test/asbach%3Asha256.d09823a72014f5730fa9a3843df62b059ad0d9dc20661e37f7220525fb6d04be?uploads [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

A little background: I use a Let's Encrypt cert for my sregistry and because Minio has to be available at port 9000 to the public to pull and push images, Minio uses the same cert. Therefore in config.py internal and external Minio-service is set to my publicly available domain name because otherwise cert verification from uwsgi, etc. will fail because - of course - the cert is not issued for localhost or 127.0.0.1. When I don't use a built image and instead just pull the ones declared in the docker-compose.yml everything works fine with that setup.

To reproduce the fault: Just build the image using the command docker build -t quay.io/vanessa/sregistry . and use this image. Options like --build-arg ENABLE_PAM=true aren't even necessary. Whatever option I set for building the image, it results in a "faulty" image. Maybe some http.client object is faulty and can't access CA certs? Unfortunately I have near to no experience with the code.

Expected behaviour: To work flawlessly like the setup does without a built but pulled docker image.

Version is the latest available one from this very repo. If you need more info, please let me know.

Every hint or help is appreciated, thanks everyone!

[vsoch]

The setup is optimized for non-SSL, and I haven't personally set up a production server with SSL - I think in the documentation I just link to the Minio site. So I'll do the same here - you'll need to also setup SSL with Minio - like most SSL setups it doesn't just work out of the box. https://docs.min.io/docs/how-to-secure-access-to-minio-server-with-tls.html

[hashkeks]

Thank you for the linked docu. In the case of Minio and Let's Encrypt certs it's actually working out of the box, even according to the docu :) I'm fairly new to this whole thing, but maybe you can tell me which of the services talk to Minio when pushing an image via singularity CLI so I can look into it. What I also do wonder about: Do non-SSL setups work? I thought singularity does need a secure connection

[vsoch]

Singularity has been updated so it dynamically can figure out if the remote needs http or https. I just develop with http so historically I would change the source code to force it to work.

The uwsgi container would be the one that talks to Minio. You can find all the setup and commands in minio.py I believe in the library module under apps.

[hashkeks]

So the problem indeed was in minio.py, or rather with botocore. I don't know why SSL worked in my previous SSL setups without any plugins enabled and without this change, but in this case the verify attribute of the s3 and s3_external objects needs to be set to true in order for it to work. So the dead simple fix in my case was in the minio.py to add the following code to also not break non-SSL setups: If MINIO_SSL is true, set the verify accordingly to true. If not, set it to false.

if MINIO_SSL:
    VERIFY = True
else:
    VERIFY = False
# https://github.com/boto/boto3/blob/develop/boto3/session.py#L185
s3 = session.client(
    "s3",
    verify=VERIFY,
    use_ssl=MINIO_SSL,
    endpoint_url=MINIO_HTTP_PREFIX + MINIO_SERVER,
    region_name=MINIO_REGION,
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)

# signature_versions
# https://github.com/boto/botocore/blob/master/botocore/auth.py#L846
s3_external = session.client(
    "s3",
    verify=VERIFY,
    use_ssl=MINIO_SSL,
    endpoint_url=MINIO_HTTP_PREFIX + MINIO_EXTERNAL_SERVER,
    region_name=MINIO_REGION,
    config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
)

I guess it's solved then, thanks.

[vsoch]

@hashkeks could you please open a PR to fix this for others too?

[hashkeks]

Issue is now closed with a small fix. https://github.com/singularityhub/sregistry/pull/384