MINIO + HTTPS - Unclear how you performed HTTPS without having IP SANs in our cert / using external Cert, perhaps the documentation could be updated to show it working with external cert?

[lmcdasm]

Hello

i tried to do the installation today and could not get the combination of components using HTTPS (which is setup correctly in NGINIX) with MINIO. Referring to this section here

https://singularityhub.github.io/sregistry/docs/install/server

As best as i can tell, you end up with a "local" endpoint (that is done and listening on the 17.16.x.y internal docker network. however, this is not possible with a "name" since a real cert (from our CA/PKI) would not have a SAN for 127.0.0.1 it it (only names).

we can see that Minio comes up and i can define it as a external server (reachable via the browser from https://mydomain.net:9000 (and SSL enabled) and i can create a bucket however (again this is a guess), since you are using the parameter here

MINIO_SERVER = "minio:9000" # Internal to sregistry -

this seems to use the docker named link and thus when you have installed a SSL cert for mydomain.net, and internally registry tried to use minio (and thus https://127.0.0.1) you get a failure.

Any idea on how i can correct this?

here is the shub/setttings/config.py

STORAGE

here i have tried some different variants, but havent had any luck

MINIO_SERVER = "singregistry.mydomain.net:9000" # Internal to sregistry MINIO_EXTERNAL_SERVER = ( "singregistry.mydomain.net:9000" # minio server for Singularity to interact with ) MINIO_BUCKET = "testbucket" MINIO_SSL = False # use SSL for minio MINIO_SIGNED_URL_EXPIRE_MINUTES = 5 MINIO_REGION = "us-east-1" MINIO_MULTIPART_UPLOAD = True

Don't clean up images in Minio that are no longer referenced by sregistry

DISABLE_MINIO_CLEANUP = False

Here is a picture showing that i am able to get into MINIO and see the test bucket created. (testbucket)

Here is the "mc command" output (config ls)

[root@grid-singregistry-master sregistry]# !323 ./mc config host ls gcs URL : https://storage.googleapis.com AccessKey : YOUR-ACCESS-KEY-HERE SecretKey : YOUR-SECRET-KEY-HERE API : S3v2 Lookup : dns

local URL : http://localhost:9000 AccessKey : SecretKey : API : Lookup : auto

myminio URL : https://singregistry.mydomain.net:9000 AccessKey : abc123abc SecretKey : abc123abc123 API : s3v4 Lookup : auto

play URL : https://play.min.io AccessKey : Q3AM3UQ867SPQQA43P2F SecretKey : zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG API : S3v4 Lookup : auto

s3 URL : https://s3.amazonaws.com AccessKey : YOUR-ACCESS-KEY-HERE SecretKey : YOUR-SECRET-KEY-HERE API : S3v4 Lookup : dns

[root@grid-singregistry-master sregistry]#

but i cant see why uswgi (but its the worker and the scheduler container continue to throw this error):

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 34, in inner response = get_response(request) File "/usr/local/lib/python3.5/site-packages/django/utils/deprecation.py", line 94, in call response = response or self.get_response(request) File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 36, in inner response = response_for_exception(request, exc) File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 90, in response_for_exception response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info()) File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 128, in handle_uncaught_exception callback, param_dict = resolver.resolve_error_handler(500) File "/usr/local/lib/python3.5/site-packages/django/urls/resolvers.py", line 597, in resolve_error_handler callback = getattr(self.urlconf_module, 'handler%s' % view_type, None) File "/usr/local/lib/python3.5/site-packages/django/utils/functional.py", line 80, in get res = instance.dict[self.name] = self.func(instance) File "/usr/local/lib/python3.5/site-packages/django/urls/resolvers.py", line 577, in urlconf_module return import_module(self.urlconf_name) File "/usr/local/lib/python3.5/importlib/init.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 985, in _gcd_import File "", line 968, in _find_and_load File "", line 957, in _find_and_load_unlocked File "", line 673, in _load_unlocked File "", line 697, in exec_module File "", line 222, in _call_with_frames_removed File "./shub/urls.py", line 19, in from shub.apps.library import urls as library_urls File "./shub/apps/library/urls.py", line 12, in from shub.apps.library import views File "./shub/apps/library/views/init.py", line 2, in from .images import ( File "./shub/apps/library/views/images.py", line 33, in from .minio import ( File "./shub/apps/library/views/minio.py", line 62, in if not minioClient.bucket_exists(MINIO_BUCKET): File "/usr/local/lib/python3.5/site-packages/minio/api.py", line 453, in bucket_exists self._url_open('HEAD', bucket_name=bucket_name) File "/usr/local/lib/python3.5/site-packages/minio/api.py", line 2080, in _url_open object_name).get_exception() minio.error.ResponseError: ResponseError: code: BadRequest, message: Bad Request, bucket_name: testbucket, object_name: None, request_id: , host_id: , region:

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.5/site-packages/django/core/handlers/wsgi.py", line 141, in call response = self.get_response(request) File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 75, in get_response response = self._middleware_chain(request) File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 36, in inner response = response_for_exception(request, exc) File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 90, in response_for_exception response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info()) File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 128, in handle_uncaught_exception callback, param_dict = resolver.resolve_error_handler(500) File "/usr/local/lib/python3.5/site-packages/django/urls/resolvers.py", line 597, in resolve_error_handler callback = getattr(self.urlconf_module, 'handler%s' % view_type, None) File "/usr/local/lib/python3.5/site-packages/django/utils/functional.py", line 80, in get res = instance.dict[self.name] = self.func(instance) File "/usr/local/lib/python3.5/site-packages/django/urls/resolvers.py", line 577, in urlconf_module return import_module(self.urlconf_name) File "/usr/local/lib/python3.5/importlib/init.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 985, in _gcd_import File "", line 968, in _find_and_load File "", line 957, in _find_and_load_unlocked File "", line 673, in _load_unlocked File "", line 697, in exec_module File "", line 222, in _call_with_frames_removed File "./shub/urls.py", line 19, in from shub.apps.library import urls as library_urls File "./shub/apps/library/urls.py", line 12, in from shub.apps.library import views File "./shub/apps/library/views/init.py", line 2, in from .images import ( File "./shub/apps/library/views/images.py", line 33, in from .minio import ( File "./shub/apps/library/views/minio.py", line 62, in if not minioClient.bucket_exists(MINIO_BUCKET): File "/usr/local/lib/python3.5/site-packages/minio/api.py", line 453, in bucket_exists self._url_open('HEAD', bucket_name=bucket_name) File "/usr/local/lib/python3.5/site-packages/minio/api.py", line 2080, in _url_open object_name).get_exception() minio.error.ResponseError: ResponseError: code: BadRequest, message: Bad Request, bucket_name: testbucket, object_name: None, request_id: , host_id: , region: [pid: 59|app: 0|req: 2/4] 10.3.103.8 () {48 vars in 972 bytes} [Tue Jul 28 22:00:28 2020] GET /favicon.ico => generated 0 bytes in 1256 msecs (HTTP/1.1 500) 0 headers in 0 bytes (0 switches on core 3) [root@grid-singregistry-master sregistry]#

running the MC TRACE< doesnt provide anything in the window and nothing shows up int he minio logs - here they are:

[root@grid-singregistry-master sregistry]# docker logs -f 3cf Attempting rotation of encrypted config, IAM users and policies on MinIO with newly supplied credentials Rotation complete, please make sure to unset MINIO_ACCESS_KEY_OLD and MINIO_SECRET_KEY_OLD envs Endpoint: https://172.17.0.4:9000 https://127.0.0.1:9000

Browser Access: https://172.17.0.4:9000 https://127.0.0.1:9000

Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide

Any tips here would be great, since if i have to add a 127.0.0.1 SAN into our wildcard cert for our domain, it means i have to reissue it to 500 + machines.

Thanks!

[vsoch]

I haven set up the server with https, but I suspect you might need to configure hostnames or write the config so it uses the correct internal / external addresses (or maybe have two)? Did you see the link to https://docs.min.io/docs/how-to-secure-access-to-minio-server-with-tls.html? That’s what I pointed the user to in the current docs, and I was hoping you would be able to follow steps there to set it up. I can’t easily locally reproduce with a custom domain, etc. so it’s definitely challenging to help via that method!

[vsoch]

A quick observations:

1. For MINIO_SERVER = "singregistry.mydomain.net:9000" # Internal to sregistry

That would be a good thing to try, but that address would need to be defined in the hostname to point to the container (which currently is named "minio"

2. My intuition is saying that you might need a certificate for both the internal and external client, since the addresses are different. If you can't get them to be the same (point 1) that is what I'd try. I'd also try these test cases of shelling into the container and trying basic instantiation of clients.

3. The part

   Attempting rotation of encrypted config, IAM users and policies on MinIO with newly supplied credentials
   Rotation complete, please make sure to unset MINIO_ACCESS_KEY_OLD and MINIO_SECRET_KEY_OLD envs
   Endpoint: https://172.17.0.4:9000 https://127.0.0.1:9000

   at first I thought was concerning, but probably is just refencing the definition of the previous envars. You might try following that advice to unset and update.

4. This seems important, in terms of location:

What I would try to do is first look into other examples of inter-container communication and https, and see if some custom work is needed. I've never had to do anything special, but when you add in custom networking it tends to complicate things. You might also try breaking it down into the simplest example to reproduce (e.g., connecting via an internal/external client and trying to connect to a bucket) and then ask the folks at Minio (e.g., the miniopy repo) specificaly about the differences in setting up the cert between an internal and external client.

[vsoch]

More things to try! Via this thread: https://github.com/minio/minio-py/issues/759#issuecomment-490250355 it looks like you might try making the internal url http, and also playing around with the secure=True/False variable when you instantiate the call.

[lmcdasm]

Hello Again.. thanks for the quick answers..

So essentially i was at those links and i had gotten to the point of the following: in docker-compose (or the build) i can pass a hostname, or alias variable so that it will have that "name" the "trick" is that you are using links in your docker-compose so as it stands, there is no "networks" variable in the scope of minio (it would have to be added in if you wanted to use aliases).

A quick observations:

1. For MINIO_SERVER = "singregistry.mydomain.net:9000" # Internal to sregistry

That would be a good thing to try, but that address would need to be defined in the hostname to point to the container (which currently is named "minio"

Along the above lines, i can "get that to work sort of", however this means that in the links definitions, i have to use a 'minio.mydomain.net" for example and then perpetuate that through the docker compose - trying that out.

1. My intuition is saying that you might need a certificate for both the internal and external client, since the addresses are different. If you can't get them to be the same (point 1) that is what I'd try. I'd also try these test cases of shelling into the container and trying basic instantiation of clients.
2. The part

Attempting rotation of encrypted config, IAM users and policies on MinIO with newly supplied credentials
Rotation complete, please make sure to unset MINIO_ACCESS_KEY_OLD and MINIO_SECRET_KEY_OLD envs
Endpoint: https://172.17.0.4:9000 https://127.0.0.1:9000

at first I thought was concerning, but probably is just refencing the definition of the previous envars. You might try following that advice to unset and update.

have rotation the credentials a couple times on restarts in the past CI build loops

1. This seems important, in terms of location:

What I would try to do is first look into other examples of inter-container communication and https, and see if some custom work is needed. I've never had to do anything special, but when you add in custom networking it tends to complicate things. You might also try breaking it down into the simplest example to reproduce (e.g., connecting via an internal/external client and trying to connect to a bucket) and then ask the folks at Minio (e.g., the miniopy repo) specificaly about the differences in setting up the cert between an internal and external client.

Thats a great idea above - thanks! i have installed s3 and MC and the "external container" link seems alright.. i think its getting HTTPS on the internal link by service name (in a docker-compose context). i am thinking i can probably just do the docker build of the minio and drop in the hostname and others directly..

For the certificate, since i have a valid one ( you can see the HTTPS is working in the UI, and i can inspect the cert on the minio browser/site) im pretty sure the mapping to the cert/private is correct and its a wildcard off our domain, so will server anything top level). I think its a question of that mapping of the endpoint name back in and the MINIO_SSL TRue flag

[lmcdasm]

More things to try! Via this thread: minio/minio-py#759 (comment) it looks like you might try making the internal url http, and also playing around with the secure=True/False variable when you instantiate the call.

Excellent! - thanks, seems like its the path im on..

Question - is there a place in docs to post reference example configurations and layouts/samples in the project (anonymized).? it might be neat to have some pictures and examples if we get this working (ideally https on the in and outside), or http/https ... could help others maybe?

Cheers D

[vsoch]

That’s a fantastic idea! Right now we have Minio under storage under setup, so we could either add more content there (or if there is enough to warrant separation) make another link. We also could make a section under deployments (right now there is just an old ansible recipe.) A combination of the two would be having the basic setup where it is now, and linking to a more detailed “Minio with HTTPS Deployment” page under deployments. I can definitely help with this however you need - I am so grateful that you are testing this out and paving the way for future users!

[lmcdasm]

Hey there..

ok.. i have a lazy guide that is basically 'from Centos to running install on docker" (pre-cursor to making some k8s manifests), and happy to contribute that with some screen shots and such (once i sort this out)>

for the issue at hand, going into the python stack trace, we can see there are two MinioClients that are defined (minioClient and minioClientExternal). and i see that you then create a session and s3 client connection for each Minio that you stand up, looks good.

However, line 62 seems (maybe to me) out of place, since you are doing a "make_bucket call" before you have a session, thus you end up with a tuple like this "host: {minio] url: /testbucket/"

(which would actually work with your http:// if MINIO_SSL is False prefix + MINIO_DOMAIN_NAME - however, (in my case, im still stuck with an FQDN issue - however, there is a "argument" that could be made that since its "internal communication" the use of a signed CA cert (digicert in my case) is overkill for the internal link path) -

However, when i commented out that 'make_bucket" and rebuilt locally with the following shub/settings/config.py and the call commented out shown below, SRegistry is now up, and working (weird huh? :P). What i suspect (and sort of intuiting from the minio() constructor is that in the case of your mark_bucket, you dont have a session, so even though its http (to minio) you still need credentials, and results in the scheduler and worker throwing a Bad request.

... all of this is a bit of guess and "history | grep build | wc -l " (docker build) = 104 and the registry is up now with https :)

shub/apps/library/views/minio.py (in your flask setup)

 44 MINIO_HTTP_PREFIX = "https://" if MINIO_SSL else "http://"
 45
 46 minioClient = Minio(
 47     MINIO_SERVER,
 48     region=MINIO_REGION,
 49     access_key=os.environ.get("MINIO_ACCESS_KEY"),
 50     secret_key=os.environ.get("MINIO_SECRET_KEY"),
 51     secure=MINIO_SSL,
 52 )
 53
 54 minioExternalClient = Minio(
 55     MINIO_EXTERNAL_SERVER,
 56     region=MINIO_REGION,
 57     access_key=os.environ.get("MINIO_ACCESS_KEY"),
 58     secret_key=os.environ.get("MINIO_SECRET_KEY"),
 59     secure=MINIO_SSL,
 60 )
 61) # Commented this out, since i dont think you can check if the bucket exists yet, you havent created the connection.
 62 #if not minioClient.bucket_exists(MINIO_BUCKET):
 63 #    minioClient.make_bucket(MINIO_BUCKET)
 64
 65 session = Session(
 66     aws_access_key_id=os.environ.get("MINIO_ACCESS_KEY"),
 67     aws_secret_access_key=os.environ.get("MINIO_SECRET_KEY"),
 68     region_name=MINIO_REGION,
 69 )
 70
 71
 72 # https://github.com/boto/boto3/blob/develop/boto3/session.py#L185
 73 s3 = session.client(
 74     "s3",
 75     verify=False,
 76     use_ssl=MINIO_SSL,
 77     endpoint_url=MINIO_HTTP_PREFIX + MINIO_SERVER,
 78     region_name=MINIO_REGION,
 79     config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
 80 )
 81
 82 # signature_versions
 83 # https://github.com/boto/botocore/blob/master/botocore/auth.py#L846

Here is my shub/settings/config.py presently:

MINIO_SERVER = "minio:9000" # Internal to sregistry MINIO_EXTERNAL_SERVER = ( "singregistry.mydomain.net:9000" # minio server for Singularity to interact with ) MINIO_BUCKET = "testbucket" MINIO_SSL = False # use SSL for minio MINIO_SIGNED_URL_EXPIRE_MINUTES = 5 MINIO_REGION = "us-east-1" MINIO_MULTIPART_UPLOAD = True

Here is what my "mc config ls " looks like.

./mc config host ls gcs URL : https://storage.googleapis.com AccessKey : YOUR-ACCESS-KEY-HERE SecretKey : YOUR-SECRET-KEY-HERE API : S3v2 Lookup : dns

local URL : http://localhost:9000 AccessKey : SecretKey : API : Lookup : auto

myminio URL : https://singregistry.mydomain.net:9000 AccessKey : xxxxxxxxxxx SecretKey : xxxxxxxxxxxxxx API : s3v4 Lookup : auto

play URL : https://play.min.io AccessKey : xxxxxxxxxxxxxxxx SecretKey : xxxxxxxxxxxxxx API : S3v4 Lookup : auto

s3 URL : https://s3.amazonaws.com AccessKey : YOUR-ACCESS-KEY-HERE SecretKey : YOUR-SECRET-KEY-HERE API : S3v4 Lookup : dns

[lmcdasm]

Oh and to add, i have it working now with LDAPs (with with a baked Private CA cert, since internally we dont want to have to have linux clients need a standard SSL cert (A Private CA is fine)

have a meeting tomorrow to do with the SAML/SSO (sorting out iDP and SP configuration and the metadata.xml to do it).

The introduction of Minio is a really great idea, there are somethings i would like to add (help to add).

• S3 is good for AWS - have you got any other Cloud Service Providers in mind or just sticking with S3.

  • if you have ever heard of rclone (https://rclone.org) they have already written backends for just about everything and have a pretty simple SDK in Python.. it might be a easy trick to co-opt that rclone tool (bake it in ) and then your minio setup would instantly have support for just about every cloud provider out there)

• how goes the K8s work? now that i have running containers, i will write up some simple K8s manifests (not helm charts yet, just kubectl apply stuff) and get these working as a deployment and look toward integration to our ingress controller (this is where i think minion plugin is good, since most people running k8s have a minio instance running now to do just this thing) - very happy!

Shoot me the branch in the repo where you want pushs to (docs directly? .RST, .MD , guidelines?) where i can post a "Use Case Example Tutorial - Single Host/Docker/External Owned Cert" or some title can be butchers, and i throw up a diagram and lazy guide/screens and we can edit from there)"

Continuing to test, but so far Logins, Teams, Collections, AD LDAP looking good!

[lmcdasm]

so.. it would seem im back to something "familiar"

the client: singularity v3.6 remote configured and working no issue openssl verify the cert shows all is good

singularity remote use company-remote API Key is "valid"

Steps of Issue: A) login to Singlurity Registry (note: Im using LDAP and its working ok) B) Create a collection called "my-test" C) on the Client i am running singularity push -U dasm-test.sif library://daniel.smith/mytest/sometthing:lates

and in the uswgi here is what i am seeing: i am seeing a call the /v1/entities endpoint>

USWGI LOG:

od.json => generated 561 bytes in 10 msecs (HTTP/1.1 200) 3 headers in 109 bytes (1 switches on core 2) GET NamedEntityView <QueryDict: {}> daniel.smith Not Found: /v1/entities/daniel.smith [pid: 54|app: 0|req: 33/144] 10.179.111.4 () {34 vars in 487 bytes} [Wed Jul 29 04:36:06 2020] GET /v1/entities/daniel.smith => generated 0 bytes in 11 msecs (HTTP/1.1 404) 4 headers in 115 bytes (1 switches on core 1) Method Not Allowed: /v1/entities [pid: 54|app: 0|req: 34/145] 10.179.111.4 () {36 vars in 491 bytes} [Wed Jul 29 04:36:06 2020] POST /v1/entities => generated 41 bytes in 1 msecs (HTTP/1.1 405) 5 headers in 157 bytes (1 switches on core 2)

CLIENT CALL IN DEBUG:

[root@ip-0AB36F04 ~]# singularity --debug push -U dasm-test.sif library://daniel.smith/mytest/sometthing:latest DEBUG [U=0,P=62325] persistentPreRun() Singularity version: 3.6.0-1.el7 DEBUG [U=0,P=62325] persistentPreRun() Parsing configuration file /etc/singularity/singularity.conf DEBUG [U=0,P=62325] handleConfDir() /root/.singularity already exists. Not creating. DEBUG [U=0,P=62325] handleRemoteConf() Ensuring file permission of 0600 on /root/.singularity/remote.yaml WARNING [U=0,P=62325] LibraryPush() Skipping container verifying DEBUG [U=0,P=62325] UploadImage() Image hash computed as b226c379e19853f16f1dc76b13b70deaf822afd6078c5bc3cd9b921d32cf7101 **DEBUG [U=0,P=62325] apiGet() apiGet calling v1/entities/daniel.smith DEBUG [U=0,P=62325] UploadImage() Entity daniel.smith does not exist in library - creating it. DEBUG [U=0,P=62325] apiCreate() apiCreate calling v1/entities** FATAL [U=0,P=62325] func50() Unable to push image to library: request did not succeed: http status code: 405

So, this leads me to the SWAGGER Page to test the apiCreate Call - i suspect that the "check if there, if not create is faling"

i looked on the API page (swagger) for the registry and i dont see you have a /v1/apiCreate call available there.

looking more into the code call now - i suspect it is cause the singularity client is calling that create v1/entities/daniel.smith and since its not exposed via your flask endpoints it fails maybe?

I put some more debug statements and the issue was the token in validate_token was returning false so not found.

This is due to the fact that the rebuilds, need a new token for the client each time (duh!) ...

[lmcdasm]

Hey there.

i have essentially got it working, however i made a mistake with the .minio-env and the order of passwords is mixed and messed. what is the correct way to reset that? wipe the .minio.sys directory and minio-images?

do i go back to the original default creds and start the chain again?

as you can see in the client DEBUG, im "almost there", however due to the miino ACCESS/KEY SECRET/KEY mixup the /v2/call is failing presently.

[root@ip-0AB36F04 ~]# singularity --debug push -U dasm-test.sif library://daniel.smith/my-test/sometthing:latest DEBUG [U=0,P=85039] persistentPreRun() Singularity version: 3.6.0-1.el7 DEBUG [U=0,P=85039] persistentPreRun() Parsing configuration file /etc/singularity/singularity.conf DEBUG [U=0,P=85039] handleConfDir() /root/.singularity already exists. Not creating. DEBUG [U=0,P=85039] handleRemoteConf() Ensuring file permission of 0600 on /root/.singularity/remote.yaml WARNING [U=0,P=85039] LibraryPush() Skipping container verifying DEBUG [U=0,P=85039] UploadImage() Image hash computed as b226c379e19853f16f1dc76b13b70deaf822afd6078c5bc3cd9b921d32cf7101 DEBUG [U=0,P=85039] apiGet() apiGet calling v1/entities/daniel.smith DEBUG [U=0,P=85039] apiGet() apiGet calling v1/collections/daniel.smith/my-test DEBUG [U=0,P=85039] apiGet() apiGet calling v1/containers/daniel.smith/my-test/sometthing DEBUG [U=0,P=85039] apiGet() apiGet calling v1/images/daniel.smith/my-test/sometthing:sha256.b226c379e19853f16f1dc76b13b70deaf822afd6078c5bc3cd9b921d32cf7101?arch=amd64 DEBUG [U=0,P=85039] postFileWrapper() Now uploading to the library DEBUG [U=0,P=85039] postFileV2() Using legacy (single part) uploader DEBUG [U=0,P=85039] legacyPostFileV2() legacyPostFileV2 calling v2/imagefile/1 DEBUG [U=0,P=85039] apiCreate() apiCreate calling v2/imagefile/1 0.0b / 2.6MiB [------------------------------------------------------------------------------] 0 % 0.0 b/s 0s FATAL [U=0,P=85039] func50() Unable to push image to library: request did not succeed: http status code: 500

This is cause Minio is out now, it was throwing the MINIM TIMEOUT For SIgned URL and i was about to increase that to a high value.

And the fix for this is to remove the _OLD from the .minio-env file and restart, so that you dont double encrypt and it is up again

[vsoch]

hey @lmcdasm! Sorry for the delay in response, I did in fact go to sleep! I've put together a branch and a page for you to work on documentation: https://github.com/singularityhub/sregistry/blob/add/minio-ssl-docs/docs/_docs/deploy/minio-ssl.md Specifically, that's linked from the sidebar under the deployment section, and also linked from the Storage setup documentation (and it links back to it). For images, you can either put them in assets/img/minio (or similar) or just an img/ folder in the same folder with the file, and then the path would be relative (actually one level up) from the page. E.g.,, with this structure:

minio-ssl.md
img/
   image1.png

In the markdown the image would be linked like:

![../img/image1.png](../img/image1.png)

Now to answer your questions!

1. Before you jump into making Kubernetes manifests, you should ping on this issue https://github.com/singularityhub/sregistry/issues/317 because I think the user has already done some work, and you might work together. It would be great to have a Kubernetes recipe as a deployment option!
2. You are correct that there are two clients, and this was a challenge with creating the service because the registry need to be able to interact with Minio internally (to request signed URLS) and externally (to use them). In fact, we have to use the internal client to request a URL that will work for the external one. It took a while for me to figure out how to do that, and I had to implement a custom signer function.
3. The make bucket call here is done on the init of the server - I'm not sure what you mean by before having a session? What kind of session do we need? The registry server uses one bucket, maybe you had in mind that each user creates their own bucket? That's not the case.
4. My quick fix would indeed be to not used a signed cert for internal communication - why would it be needed? But if you think t's important I'd reach out and open an issue for minio-py - they have a lovely and very helpful community. In that the names are different for internal vs. external, I suspect you would need to register both names with a certificate somwhow, or just generate two of them.
5. Singularity Registry will of course go up if you don't make the bucket, but you'll hit errors when you try to interact with it (and it doesn't exist). If you want to try commenting it out and then manually creating it you could try that, but I'd ask why you don't want to create it at the server start?

Just to be clear - the minio client that creates the bucket is authenticated

minioClient = Minio(
    MINIO_SERVER,
    region=MINIO_REGION,
    access_key=os.environ.get("MINIO_ACCESS_KEY"),
    secret_key=os.environ.get("MINIO_SECRET_KEY"),
    secure=MINIO_SSL,
)

Also, the server is not implemented in Flask, it's in Django - the "framework for perfectionists!" It's probably my favorite :)

6. In case you didn't see it, if you are setting up SAML there is a plugin for it! https://singularityhub.github.io/sregistry/docs/plugins/saml And I suspect you saw the LDAP one too :)

7. if you want another cloud service, there is a Google Build plugin that uses Cloud Build - https://singularityhub.github.io/sregistry/docs/plugins/google-build and of course Google Storage. That doesn't mean pushing to Google Storage though. The reason for Minio is to support a local container (minio) that can have multipart upload for push. I haven't had any requests for other kinds of cloud storage.

8. Definitely ping the other user on Kubernetes!

9. Could you tell me more about what you have in mind for rclone? Maybe give an example of how you would see that working, implementation wise? It's a cool idea, indeed - but I'm wondering if others haven't done it, if there is a reason.

10. I'm not sure the swagger API has all of the library endpoints, did you try the standard API page?

11. I'm not sure about resetting minio, I've never done it before! But you might just delete all containers and start from scratch? Why are you using the legacy endpoint and not multipart? The legacy endpoint is going to give you a lot of trouble with large images, hence why Minio was created.

[vsoch]

I think it would be worth getting Minio working, and if you can come up with a basic use case of one of the clients having a permission error to connect to the bucket, and then showing the mc debug output to the minio team, they could help you out. Remember that you can shell into the docker container, then do python manage.py shell and then try instantiating the various minio clients / manually creating the bucket and testing. Keep me posted!

[lmcdasm]

hey @lmcdasm! Sorry for the delay in response, I did in fact go to sleep! No apologies, sleep is a good thing!

This (below) is amazing, i will start to .md up my stuff and push as i write.

I've put together a branch and a page for you to work on documentation: >https://github.com/singularityhub/sregistry/blob/add/minio-ssl-docs/docs/_docs/deploy/minio-ssl.md Specifically, that's linked >rom the sidebar under the deployment section, and also linked from the Storage setup documentation (and it links back to it). >For images, you can either put them in assets/img/minio (or similar) or just an img/ folder in the same folder with the file, and t>hen the path would be relative (actually one level up) from the page. E.g.,, with this structure:

minio-ssl.md
img/
   image1.png

In the markdown the image would be linked like:

![../img/image1.png](../img/image1.png)

Now to answer your questions!

1. Before you jump into making Kubernetes manifests, you should ping on this issue #317 because I think the user has already done some work, and you might work together. It would be great to have a Kubernetes recipe as a deployment option! dasm - aye.. saw that while browsing, will hit it up for sure!

2. You are correct that there are two clients, and this was a challenge with creating the service because the registry need to be able to interact with Minio internally (to request signed URLS) and externally (to use them). In fact, we have to use the internal client to request a URL that will work for the external one. It took a while for me to figure out how to do that, and I had to implement a [custom signer function] (https://github.com/singularityhub/sregistry/blob/master/shub/apps/library/views/minio.py#L94). dasm - this is probably about where i was at. and i think its really a question of keys and minio.config with the TLS/SSL . What i noticed was that when the "bucket call" that i commented out was being called on restart it was hitting a "expired timer".

i believe that somewhere in the MINIO installation part there was something to add for 7 minutes or something.. on successive builds/restarts - i suspect that this might have expired? is that mc call to add a signed endpoint to minio (myminio in the docs) you are meaning in the CSG (i havent had a chance to look yet so if yet, you can leave me to find it :P)

3. The make bucket call here is done on the init of the server - I'm not sure what you mean by before having a session? What kind of session do we need? The registry server uses one bucket, maybe you had in mind that each user creates their own bucket? That's not the case.

OK.. so.. i will show a bit of the hand at the architecture here

discussion point - on a VM/Node base abstraction layer environment:

• i have the ability to present a HPFS-parallel (Grid computing calibre) filesystem (RDMA/IB - 25 to 100Gbst) as the "minio-images mount point lets say" (before you say wait dan there are easier ways, hear out the progression i am working out at a platform level) - this means that the call through the docker container (minio) is in fact a context switch to a block device backed FS at the node level. what am i getting at:

  • this HPFS i can stretch to almost everywhere we have cloud providers (all of them mostly and across them seamlessly)
  • The IOPS and such are tunable and all sorts of other nice things.

  with the net/net being that minio container gets a slice of a very fast filesystem to work with and the POSIX based. When we talk about replication between "datacenters" - there is in fact no "replication" - since all clients see the filesystem (and we structure the mapping a rules/RBAC and other models way - we then are in a paradigm where we have one or two central hubs that provide slices of "storage fabric" out to applications (in this case a VM with containers on it). the fabric of course is 5, 9s redundnant, self-scaling.

Discussion - take it a bit further into K8S:

• we see a large chain push, pull in terms of production of data required for application in high end computing/real-time environments, Models and Logic KB based programs require massive amounts of "loading" of those inputs before they can run at runtime.

  • when we then look at how we run our applications on K8s, and replica sets - we then have this issue of having many read of the same input data, this leads to very load startup times and means we have to either have a big fat registry that can handle all these requests and we end up keeping many copies of the same input data around a global scope since we have to replicate the publishing points in a production network>

• we will look at Minio then and in this case with an application like SREGISTRY, this allows me to present the containers via the simple orchestration (bind mound to the filesystem we start the docker-compose from in this singleton deployment case), it allows me to run the sregistry in an atomic fashion.. where the provide the minio backend is abstracted.. the registry could be destroyed (or i could start up many copies of them and - parallelism/code updates notwithstanding) i will never "lose" my container and my sregistry is disaggregated on a GRID backed filesystem

  • in K8s there is the concept of the CSI plugin, where we could write our own adapter for this HPFS filesystem, however that its alot of work and if i can use a very simple implementation of minio (blobfuse with a /tmp/mapper is simply not going to cut it for the rates we are talking about) - then this allows me also to scale the sregistry to some pretty nice sizes and since i can stretch the data wherever i want, my redundant copy is only a kubectl apply away -

massive oversimplification of the work needed... :)

4. My quick fix would indeed be to not used a signed cert for internal communication - why would it be needed? But if you think t's important I'd reach out and open an issue for minio-py - they have a lovely and very helpful community. In that the names are different for internal vs. external, I suspect you would need to register both names with a certificate somehow, or just generate two of them.

DASM - yes, however i do have two certs and can do a internal and external without an issue if need be. as for why would it be needed, there is the idea that today you use nginx as your "ingress controller" essentially and this is fine and can port nicely to k8s .

however, when we want to talk in the realm of service registry and Service mesh, where there is a service that provides a common ingress that all apps in a cluster bind their FQDNs to this gateway (and thus certificate management at a deployment FQDN level is vastly improved since its one cert (or many) for the cluster ). in this case the ngnix front door will go and at that point the idea of external client/internal client might become "irrelevant" - and even desired since it would make the idea of "many replicas of sregistry components" (minio_1, minio_2,minio_3 - running in a given name space with a Service name of "minio" but now cluster redundant - and with HPFS redundant parallel storage) we know have something very robust and geo-distribution ready.

5. Singularity Registry will of course go up if you don't make the bucket, but you'll hit errors when you try to interact with it (and it doesn't exist). If you want to try commenting it out and then manually creating it you could try that, but I'd ask why you don't want to create it at the server start? DASM - i was just hacking around, adding print statements and seeing what i could get to start.. i think that since i had a directory (bucket) in the minio-images/ directory since the testbucket (that i created with a API call to test and have configured) was there and so ti did come up..

Just to be clear - the minio client that creates the bucket is authenticated

minioClient = Minio(
    MINIO_SERVER,
    region=MINIO_REGION,
    access_key=os.environ.get("MINIO_ACCESS_KEY"),
    secret_key=os.environ.get("MINIO_SECRET_KEY"),
    secure=MINIO_SSL,
)

Also, the server is not implemented in Flask, it's in Django - the "framework for perfectionists!" It's probably my favorite :) DASM - oops - sorry, i see views,static,urls, and its all JINJA to me :)

6. In case you didn't see it, if you are setting up SAML there is a plugin for it! https://singularityhub.github.io/sregistry/docs/plugins/saml And I suspect you saw the LDAP one too :)

DASM - yup, worked out the details, once im back back to a working rig ( i think i just botched up Minioa a bit with trying things), im going to start on that integration.> For the docs, we can show a "LDAP reference Configuration' and " SSO/SAML iDp" one as well/ varient

1. if you want another cloud service, there is a Google Build plugin that uses Cloud Build - https://singularityhub.github.io/sregistry/docs/plugins/google-build and of course Google Storage. That doesn't mean pushing to Google Storage though. The reason for Minio is to support a local container (minio) that can have multipart upload for push. I haven't had any requests for other kinds of cloud storage.

DASM - is not super pressing, we use a couple different Cloud Providers

2. Definitely ping the other user on Kubernetes! DASM - yup ypup
3. Could you tell me more about what you have in mind for rclone? Maybe give an example of how you would see that working, implementation wise? It's a cool idea, indeed - but I'm wondering if others haven't done it, if there is a reason. DASM - back to above, i was thinking that if you wanted a way (i will look more at the GOOGLE stuff, to be honest, i breezed by since i thought it was GCP specific and its not a target for me right now) that you could simple make python calls to different Cloud Provider Storage Backends, you could have an adapter that does read/writes to various Object base Storage that Cloud providers have which is presented either as filesystem or object.. This would allow when sregistry wants to "write" (store) a container - that is actually a call to Cloud Object Storage (and the rclone py implementation already has all the different stanza for about 60 different protocols - not just Cloud providers, http, ssh, https, http/2 in the works)..

Writing this, the use case of having instances of sregistry deployed globally and rather than using a disk/network fil PVC in k8s, that when SREG is deployed it can be configured to use many different Cloud Provide Object storage (fast and cheap) ) - maybe a "feature request"

4. I'm not sure the swagger API has all of the library endpoints, did you try the standard API page? DASM - i did not, but will look at it
5. I'm not sure about resetting minio, I've never done it before! But you might just delete all containers and start from scratch? DASM - i think that is where im at. I was able to get it to start in safe mode, but not able to get the mc admin config command to initialize the setup - googlin' around for hit and starting on that minio.py code to see where it might lie, but other than my own diving into code, will probably try the nuclear option shortly.

Why are you using the legacy endpoint and not multipart? The legacy endpoint is going to give you a lot of trouble with large images, hence why Minio was created. DASM - im "not" intentionally, after the getEntity and getCollection that is what singularity (3.6) was doing - the debug shows the call to the endpoint its making, i was looking for the scs-client library to see what tehy were doing when the createApi when they pass it that context, but then i too went to sleep>

Action plan at this point, have lunch :P Then wipe that slice and start with a fresh bucket, put back the make bucket call cause it "was work" (minio) before and then it hit that error saying the "keys" were no longer valid.. at any rate, will get back to a solid state (hacks out) and see where we are at.

hopefully i didnt ramble too much.. not enough coffee yet. Cheers, D

[lmcdasm]

Hey there.

So we are back in business, MINIO is up, HTTPS is working, i can login an i can create Collections and such>

when im doing the following Singularity command (and i have updated my .sregistry and i coped the new token in after loging in ) for my singularity client, see below

CLIENT SIDE:

UPDATE KEY: [root@ip-0AB36F04 ~]# singularity remote login INFO: Authenticating with default remote. Generate an API Key at https://singregistry.hosting.cerence.net/auth/tokens, and paste here: API Key: INFO: API Key Verified!

WHICH ONE IS IN USE: [root@ip-0AB36F04 ~]# singularity remote list NAME URI GLOBAL SylabsCloud cloud.sylabs.io YES [company-remote] singregistry.hosting.cerence.net YES You have new mail in /var/spool/mail/root [root@ip-0AB36F04 ~]#

[root@ip-0AB36F04 ~]# singularity remote status INFO: Checking status of default remote. SERVICE STATUS VERSION Keystore Service OK v1.0.0 Library Service OK v1.0.0 Token Service OK v1.0.0 [root@ip-0AB36F04 ~]#

SINGULARITY PUSH (with debug turned on):

[root@ip-0AB36F04 ~]# singularity --debug push -U dasm-test.sif library://daniel.smith/dasm-test/sometthing:latest DEBUG [U=0,P=16381] persistentPreRun() Singularity version: 3.6.0-1.el7 DEBUG [U=0,P=16381] persistentPreRun() Parsing configuration file /etc/singularity/singularity.conf DEBUG [U=0,P=16381] handleConfDir() /root/.singularity already exists. Not creating. DEBUG [U=0,P=16381] handleRemoteConf() Ensuring file permission of 0600 on /root/.singularity/remote.yaml WARNING [U=0,P=16381] LibraryPush() Skipping container verifying DEBUG [U=0,P=16381] UploadImage() Image hash computed as b226c379e19853f16f1dc76b13b70deaf822afd6078c5bc3cd9b921d32cf7101 DEBUG [U=0,P=16381] apiGet() apiGet calling v1/entities/daniel.smith DEBUG [U=0,P=16381] apiGet() apiGet calling v1/collections/daniel.smith/dasm-test DEBUG [U=0,P=16381] apiGet() apiGet calling v1/containers/daniel.smith/dasm-test/sometthing DEBUG [U=0,P=16381] apiGet() apiGet calling v1/images/daniel.smith/dasm-test/sometthing:sha256.b226c379e19853f16f1dc76b13b70deaf822afd6078c5bc3cd9b921d32cf7101?arch=amd64 DEBUG [U=0,P=16381] postFileWrapper() Now uploading to the library DEBUG [U=0,P=16381] postFileV2() Using legacy (single part) uploader DEBUG [U=0,P=16381] legacyPostFileV2() legacyPostFileV2 calling v2/imagefile/2 DEBUG [U=0,P=16381] apiCreate() apiCreate calling v2/imagefile/2 FATAL [U=0,P=16381] func50() Unable to push image to library: request did not succeed: http status code: 500 You have new mail in /var/spool/mail/root [root@ip-0AB36F04 ~]#

HERE IS THE USGW LOG FOR THAT PUSH - you can see its giving an Access Denied (dont be offended at my tracing hacks)

ytes in 9 msecs (HTTP/1.1 200) 3 headers in 109 bytes (1 switches on core 2) GET NamedEntityView DASM HAACK IN PLACE <QueryDict: {}> GET None /v1/entities/daniel.smith {} {'Host': 'singregistry.hosting.cerence.net', 'Authorization': 'BEARER 099e260ba86feb7664be19378fe40273da8c7a9e', 'Accept-Encoding': 'gzip', 'Content-Length': '', 'Content-Type': '', 'User-Agent': 'Go-http-client/1.1'} daniel.smith token before split BEARER 099e260ba86feb7664be19378fe40273da8c7a9e Found a token in validdate_token() token: 099e260ba86feb7664be19378fe40273da8c7a9e token: True foudn token in getNamedEntityView() yes we found user Generated user data: {'customData': '', 'createdAt': '2020-07-29T16:54:05.120524Z', 'defaultPrivate': False, 'name': 'daniel.smith', 'id': '1', 'collections': ['1'], 'quota': 0, 'deleted': False, 'updatedAt': '2020-07-29T16:54:06.099455Z', 'deletedAt': '0001-01-01T00:00:00Z', 'size': 0, 'description': 'daniel.smith', 'updatedBy': '', 'createdBy': ''} [pid: 53|app: 0|req: 7/34] 10.179.111.4 () {34 vars in 487 bytes} [Wed Jul 29 11:59:41 2020] GET /v1/entities/daniel.smith => generated 313 bytes in 22 msecs (HTTP/1.1 200) 5 headers in 142 bytes (1 switches on core 0) GET GetNamedCollectionView token before split BEARER 099e260ba86feb7664be19378fe40273da8c7a9e Found a token in validdate_token() token: 099e260ba86feb7664be19378fe40273da8c7a9e [pid: 58|app: 0|req: 15/35] 10.179.111.4 () {34 vars in 513 bytes} [Wed Jul 29 11:59:41 2020] GET /v1/collections/daniel.smith/dasm-test => generated 355 bytes in 17 msecs (HTTP/1.1 200) 5 headers in 142 bytes (1 switches on core 2) GET GetNamedContainerView token before split BEARER 099e260ba86feb7664be19378fe40273da8c7a9e Found a token in validdate_token() token: 099e260ba86feb7664be19378fe40273da8c7a9e [pid: 59|app: 0|req: 7/36] 10.179.111.4 () {34 vars in 533 bytes} [Wed Jul 29 11:59:41 2020] GET /v1/containers/daniel.smith/dasm-test/sometthing => generated 541 bytes in 22 msecs (HTTP/1.1 200) 5 headers in 142 bytes (1 switches on core 0) GET PushNamedContainerView token before split BEARER 099e260ba86feb7664be19378fe40273da8c7a9e Found a token in validdate_token() token: 099e260ba86feb7664be19378fe40273da8c7a9e [pid: 59|app: 0|req: 8/37] 10.179.111.4 () {34 vars in 690 bytes} [Wed Jul 29 11:59:41 2020] GET /v1/images/daniel.smith/dasm-test/sometthing:sha256.b226c379e19853f16f1dc76b13b70deaf822afd6078c5bc3cd9b921d32cf7101?arch=amd64 => generated 565 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 142 bytes (1 switches on core 2) [pid: 59|app: 0|req: 9/38] 10.179.111.4 () {34 vars in 453 bytes} [Wed Jul 29 11:59:41 2020] GET /version => generated 58 bytes in 6 msecs (HTTP/1.1 200) 5 headers in 141 bytes (1 switches on core 3) POST RequestPushImageFileView token before split BEARER 099e260ba86feb7664be19378fe40273da8c7a9e Found a token in validdate_token() token: 099e260ba86feb7664be19378fe40273da8c7a9e Internal Server Error: /v2/imagefile/2 Traceback (most recent call last): File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 34, in inner response = get_response(request) File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 115, in _get_response response = self.process_exception_by_middleware(e, request) File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 113, in _get_response response = wrapped_callback(request, *callback_args, callback_kwargs) File "/usr/local/lib/python3.5/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view return view_func(*args, *kwargs) File "/usr/local/lib/python3.5/site-packages/django/views/generic/base.py", line 71, in view return self.dispatch(request, args, kwargs) File "/usr/local/lib/python3.5/site-packages/ratelimit/mixins.py", line 58, in dispatch )(super(RatelimitMixin, self).dispatch)(*args, kwargs) File "/usr/local/lib/python3.5/site-packages/ratelimit/decorators.py", line 30, in _wrapped return fn(*args, *kw) File "/usr/local/lib/python3.5/site-packages/rest_framework/views.py", line 505, in dispatch response = self.handle_exception(exc) File "/usr/local/lib/python3.5/site-packages/rest_framework/views.py", line 465, in handle_exception self.raise_uncaught_exception(exc) File "/usr/local/lib/python3.5/site-packages/rest_framework/views.py", line 476, in raise_uncaught_exception raise exc File "/usr/local/lib/python3.5/site-packages/rest_framework/views.py", line 502, in dispatch response = handler(request, args, kwargs) File "./shub/apps/library/views/images.py", line 356, in post expires=timedelta(minutes=MINIO_SIGNED_URL_EXPIRE_MINUTES), File "/usr/local/lib/python3.5/site-packages/minio/api.py", line 1563, in presigned_put_object expires) File "/usr/local/lib/python3.5/site-packages/minio/api.py", line 1502, in presigned_url request_date=request_date) File "/usr/local/lib/python3.5/site-packages/minio/signer.py", line 81, in presign_v4 raise InvalidArgumentError('Invalid access_key and secret_key.') minio.error.InvalidArgumentError: InvalidArgumentError: message: Invalid access_key and secret_key. [pid: 59|app: 0|req: 10/39] 10.179.111.4 () {36 vars in 497 bytes} [Wed Jul 29 11:59:41 2020] POST /v2/imagefile/2 => generated 850 bytes in 40 msecs (HTTP/1.1 500) 4 headers in 142 bytes (1 switches on core 1)

Thus its not "me" that is chosing to use that endpoint - im looking in the singuliarity API to see how to force it to use a configured endpoint (or to override it).

Cheers, D

[vsoch]

Thanks for the details! A lot of it goes over my head, but I'm definitely here to learn and help you as much as I can!

i believe that somewhere in the MINIO installation part there was something to add for 7 minutes or something.. on successive builds/restarts - i suspect that this might have expired? is that mc call to add a signed endpoint to minio (myminio in the docs) you are meaning in the CSG (i havent had a chance to look yet so if yet, you can leave me to find it :P)

I'm not sure about what you are referring to here, at least without https I've been able to restart the containers without any issue.

and i have updated my .sregistry and i coped the new token in after loging in

For the library endpoint you don't need the .sregistry file, that's for an older push without it.

File "/usr/local/lib/python3.5/site-packages/minio/signer.py", line 81, in presign_v4 raise InvalidArgumentError('Invalid access_key and secret_key.') minio.error.InvalidArgumentError: InvalidArgumentError: message: Invalid access_key and secret_key. [pid: 59|app: 0|req: 10/39] 10.179.111.4 () {36 vars in 497 bytes} [Wed Jul 29 11:59:41 2020] POST /v2/imagefile/2 => generated 850 bytes in 40 msecs (HTTP/1.1 500) 4 headers in 142 bytes (1 switches on core 1)

This is an issue with the key/secret for the internal client. What do you mean it's not you choosing to use it? As I suggested before, I think the core of this issue is with the ssl setup, because it works without a hitch with http.

Writing this, the use case of having instances of sregistry deployed globally and rather than using a disk/network fil PVC in k8s, that when SREG is deployed it can be configured to use many different Cloud Provide Object storage (fast and cheap) ) - maybe a "feature request"

Indeed! Singularity Hub (different but similar) has all storage interactions directly from Google Storage, which basically doesn't stress the host running it - that host just generates a signed URL and sends it off! :)

[lmcdasm]

A bit more of an update

We can see that the MINIO Access Key error is still being thrown. we can see that in the collection, the builds are actually "there"

as seen in this pic:

if we go in the collection we see they the builds are not really there though (and this makes sense based on the client debug, since the call to create the container label and hash and such all happens before hand).. All good>

When i reanable your make bucket call, it falls with this call to host/testbucket and i dont get any futher..

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.5/site-packages/django/core/handlers/wsgi.py", line 141, in call response = self.get_response(request) File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 75, in get_response response = self._middleware_chain(request) File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 36, in inner response = response_for_exception(request, exc) File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 90, in response_for_exception response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info()) File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 128, in handle_uncaught_exception callback, param_dict = resolver.resolve_error_handler(500) File "/usr/local/lib/python3.5/site-packages/django/urls/resolvers.py", line 597, in resolve_error_handler callback = getattr(self.urlconf_module, 'handler%s' % view_type, None) File "/usr/local/lib/python3.5/site-packages/django/utils/functional.py", line 80, in get res = instance.dict[self.name] = self.func(instance) File "/usr/local/lib/python3.5/site-packages/django/urls/resolvers.py", line 577, in urlconf_module return import_module(self.urlconf_name) File "/usr/local/lib/python3.5/importlib/init.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 985, in _gcd_import File "", line 968, in _find_and_load File "", line 957, in _find_and_load_unlocked File "", line 673, in _load_unlocked File "", line 697, in exec_module File "", line 222, in _call_with_frames_removed File "./shub/urls.py", line 19, in from shub.apps.library import urls as library_urls File "./shub/apps/library/urls.py", line 12, in from shub.apps.library import views File "./shub/apps/library/views/init.py", line 2, in from .images import ( File "./shub/apps/library/views/images.py", line 33, in from .minio import ( File "./shub/apps/library/views/minio.py", line 62, in if not minioClient.bucket_exists(MINIO_BUCKET): File "/usr/local/lib/python3.5/site-packages/minio/api.py", line 453, in bucket_exists self._url_open('HEAD', bucket_name=bucket_name) File "/usr/local/lib/python3.5/site-packages/minio/api.py", line 2080, in _url_open object_name).get_exception() minio.error.ResponseError: ResponseError: code: BadRequest, message: Bad Request, bucket_name: testbucket, object_name: None, request_id: , host_id: , region: [pid: 59|app: 0|req: 1/2] 10.3.103.8 () {50 vars in 1155 bytes} [Wed Jul 29 15:32:12 2020] GET / => generated 0 bytes in 1598 msecs (HTTP/1.1 500) 0 headers in 0 bytes (0 switches on core 0)

[vsoch]

You are trying to debug layers of errors, but the core issue is that the make bucket command is failing. As I already suggested, you should interactively produce this error, and provide the details of your SSL setup to the minio crew and get feedback. You of course aren't going to have anything upstream working if the basic initial connection to create a bucket doesn't work. Feel free to cc me on the minio issue, thanks!

[lmcdasm]

heya.. yup.. im working through it now. and will let you know (update ticket here and upstream) once i understand that part.

Thanks again, will keep ya posted on things.. its not "that far away"

D

[lmcdasm]

Ok.. so to recap where we are at (sorry for the delay, i had to find a machine with 15G swap to setup the image)

Having put the credentials right into the minioClient and minioExternalClient calls, seems to have got things working end to end when its legacy, but below you see i get a new error with "missing credentials". and now im to the

res = s3.create line that you outlined, so im doing some debugging again in images.py.. something is getting lost in the credential passing, since they are coded into minio.py, so its the use of the MINIO_ACCESS and MINIO_SECRET - when i added the creds right into the session instantiation i think it will work (building now, will know shortly).

for the test of large files, here is what i have done>

• made a 5GB file - bigfile.file (dd if=/dev/urandom of=target-file bs=5M count=1000) _ better to use urandon so we dont compress
• using a simple singularity def file, I build up the container.

My SING DEF:

Bootstrap: docker From: ubuntu:18.04

%setup

%files ./target-file /target-file

%environment export LISTEN_PORT=12345 export LC_ALL=C

%post apt-get update && apt-get install -y netcat NOW=date echo "export NOW=\"${NOW}\"" >> $SINGULARITY_ENVIRONMENT

%runscript echo "Container was created $NOW" echo "Arguments received: $*" exec echo "$@"

%startscript nc -lp $LISTEN_PORT

%test grep -q NAME=\"Ubuntu\" /etc/os-release if [ $? -eq 0 ]; then echo "Container base is Ubuntu as expected." else echo "Container base is not Ubuntu." fi

%labels Author d@sylabs.io Version v0.0.1

%help This is a demo container used to illustrate a def file that uses all supported sections.

MY CONTAINER :

[root@ip-0AB36F04 ~]# ls -tlr total 5334776 -rw-r--r--. 1 root root 785 Jul 31 00:47 big_singularity.sing -rwxr-xr-x. 1 root root 5290553344 Jul 31 01:03 bigcontainer.sif You have new mail in /var/spool/mail/root [root@ip-0AB36F04 ~]#

Found a token in validdate_token() token: 7829288b4e12e1d1ac68e4a7c30854d57f2d3e27 Internal Server Error: /v2/imagefile/8/_multipart Traceback (most recent call last): File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 34, in inner response = get_response(request) File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 115, in _get_response response = self.process_exception_by_middleware(e, request) File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 113, in _get_response response = wrapped_callback(request, *callback_args, callback_kwargs) File "/usr/local/lib/python3.5/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view return view_func(*args, *kwargs) File "/usr/local/lib/python3.5/site-packages/django/views/generic/base.py", line 71, in view return self.dispatch(request, args, kwargs) File "/usr/local/lib/python3.5/site-packages/rest_framework/views.py", line 505, in dispatch response = self.handle_exception(exc) File "/usr/local/lib/python3.5/site-packages/rest_framework/views.py", line 465, in handle_exception self.raise_uncaught_exception(exc) File "/usr/local/lib/python3.5/site-packages/rest_framework/views.py", line 476, in raise_uncaught_exception raise exc File "/usr/local/lib/python3.5/site-packages/rest_framework/views.py", line 502, in dispatch response = handler(request, *args, kwargs) File "./shub/apps/library/views/images.py", line 293, in post res = s3.create_multipart_upload(Bucket=MINIO_BUCKET, Key=storage) File "/usr/local/lib/python3.5/site-packages/botocore/client.py", line 316, in _api_call return self._make_api_call(operation_name, kwargs) File "/usr/local/lib/python3.5/site-packages/botocore/client.py", line 622, in _make_api_call operation_model, request_dict, request_context) File "/usr/local/lib/python3.5/site-packages/botocore/client.py", line 641, in _make_request return self._endpoint.make_request(operation_model, request_dict) File "/usr/local/lib/python3.5/site-packages/botocore/endpoint.py", line 102, in make_request return self._send_request(request_dict, operation_model) File "/usr/local/lib/python3.5/site-packages/botocore/endpoint.py", line 132, in _send_request request = self.create_request(request_dict, operation_model) File "/usr/local/lib/python3.5/site-packages/botocore/endpoint.py", line 116, in create_request operation_name=operation_model.name) File "/usr/local/lib/python3.5/site-packages/botocore/hooks.py", line 356, in emit return self._emitter.emit(aliased_event_name, kwargs) File "/usr/local/lib/python3.5/site-packages/botocore/hooks.py", line 228, in emit return self._emit(event_name, kwargs) File "/usr/local/lib/python3.5/site-packages/botocore/hooks.py", line 211, in _emit response = handler(**kwargs) File "/usr/local/lib/python3.5/site-packages/botocore/signers.py", line 90, in handler return self.sign(operation_name, request) File "/usr/local/lib/python3.5/site-packages/botocore/signers.py", line 160, in sign auth.add_auth(request) File "/usr/local/lib/python3.5/site-packages/botocore/auth.py", line 357, in add_auth raise NoCredentialsError botocore.exceptions.NoCredentialsError: Unable to locate credentials [pid: 59|app: 0|req: 29/79] 10.179.111.4 () {36 vars in 517 bytes} [Thu Jul 30 20:06:23 2020] POST /v2/imagefile/8/_multipart => generated 850 bytes in 412 msecs (HTTP/1.1 500) 4 headers in 142 bytes (1 switches on core 2)

THE SINGULARITY (EXTERNAL - you can see tries to use the MULTIPART) call

@ip-0AB36F04 ~]# singularity --debug push -U bigcontainer.sif library://daniel.smith/dasm-test/biggie:latest DEBUG [U=0,P=123925] persistentPreRun() Singularity version: 3.6.0-1.el7 DEBUG [U=0,P=123925] persistentPreRun() Parsing configuration file /etc/singularity/singularity.conf DEBUG [U=0,P=123925] handleConfDir() /root/.singularity already exists. Not creating. DEBUG [U=0,P=123925] handleRemoteConf() Ensuring file permission of 0600 on /root/.singularity/remote.yaml WARNING [U=0,P=123925] LibraryPush() Skipping container verifying DEBUG [U=0,P=123925] UploadImage() Image hash computed as 916c204a48e5906bde38f6ada2e3d33bc2c02a66517a92e322b7b876602fe89c DEBUG [U=0,P=123925] apiGet() apiGet calling v1/entities/daniel.smith DEBUG [U=0,P=123925] apiGet() apiGet calling v1/collections/daniel.smith/dasm-test DEBUG [U=0,P=123925] apiGet() apiGet calling v1/containers/daniel.smith/dasm-test/biggie DEBUG [U=0,P=123925] apiGet() apiGet calling v1/images/daniel.smith/dasm-test/biggie:sha256.916c204a48e5906bde38f6ada2e3d33bc2c02a66517a92e322b7b876602fe89c?arch=amd64 DEBUG [U=0,P=123925] postFileWrapper() Now uploading to the library DEBUG [U=0,P=123925] postFileV2() Attempting to use multipart uploader DEBUG [U=0,P=123925] startMultipartUpload() startMultipartUpload calling v2/imagefile/8/_multipart DEBUG [U=0,P=123925] apiCreate() apiCreate calling v2/imagefile/8/_multipart 0.0b / 4.9GiB [------------------------------------------------------------------------------] 0 % 0.0 b/s 0s DEBUG [U=0,P=123925] postFileV2Multipart() Error starting multipart upload: request did not succeed: http status code: 500 FATAL [U=0,P=123925] func50() Unable to push image to library: request did not succeed: http status code: 500 You have new mail in /var/spool/mail/root [root@ip-0AB36F04 ~]#

[vsoch]

Look at the core error, it's unable to find credentials. This seems like a new bug, so likely your environment variables aren't getting found for the Minio keys. The session that uses them is here https://github.com/singularityhub/sregistry/blob/89caaf05f91f63fa34e3f461eb469603d9f7ddcc/shub/apps/library/views/minio.py#L65.

[lmcdasm]

Hey there.

So, once i got the credentials, what i saw is that the Switch for MINIO_SSL in the s3.client part (just below session creation where you have the keys) was sending it to http://minio:9000 and this generated and error (Connection closed by peer).

im fixing it so its all https through and through and should see shortly.

closer and closer - but i think you're right, something is going on in the environment - im working my way backwards to that point

[lmcdasm]

and thanks so much for you patience as i step through this... gonna feel dumb when i find it, i think :P

[vsoch]

I’m going to feel elated that we figured it out! Well mostly you, I’m here for moral support :)

I’m going to be signing off for the evening, and I have a talk first thing tomorrow but should be around after that. Don’t forget to take some breaks on your end too!

[lmcdasm]

as will the 200 so grid devs that are waiting for me to launch this :)

have a good night and ill keep you updated (and youll see example / tutorial code pushes as well once its working : ))

[lmcdasm]

And Voila, we have working action HTTPS end to end (with Verfiy enabled).

there was another place at the end of the transaction

minioClient.remove_object that i switched to minioExternalClient and then it worked>

from what i understand, the idea was to have the calls use an internal Http(s)-less path and then external calls https - however, using the https://127.0.0.1 local definition becomes impossible and using the name 'minio' would really work.

here are the logs showing that its working :) - im going to run our rest rig and give it some paces and then think about perhaps some improvements that might allow for the 4 combinations to exist with internal/external (http/https http/http, https/http and https/https) a bit more readily.

Again, happy to have a working build!

SREG LOG:

SINGULARITY CLIENT CALL:

Cheers!

[vsoch]

Woohoo!! What great work!! I can't wait to see it! Now we see the POST that I was talking about - the multipartUploadPart bit that hits the _multipart url.

[vsoch]

Hey any updates? I expected a PR and a month has passed, are you doing ok?

[vsoch]

Closing issue, no response.