Unable to delete the containers using sregistry-cli

[ashwinidr23]

I am unable to delete the containers in sregistryhub using sregistry-cli. however i can delete using GUI. but since i want to delete more containers at once i would like to use sregistry-cli for this.

i get the following error when i try to delete the repository sregistry delete inadr/inadr-lolcow2:222 [client|registry] [database|sqlite:////home/inadr/.singularity/sregistry.db] Are you sure you want to remove inadr/inadr-lolcow2:222?y ERROR Beep boop! Internal Server Error: 500

these are the error logs docker-compose logs --tail=10 -f uwsgi Attaching to sregistry_uwsgi_1 uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/ratelimit/decorators.py", line 27, in _wrapped uwsgi_1 | increment=True) uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/ratelimit/utils.py", line 127, in is_ratelimited uwsgi_1 | usage = get_usage_count(request, group, fn, key, rate, method, increment) uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/ratelimit/utils.py", line 154, in get_usage_count uwsgi_1 | value = _SIMPLE_KEYSkey uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/ratelimit/utils.py", line 34, in uwsgi_1 | 'ip': lambda r: r.META['REMOTE_ADDR'], uwsgi_1 | AttributeError: 'Container' object has no attribute 'META' uwsgi_1 | [pid: 43|app: 0|req: 1070/2027] 144.199.213.236 () {44 vars in 805 bytes} [Tue Nov 17 04:45:21 2020] DELETE /api/container/inadr3/inadr3-lolcow2:222 => generated 850 bytes in 16 msecs (HTTP/1.1 500) 4 headers in 142 bytes (1 switches on core 2)

I tried to figure out for a solution online, but i could not, any pointers please?

PS: i am still using sregistry-cli and not singularity remote feature because i am having trouble configuring with HTTPS due to certificate issues which are beyond my control at the moment!

[vsoch]

hey @ashwinidr23 - I'd be happy to work on this for you! The error message above actually looks related to rate limiting. We could go back to this (what I consider deprecated) client, but another idea is to add the singularity delete library endpoint directly (which isn't currently implemented):

https://sylabs.io/guides/3.5/user-guide/cli/singularity_delete.html

I'm quite busy this week but I think I should be able to make some time to work on this later this month. In the mean time, I would strongly encourage you to figure out adding https! You can compile Singularity in a way to not require it, but it's much better that it's secure.

[ashwinidr23]

Thank you very much. Either the sregistry-cli or singularity delete end point should help. There were some certificate issues that I noticed the last time I tried to configure with HTTPS. I did not try later, but it is in my list to make it work.

[ashwinidr23]

hey @vsoch , I tried updating the sregistry version to latest version(1.1.29) and configuring the sregistry with HTTPS and here are few updates. i was getting minio.error.AccessDenied error i went through the known issues and found that i was getting the same error as mentioned in #301. only in my case uwsgi was not getting started instead of scheduler and worker. After going through this issue, i updated uwsgi part of docker-compose.yml file with "env_file:

• ./.minio-env"
  which resolved my problem.(i am not sure if this problem is just for me or the code needs to be updated.)

PS: i did not get this error with http

However- configuring the registry did not resolve the primary issue that i was facing (i.e., pulling/pushing/deleting images to sregistry using sregistry-cli or singularity) when i use sregistry-cli, push - works fine pull - works after i export "REQUESTS_CA_BUNDLE=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem " delete - does not work

-bash-4.2$ sregistry delete registry://adr23/lolcow:2 [client|registry] [database|sqlite:////var/lib/builder/.singularity/sregistry.db] Are you sure you want to remove adr23/lolcow:2?y ERROR Beep boop! Internal Server Error: 500 (i understand this is something to do with rate limit from your previous comment)

when i use singularity: i am able to add the remote end point and verify the token, however when i try to push i get this error push: _singularity push -U --library https://domain_name lolcowlatest.sif library://adr23/test/lolcow:1 WARNING: Skipping container verifying 0 B / 79.91 MiB [------------------------------------------------------------------------------------------------------] 0.00% 11s FATAL: Unable to push image to library: request did not succeed: http status code: 500 or _singularity push -U lolcowlatest.sif library://adr23/test/lolcow:v1 WARNING: Skipping container verifying 0 B / 79.91 MiB [-------------------------------------------------------------------------------------------------------] 0.00% 6s FATAL: Unable to push image to library: request did not succeed: http status code: 500

pull : works fine

i did not find any details about deleting the containers using singularity in the document here : https://singularityhub.github.io/sregistry/docs/client#pancakes-installation can you please help me with details or point me to a document for deleting a container using singularity?(since u mentioned sregistry-cli is almost "depreciated")

i want to be able to delete containers using command-line either using sregistry-cli/singularity(which i can automate rather than using GUI).

Thanks in advance!

[vsoch]

Okay let's work through your comments one by one! First, you are correct that the https docker-compose.yml file is missing the mount of the minio environment - I'll add that shortly in a PR. For sregistry, I don't consider sregistry-cli to be supported anymore - you should just be using Singularity. And then the 500 error indicates a server error, so you need to look at the uwsgi logs to see what that is.

[vsoch]

i did not find any details about deleting the containers using singularity in the document here : https://singularityhub.github.io/sregistry/docs/client#pancakes-installation can you please help me with details or point me to a document for deleting a container using singularity?(since u mentioned sregistry-cli is almost "depreciated")

As far as I know, deleting containers with the Singularity client is not supported. But you could ask the hpcng folks if this is changed!

[vsoch]

Missing minio env is added: https://github.com/singularityhub/sregistry/pull/334

[ashwinidr23]

Okay let's work through your comments one by one! First, you are correct that the https docker-compose.yml file is missing the mount of the minio environment - I'll add that shortly in a PR. For sregistry, I don't consider sregistry-cli to be supported anymore - you should just be using Singularity. And then the 500 error indicates a server error, so you need to look at the uwsgi logs to see what that is.

here is the log for push error singularity push -U lolcow_latest.sif library://builder/test/lolcow:v1 WARNING: Skipping container verifying 0 B / 79.91 MiB [------------------------------------------------------------------------------------------------------] 0.00% 10s FATAL: Unable to push image to library: request did not succeed: http status code: 500

docker-compose logs --tail=30 uwsgi uwsgi_1 | [pid: 63|app: 0|req: 35/49] () {34 vars in 448 bytes} [Wed Jan 13 04:43:00 2021] GET /version => generated 58 bytes in 1 msecs (HTTP/1.1 200) 5 headers in 141 bytes (1 switches on core 3) uwsgi_1 | POST RequestMultiPartPushImageFileView uwsgi_1 | Internal Server Error: /v2/imagefile/1/_multipart uwsgi_1 | Traceback (most recent call last): uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 34, in inner uwsgi_1 | response = get_response(request) uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 115, in _get_response uwsgi_1 | response = self.process_exception_by_middleware(e, request) uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 113, in _get_response uwsgi_1 | response = wrapped_callback(request, *callback_args, callback_kwargs) uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view uwsgi_1 | return view_func(*args, *kwargs) uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/django/views/generic/base.py", line 71, in view uwsgi_1 | return self.dispatch(request, args, kwargs) uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/rest_framework/views.py", line 505, in dispatch uwsgi_1 | response = self.handle_exception(exc) uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/rest_framework/views.py", line 465, in handle_exception uwsgi_1 | self.raise_uncaught_exception(exc) uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/rest_framework/views.py", line 476, in raise_uncaught_exception uwsgi_1 | raise exc uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/rest_framework/views.py", line 502, in dispatch uwsgi_1 | response = handler(request, *args, **kwargs) uwsgi_1 | File "./shub/apps/library/views/images.py", line 289, in post uwsgi_1 | res = s3.create_multipart_upload(Bucket=MINIO_BUCKET, Key=storage) uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/botocore/client.py", line 357, in _api_call uwsgi_1 | return self._make_api_call(operation_name, kwargs) uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/botocore/client.py", line 676, in _make_api_call uwsgi_1 | raise error_class(parsed_response, operation_name) uwsgi_1 | botocore.exceptions.ClientError: An error occurred (503) when calling the CreateMultipartUpload operation (reached max retries: 4): Service Unavailable uwsgi_1 | [pid: 63|app: 0|req: 36/50] () {36 vars in 512 bytes} [Wed Jan 13 04:43:00 2021] POST /v2/imagefile/1/_multipart => generated 850 bytes in 9732 msecs (HTTP/1.1 500) 4 headers in 142 bytes (1 switches on core 1)

and when i look at the sregistry GUI, i find a build created after this but when i navigate to the container, it is empty. since the collection is not empty i am not able to delete the collection even on GUI now. please find the screen shots attached

-bash-4.2$sregistry search gives the name of the file which was "not pushed" -bash-4.2$ sregistry search [client|registry] [database|sqlite:////var/lib/builder/.singularity/sregistry.db] Collections 1 test/lolcow:DUMMY-ed186922-9d3c-4537-9399-51b622fdf55b https://hpcs-sregistry.shell.com/containers/1

contents in images folder [root@myvm images]# ls _upload

[vsoch]

That error message tells us that the minio container is not running. Your next thing to debug is:

docker-compose logs --tail=30 minio

As I said before, the sregistry-cli command is not expected to work.

[ashwinidr23]

[root@myvm sregistry]# docker-compose logs --tail=30 minio Attaching to sregistry_minio_1 minio_1 | Attempting encryption of all config, IAM users and policies on MinIO backend minio_1 | Endpoint: http://172.17.0.3:9000 http://127.0.0.1:9000 minio_1 | minio_1 | Browser Access: minio_1 | http://172.17.0.3:9000 http://127.0.0.1:9000 minio_1 | minio_1 | Object API (Amazon S3 compatible): minio_1 | Go: https://docs.min.io/docs/golang-client-quickstart-guide minio_1 | Java: https://docs.min.io/docs/java-client-quickstart-guide minio_1 | Python: https://docs.min.io/docs/python-client-quickstart-guide minio_1 | JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide minio_1 | .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide

[ashwinidr23]

i did not find any errors in minio container

[vsoch]

Have you changed anything about the setup? If you follow the instructions in the documentation to use the mc binary to look at logs, do you see that the request is hitting it?

[ashwinidr23]

i have not changed anything in the setup. ./mc config host ls - this was successfull

[root@myvm sregistry]# ./mc config host ls gcs URL : https://storage.googleapis.com AccessKey : YOUR-ACCESS-KEY-HERE SecretKey : YOUR-SECRET-KEY-HERE API : S3v2 Path : dns

local URL : http://localhost:9000 AccessKey : SecretKey : API : Path : auto

myminio URL : http://127.0.0.1:9000 AccessKey : access_key_here SecretKey : secret_key_here API : s3v4 Path : auto

play URL : https://play.min.io AccessKey : access_key_here SecretKey : secret_key_here API : S3v4 Path : auto

s3 URL : https://s3.amazonaws.com AccessKey : YOUR-ACCESS-KEY-HERE SecretKey : YOUR-SECRET-KEY-HERE API : S3v4 Path : dns

but i do not get any logs. i tried to push a container while running the below command. but i cud not get any logs ! [root@myvm sregistry]# ./mc admin trace -v myminio ^Cmc: Unable to listen to http trace. Canceling upon user request.

[vsoch]

That’s the issue then - the request never reaches the Minio server, and that’s exactly what the 500 error is telling you. There is something off about your setup and you’ll have to debug. I would try shelling into the various containers, Minio and uwsgi, and seeing if you can see (ping) the others based on their hostnames that are supposed to be setup by Docker compose. You can also check the /etc/hosts file. If that works, the next step is to interactively create the Minio client from the uwsgi container (the file Minio.py) and test interactions with your Minio container. I suspect the first test won’t work and there is something off about the networking for your containers, but that’s just a hunch.

[ashwinidr23]

do you mean to login to uwsgi container and install mc there and try to run trace there? can you please give me some details about the minio.py file as i have not worked with minio before?

[vsoch]

I mean to shell into the uwsgi container, and then get an interactive python shell (python manage.py shell) and use the minio.py file as example code to create connections to the server (the minioClient and minioExternalClient) and then test different uses of it manually from the images.py file, and keep the minio logs open in another shell to always see the server response (if it even hits, which from what you've shown me it's not). You will likely trigger that same error, and you should debug from there. But first you should test the basic networking, because if it's an issue with docker-compose not allowing the containers to see one another, that's one level up and should be addressed first. I'm telling you exactly how I would debug this if it was sitting in front of me.

[ashwinidr23]

Here is my observation so far, network seems to be working fine, i am able to ping the containers from inside one container to another, /etc/hosts also looks ok. like mentioned in the setup documents if i change the minio access key and secret key , i am unable to get the sregistry up and running logs from docker-compose: minio_1 | minio_1 | minio_1 | Attempting rotation of encrypted config, IAM users and policies on MinIO with newly supplied credentials minio_1 | ERROR Unable to initialize the server: Unable to initialize sub-systems: Unable to migrate all config at .minio.sys/co nfig/: Invalid rotating credentials sregistry_minio_1 exited with code 1

[root@myvm sregistry]# ./mc config host add myminio http://127.0.0.1:9000 $MINIO_ACCESS_KEY $MINIO_SECRET_KEY Enter Access Key: newaccesskey Enter Secret Key: mc: Unable to initialize new alias from the provided credentials. Get "http://127.0.0.1:9000/probe-bucket-sign-kumm11bw9rxn/?location=": dial tcp 127.0.0.1:9000: connect: connection refused.

however if i do not change the access key and secret key , i can get the sregistry up and running, but singularity pull/push does not seem to work as none of the requests are reaching minio in this case.