search

singularityhub / sregistry-cli

Singularity Global Client for container management
https://singularityhub.github.io/sregistry-cli/
Mozilla Public License 2.0
14 stars 18 forks source link

unable to connect to gcr.io #185

Closed kanthsai143 closed 5 years ago

kanthsai143 commented 5 years ago

Describe the bug unable to connect to gcr.io repo

first try error: sregistry pull docker://gcr.io/myrepo/container:latest Traceback (most recent call last): File "/bin/sregistry", line 11, in load_entry_point('sregistry==0.1.37', 'console_scripts', 'sregistry')() File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/client/init.py", line 368, in main subparser=subparsers[args.command]) File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/client/pull.py", line 42, in main save=do_save) File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/main/docker/pull.py", line 74, in pull kwargs=kwargs) File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/main/docker/pull.py", line 118, in _pull file_name = self._get_storage_name(names) File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/main/base/settings.py", line 121, in get_storage_name storage_folder = "%s/%s" %(self.storage, storage_folder) AttributeError: 'Client' object has no attribute 'storage'

Second try error sregistry pull docker://gcr.io/myrepo/container:latest

ERROR Your credentials are expired! Unauthorized: 401 ERROR Your credentials are expired! Unauthorized: 401

Third try error sregistry pull docker://gcr.io/myrepo/container:latest

Traceback (most recent call last): File "/bin/sregistry", line 11, in load_entry_point('sregistry==0.1.37', 'console_scripts', 'sregistry')() File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/client/init.py", line 310, in main from sregistry.main import Client as cli File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/main/init.py", line 118, in Client = get_client() File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/main/init.py", line 112, in get_client cli = Client() File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/main/docker/init.py", line 42, in init self._update_secrets() File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/main/docker/init.py", line 143, in _update_secrets if credentials is not None and auth is None: UnboundLocalError: local variable 'auth' referenced before assignment

To Reproduce First try reproduce: gcr.io login was configured through a service account so activated backend docker and tried to do a pull got the following error

second try reproduce: /usr/bin/pip3.6 install sregistry[google-storage]

third try reproduce: /usr/bin/pip3.6 install sregistry I have entered gcr.io credentials now export SREGISTRY_DOCKERHUB_SECRETS=/path/config.json

Expected behavior Should be able to push/pull from/to gcr.io with sregistry. Docker pull works all this time

docker pull gcr.io/myrepo/container:latest

If applicable, add versions and screenshots to help explain your problem. latest versions of docker containers

vsoch commented 5 years ago

Did you mean to post this on https://github.com/singularityhub/sregistry-cli ? The server here wouldn't be relevant for gcr.io.

kanthsai143 commented 5 years ago

yeah. can you move it?

vsoch commented 5 years ago

yep! Just wanted to check first :)

vsoch commented 5 years ago

Where are you exporting the SREGISTRY_CLIENT variable?

vsoch commented 5 years ago

Can you please provide the full set of commands you did so I can reproduce the error exactly - the error messages don't give me this information.

vsoch commented 5 years ago

okay, without that information, here is how we can debug this. First, can you please try pulling a gcr.io image that is public? Here is one that I just pulled:

$ sregistry pull docker://gcr.io/cloud-builders/gcloud-slim
[client|docker] [database|sqlite:////home/vanessa/.singularity/sregistry.db]
Progress |===================================| 100.0% 
Progress |===================================| 100.0% 
Progress |====|------------------------------|  11.8% 
Progress |===================================| 100.0% 
[4/4] |===================================| 100.0% 
Exploding /usr/local/libexec/singularity/bootstrap-scripts/environment.tar
Exploding /home/vanessa/.singularity/sregistry-tmp.z73b04yr/docker/sha256:75f546e73d8b1fa36e85508de5819396d52b75d98d85987fa2d2b511682c9dc1.tar.gz
Exploding /home/vanessa/.singularity/sregistry-tmp.z73b04yr/docker/sha256:0f3bb76fc39050f39722302e4e90894f704ec844aa67ba7609ce7190d948778a.tar.gz
Exploding /home/vanessa/.singularity/sregistry-tmp.z73b04yr/docker/sha256:3c2cba919283a210665e480bcbf943eaaf4ed87a83f02e81bb286b8bdead0e75.tar.gz
Exploding /home/vanessa/.singularity/sregistry-tmp.z73b04yr/docker/sha256:f2f801d7166b7c66a6e32e8047e5cfc5ade4444e0c59b7744468703a6a98bebf.tar.gz
Exploding /home/vanessa/.singularity/sregistry-tmp.z73b04yr/docker/metadata/sha256:9c5385d7a5a6c48c87a81dd786a9f1db0d8775763c8d9aa522f2f0e857b6fbb4.tar.gz
WARNING: Authentication token file not found : Only pulls of public images will succeed
INFO:    Starting build...
INFO:    Creating SIF file...
INFO:    Build complete: /home/vanessa/.singularity/shub/cloud-builders-gcloud-slim-latest.sif
WARNING: Authentication token file not found : Only pulls of public images will succeed
INFO:    Starting build...
INFO:    Creating SIF file...
INFO:    Build complete: /home/vanessa/.singularity/shub/cloud-builders-gcloud-slim-latest.sif
[container][new] cloud-builders/gcloud-slim-latest-latest@23c43e89a7ebfe7f0d78fdcefca9c2f3
Success! /home/vanessa/.singularity/shub/cloud-builders-gcloud-slim-latest-latest@23c43e89a7ebfe7f0d78fdcefca9c2f3.sif

I didn't export anything special (the docker:// uri determined the client). Let's confirm that the above works for you, and then discuss how you are trying to authenticate.

kanthsai143 commented 5 years ago

Unable to activate docker now $ sregistry backend activate docker Traceback (most recent call last): File "/bin/sregistry", line 11, in load_entry_point('sregistry==0.1.37', 'console_scripts', 'sregistry')() File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/client/init.py", line 310, in main from sregistry.main import Client as cli File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/main/init.py", line 118, in Client = get_client() File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/main/init.py", line 112, in get_client cli = Client() File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/main/docker/init.py", line 42, in init self._update_secrets() File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/main/docker/init.py", line 143, in _update_secrets if credentials is not None and auth is None: UnboundLocalError: local variable 'auth' referenced before assignment

$ sregistry pull docker://gcr.io/cloud-builders/gcloud-slim Traceback (most recent call last): File "/bin/sregistry", line 11, in load_entry_point('sregistry==0.1.37', 'console_scripts', 'sregistry')() File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/client/init.py", line 310, in main from sregistry.main import Client as cli File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/main/init.py", line 118, in Client = get_client() File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/main/init.py", line 112, in get_client cli = Client() File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/main/docker/init.py", line 42, in init self._update_secrets() File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/main/docker/init.py", line 143, in _update_secrets if credentials is not None and auth is None: UnboundLocalError: local variable 'auth' referenced before assignment

vsoch commented 5 years ago

You shouldn't need to activate anything - using the docker:// uri will specify that is the client.

kanthsai143 commented 5 years ago

Okay. I tried that too. Pasting the output again here

$ sregistry pull docker://gcr.io/cloud-builders/gcloud-slim Traceback (most recent call last): File "/bin/sregistry", line 11, in load_entry_point('sregistry==0.1.37', 'console_scripts', 'sregistry')() File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/client/init.py", line 310, in main from sregistry.main import Client as cli File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/main/init.py", line 118, in Client = get_client() File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/main/init.py", line 112, in get_client cli = Client() File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/main/docker/init.py", line 42, in init self._update_secrets() File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/main/docker/init.py", line 143, in _update_secrets if credentials is not None and auth is None: UnboundLocalError: local variable 'auth' referenced before assignment

vsoch commented 5 years ago

okay I'm going to open up a branch to fix these small issues - you can pull and install from it to update and help test:

git clone -b fix/gcr.io-docker https://www.github.com/singularityhub/sregistry-cli
cd sregistry-cli
python setup.py install

The "auth" referenced before assignment issue should be fixed there. I'll push other fixes as we run into them.

vsoch commented 5 years ago

Once you install, please try the pull again! The reason is because auth is only defined if a username and password are found:

        # First try for SINGULARITY exported, then try sregistry
        username = self._get_setting('SINGULARITY_DOCKER_USERNAME')
        password = self._get_setting('SINGULARITY_DOCKER_PASSWORD')
        username = self._get_setting('SREGISTRY_DOCKERHUB_USERNAME', username)
        password = self._get_setting('SREGISTRY_DOCKERHUB_PASSWORD', password)

        # Option 1: the user exports username and password
        if username is not None and password is not None:
            auth = basic_auth_header(username, password)
            self.headers.update(auth)

So you had one / both of username/password set to None, auth wasn't defined, and the next block (needing the variable auth) fails.

kanthsai143 commented 5 years ago

Got the following error now :/ $ sregistry pull docker://gcr.io/cloud-builders/gcloud-slim Traceback (most recent call last): File "/bin/sregistry", line 11, in load_entry_point('sregistry==0.1.37', 'console_scripts', 'sregistry')() File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/client/init.py", line 310, in main from sregistry.main import Client as cli File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/main/init.py", line 118, in Client = get_client() File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/main/init.py", line 112, in get_client cli = Client() File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/main/docker/init.py", line 42, in init self._update_secrets() File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/main/docker/init.py", line 145, in _update_secrets credentials = read_json(credentials) File "/usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/utils/fileio.py", line 328, in read_json data = json.load(filey) File "/usr/lib64/python3.6/json/init.py", line 299, in load parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw) File "/usr/lib64/python3.6/json/init.py", line 354, in loads return _default_decoder.decode(s) File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

vsoch commented 5 years ago

It looks like the value from your sregistry secrets was set to None, so it can't be decoded. This would not happen if we didn't have the original bug, but there should probably be an extra check for it anyway. This is what is telling me it's trying to parse None:

raise JSONDecodeError("Expecting value", s, err.value) from None

Did you export a path to your credentials file, and if so, can you show me and try unsetting it? And then look in the file to see if there are any values of None?

kanthsai143 commented 5 years ago

doing unset worked. the error below can be ignored I guess.

$ sregistry pull docker://gcr.io/cloud-builders/gcloud-slim [client|docker] [database|sqlite:////root/.singularity/sregistry.db] Progress |===================================| 100.0% Progress |===================================| 100.0% Progress |===================================| 100.0% Progress |===================================| 100.0% [4/4] |===================================| 100.0% Exploding /usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/main/docker/environment.tar Exploding /root/.singularity/sregistry-tmp.lwfphaa/docker/sha256:75f546e73d8b1fa36e85508de5819396d52b75d98d85987fa2d2b511682c9dc1.tar.gz Exploding /root/.singularity/sregistry-tmp.lwfphaa/docker/sha256:0f3bb76fc39050f39722302e4e90894f704ec844aa67ba7609ce7190d948778a.tar.gz Exploding /root/.singularity/sregistry-tmp.lwfphaa/docker/sha256:3c2cba919283a210665e480bcbf943eaaf4ed87a83f02e81bb286b8bdead0e75.tar.gz Exploding /root/.singularity/sregistry-tmp.lwfphaa/docker/sha256:f2f801d7166b7c66a6e32e8047e5cfc5ade4444e0c59b7744468703a6a98bebf.tar.gz Exploding /root/.singularity/sregistry-tmp.lwfphaa_/docker/metadata/sha256:9c5385d7a5a6c48c87a81dd786a9f1db0d8775763c8d9aa522f2f0e857b6fbb4.tar.gz ERROR: Unknown command 'build' ERROR: Unknown command 'build' ERROR Return Code 1: ("ERROR: Unknown command 'build'\n",) Downloading with native Singularity, please wait...

Singularity Registry Global Client v0.1.37 [docker] usage: sregistry [-h] [--debug] [--quiet] {version,backend,shell,images,inspect,get,add,mv,rename,rm,rmi,pull}

vsoch commented 5 years ago

It looks like you are using an older version of singularity - which one? "Build" was added after bootstrap, and it looks like you might have bootstrap.

kanthsai143 commented 5 years ago

singularity --version 2.2.1

vsoch commented 5 years ago

oh there you go! So - you need to update imminently. I don't always say this, but there were some fairly big security issues with versions that old. You should upgrade at least to 2.6.0.

vsoch commented 5 years ago

Once you've upgraded:

Then I'll see if I can help to figure out if/how it could lead to error.

kanthsai143 commented 5 years ago

It worked after installing new version of singularity

$ sregistry pull docker://gcr.io/cloud-builders/gcloud-slim [client|docker] [database|sqlite:////root/.singularity/sregistry.db] Progress |===================================| 100.0% Progress |===================================| 100.0% Progress |===================================| 100.0% Progress |===================================| 100.0% [4/4] |===================================| 100.0% Exploding /usr/lib/python3.6/site-packages/sregistry-0.1.37-py3.6.egg/sregistry/main/docker/environment.tar Exploding /root/.singularity/sregistry-tmp.t5lieqtk/docker/sha256:75f546e73d8b1fa36e85508de5819396d52b75d98d85987fa2d2b511682c9dc1.tar.gz Exploding /root/.singularity/sregistry-tmp.t5lieqtk/docker/sha256:0f3bb76fc39050f39722302e4e90894f704ec844aa67ba7609ce7190d948778a.tar.gz Exploding /root/.singularity/sregistry-tmp.t5lieqtk/docker/sha256:3c2cba919283a210665e480bcbf943eaaf4ed87a83f02e81bb286b8bdead0e75.tar.gz Exploding /root/.singularity/sregistry-tmp.t5lieqtk/docker/sha256:f2f801d7166b7c66a6e32e8047e5cfc5ade4444e0c59b7744468703a6a98bebf.tar.gz Exploding /root/.singularity/sregistry-tmp.t5lieqtk/docker/metadata/sha256:9c5385d7a5a6c48c87a81dd786a9f1db0d8775763c8d9aa522f2f0e857b6fbb4.tar.gz Building image from sandbox: /tmp/sregistry-sandbox.90_gbbcj Building Singularity image... Singularity container built: /root/.singularity/shub/cloud-builders-gcloud-slim-latest.sif Cleaning up... [container][new] cloud-builders/gcloud-slim-latest-latest@1218023fb9896edeb07d0cebd529d48a Success! /root/.singularity/shub/cloud-builders-gcloud-slim-latest-latest@1218023fb9896edeb07d0cebd529d48a.sif

Now back to private gcr.io

Is it possible to use a service account instead of username and password while connecting gcr.io? Also, I just realized that when I set gcr cerdentials public gcr.io is not accesible

vsoch commented 5 years ago

hmm, so this is uncharted territory - if gcr.io works like a Docker Registry, then you should be able to export your username and password:

export SINGULARITY_DOCKER_USERNAME=xxxxxxxxxxx
export SINGULARITY_DOCKER_PASSWORD=xxxxxxxxxxxxxx

Those would be environment variable ways to do this. There is also this --login flag you could try (native with singularity) -> https://www.sylabs.io/guides/3.0/user-guide/singularity_and_docker.html#authentication-via-interactive-login

To start, let's forget about sregistry-cli, and see if you can get a pull with your credentials working with just singularity. If you can't get that working, then you need to report an issue there first (and then come back here). If you do get it working, then we need to make sure the functionality works with sregistry client.

kanthsai143 commented 5 years ago

When I do docker pull gcr.io/myprivaterepo/container:latest

It works fine with because I have a service account configured through gcloud authentication.

But it doesn't work for sregistry pull gcr.io/myprivaterepo/container:latest because of environment variables. If I am using this on different systems, I don't want to expose username and password through environment variables

vsoch commented 5 years ago

As I mentioned, please try the pull with Singularity native. The client is just a wrapper to that.

kanthsai143 commented 5 years ago

Interesting! I tried the same public container. But had this issue.

$ singularity pull docker://gcr.io/cloud-builders/gcloud-slim

WARNING: pull for Docker Hub is not guaranteed to produce the WARNING: same image on repeated pull. Use Singularity Registry WARNING: (shub://) to pull exactly equivalent images. Docker image path: gcr.io/cloud-builders/gcloud-slim:latest ERROR MANIFEST_UNKNOWN: Manifest with tag 'latest' has media type 'application/vnd.docker.distribution.manifest.v2+json', but client accepts 'application/json'. Cleaning up... ERROR: pulling container failed!

vsoch commented 5 years ago

Here is the core of the issue! In that gcr.io is a Docker Registry, we need to get it working first with native Singularity. Please open an issue at https://github.com/sylabs/singularity/issues and then when it's working with Singularity, we can bring discussion back here.

vsoch commented 5 years ago

Any updates on this front @kanthsai143 or can I close the issue?

kanthsai143 commented 5 years ago

Please close. Thanks for your help

On Wed, May 29, 2019, 7:39 AM Vanessasaurus notifications@github.com wrote:

Any updates on this front @kanthsai143 https://github.com/kanthsai143 or can I close the issue?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/singularityhub/sregistry-cli/issues/185?email_source=notifications&email_token=AEVAWRM4QNCA5KNKPUADOGDPXZTOJA5CNFSM4G2VKHV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWPBRDI#issuecomment-496900237, or mute the thread https://github.com/notifications/unsubscribe-auth/AEVAWRLQIF7G7TIIRJ5W2BTPXZTOJANCNFSM4G2VKHVQ .

vsoch commented 5 years ago

Very welcome! Closing.