search

singularityhub / sregistry-cli

Singularity Global Client for container management
https://singularityhub.github.io/sregistry-cli/
Mozilla Public License 2.0
14 stars 18 forks source link

sregistry, registry client, should honor registry base #187

Closed tamasgal closed 5 years ago

tamasgal commented 5 years ago

Sorry to bother you again, don't know which communication channel to use ;)

So I set up singularity.km3net.de and now everything works, including the Google Oauth and uploading of images. However, pulling is not working:

░ tgal@staticbox:~ py-3.7.2
░ 16:11:40 [I] ✘130 ➜ singularity pull shub://singularity.km3net.de/common/jpp:v11.0.0-rc9
WARNING: Authentication token file not found : Only pulls of public images will succeed
FATAL:   Failed to get manifest from Shub: No response received from singularity hub

Although the image is there and it should be public.

vsoch commented 5 years ago

I would bet they changed the client to not allow custom registries - although technically this should work (as it used to) because the Singularity Registry Server endpoint works the same. This would be something you'd want to open up as an issue on https://github.com/sylabs/singularity/issues. Let me know what you learn! For the time being, try just using the sregistry client.

And this is exactly where you ask questions, so no need to apologize, etc.

tamasgal commented 5 years ago

OK thanks!

tamasgal commented 5 years ago

Alright, I did: https://github.com/sylabs/singularity/issues/2819

tamasgal commented 5 years ago

So how can I use sregistry to pull from our server? It seems it always connects to a local repository 🤔

vsoch commented 5 years ago

Local repository?

vsoch commented 5 years ago

In your config file you can set the base (which defaults to localhost)

    "registry": {
        "base": "http://127.0.0.1",
        "username": "vsoch",
        "token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    }
tamasgal commented 5 years ago 
░ tgal@staticbox:~ py-3.7.2
░ 16:22:45 [I] ✘130 ➜ sregistry pull singularity.km3net.de/common/jpp:v11.0.0-rc9
[client|hub] [database|sqlite:////home/tgal/.singularity/sregistry.db]
ERROR Beep boop! Not Found: 404

I am probably mixing everything up :joy:

vsoch commented 5 years ago

look at [client:hub] that is telling you exactly the client you are using - it's the wrong one. Try as I showed you exporting SREGISTRY_CLIENT. Or if you don't want to set an envar:

sregistry pull registry://singularity.km3net.de/common/jpp:v11.0.0-rc9

Are you finding and reading the docs?

https://singularityhub.github.io/sregistry-cli/clients

tamasgal commented 5 years ago

Hmm actually I am a bit confused by the docs. I found and also tried sregistry pull registry://... but it gave:

░ tgal@staticbox:~ py-3.7.2
░ 15:35:03 [I] ✘1 ➜ sregistry pull registry://singularity.km3net.de/common/jpp:v11.0.0-rc9
[client|registry] [database|sqlite:////home/tgal/.singularity/sregistry.db]
Traceback (most recent call last):
  File "/home/tgal/.pyenv/versions/3.7.2/bin/sregistry", line 11, in <module>
    load_entry_point('sregistry==0.1.37', 'console_scripts', 'sregistry')()
  File "/home/tgal/.pyenv/versions/3.7.2/lib/python3.7/site-packages/sregistry-0.1.37-py3.7.egg/sregistry/client/__init__.py", line 368, in main
    subparser=subparsers[args.command])
  File "/home/tgal/.pyenv/versions/3.7.2/lib/python3.7/site-packages/sregistry-0.1.37-py3.7.egg/sregistry/client/pull.py", line 42, in main
    save=do_save)
  File "/home/tgal/.pyenv/versions/3.7.2/lib/python3.7/site-packages/sregistry-0.1.37-py3.7.egg/sregistry/main/registry/pull.py", line 54, in pull
    manifest = self._get(url)
  File "/home/tgal/.pyenv/versions/3.7.2/lib/python3.7/site-packages/sregistry-0.1.37-py3.7.egg/sregistry/main/base/http.py", line 116, in get
    quiet=quiet)
  File "/home/tgal/.pyenv/versions/3.7.2/lib/python3.7/site-packages/sregistry-0.1.37-py3.7.egg/sregistry/main/base/http.py", line 335, in call
    stream=stream)
  File "/home/tgal/.pyenv/versions/3.7.2/lib/python3.7/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/home/tgal/.pyenv/versions/3.7.2/lib/python3.7/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/tgal/.pyenv/versions/3.7.2/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/tgal/.pyenv/versions/3.7.2/lib/python3.7/site-packages/requests/sessions.py", line 640, in send
    adapter = self.get_adapter(url=request.url)
  File "/home/tgal/.pyenv/versions/3.7.2/lib/python3.7/site-packages/requests/sessions.py", line 731, in get_adapter
    raise InvalidSchema("No connection adapters were found for '%s'" % url)
requests.exceptions.InvalidSchema: No connection adapters were found for 'None/container/singularity.km3net.de/common/jpp:v11.0.0-rc9'
vsoch commented 5 years ago

That suggests that your base isn't set in your config (the value it's finding is None). Please show it to me?

tamasgal commented 5 years ago

Yep, I had a typo, sorry:

{
    "registry": {
        "base": "http://singularity.km3net.de"
    }
}

So this is the actual error:

░ tgal@staticbox:~ py-3.7.2 took 6s
░ 16:33:59 [I] ➜ sregistry pull registry://singularity.km3net.de/common/jpp:v11.0.0-rc9
[client|registry] [database|sqlite:////home/tgal/.singularity/sregistry.db]
ERROR Beep boop! Not Found: 404

░ tgal@staticbox:~ py-3.7.2
░ 16:34:02 [I] ✘1 ➜ cat .sregistry
{
    "registry": {
        "base": "http://singularity.km3net.de"
    }
}

I am a bit lost... Now reading the docs again

vsoch commented 5 years ago

http://km3net.singularity.de != singularity.km3net.de ... is that intentional?

tamasgal commented 5 years ago

look at [client:hub] that is telling you exactly the client you are using - it's the wrong one.

Now I am starting to understand the output messages ;)

tamasgal commented 5 years ago

http://km3net.singularity.de != singularity.km3net.de ... is that intentional?

I updated, that was part of the typo 😆 The output below is the correct one, so it should be fine.

vsoch commented 5 years ago

Does your user have pull permission?

tamasgal commented 5 years ago

I set it up as a public container: http://singularity.km3net.de/containers/1

I though I don't need to auth?

To be honest, I am trying to get it working with as minimal setup as possible for our HPC users. Without config files or additional tools. We have centralised Python installations on different HPC farms so I can install registry and sregistry but having to deal with user authentification will cause problems 😕

vsoch commented 5 years ago

Ah I think I know the issue. You need to set the base, but then not export it again. What you are providing to the client is just the uri for the base (not including the base itself).

vsoch commented 5 years ago

No you don't need to auth, I just pulled your container :)

vsoch commented 5 years ago 
$ SREGISTRY_CLIENT=registry sregistry pull common/jpp:v11.0.0-rc9
[client|registry] [database|sqlite:////home/vanessa/.singularity/sregistry.db]
Progress ||----------------------------------|   0.0%
tamasgal commented 5 years ago

indeed:

░ tgal@staticbox:~ py-3.7.2
░ 16:36:29 [I] ✘130 ➜ sregistry pull common/jpp:v11.0.0-rc9
[client|registry] [database|sqlite:////home/tgal/.singularity/sregistry.db]
Progress ||----------------------------------|   1.8% ^CTraceback (most recent call last):
vsoch commented 5 years ago

I think this is a bug - it should be possible to specify the base with the URI. I'll fix this up later today so let's keep this open.

tamasgal commented 5 years ago

Thanks!

So is there a way to omit this .sregistry file?

tamasgal commented 5 years ago

Ah ok, thank you Vanessa, as always ;)

vsoch commented 5 years ago

of course! Please let me know if you run into any issues - there was a security fix for Django recently and it's a big enough change so there have been little things coming up here and there.

I'm busy atm but will be able to work on this later today.

vsoch commented 5 years ago

And don't forget to add https to your registry!

tamasgal commented 5 years ago

Yep, that's the next step ;)

vsoch commented 5 years ago

I've transferred the issue to the client repository, as this is where the fix will need to be.

vsoch commented 5 years ago

hey @tamasgal ! I have a PR ready for you to test here https://github.com/singularityhub/sregistry-cli/pull/188. It should fix the issue with not being able to push / pull when you specify a registry base. See the PDF attached at the top of the PR for additions to the docs (to help guide your testing). To install, you can clone the branch and then install from it.

pip uninstall sregistry
pip uninstall sregistry
# do that until it tells you it isn't installed

git checkout -b fix/registry-base-required https://www.github.com/singularityhub/sregistry-cli

When you've tested and confirmed working, we can merge. Thanks!

tamasgal commented 5 years ago

Thank you Vanessa, works like a charm! (and thanks for the subtle reminder to use https haha)

░ tgal@staticbox:~ py-3.7.2
░ 16:11:37 [I] ✘1 ➜ SREGISTRY_REGISTRY_NOHTTPS=true sregistry pull registry://singularity.km3net.de/common/jpp:v11.0.0-rc9
[client|registry] [database|sqlite:////home/tgal/.singularity/sregistry.db]
Progress ||----------------------------------|   2.7%
vsoch commented 5 years ago

ready for merge?

vsoch commented 5 years ago

Haha yes, I try to make life harder for the user if he/she makes less secure decisions about registry deployment :P Nothing like having an extra annoying environment variable as a subtle reminder!

tamasgal commented 5 years ago

Yes please!

tamasgal commented 5 years ago

Haha yes, I try to make life harder for the user if he/she makes less secure decisions about registry deployment :P Nothing like having an extra annoying environment variable as a subtle reminder!

I totally understand, I manage a whole load of letsencrypt scripts for our collaboration, I just like to set SSL up when everything is working, just for convenience ;)