[testing] Integration with Globus

[vsoch]

What are your thoughts on how this would work? Feel free to answer as many or as few of the below as you choose. Specifically:

• Do you envision an integration with Singularity Registry, Hub, or both?
• How does a registry endpoint (a local fileystem), or the hub cloud storage (Google Cloud Storage), interact with Globus clients?
• Who has control to initiate a sharing of a container? How does it work?
• Can entire folders / collections be shared automatically? How does this work (in terms of webhooks - does each collection send out a webhook that users subscribe to, or does the registry keep track of subscribed users and initiate the transfer?)
• Do you see benefit to Globus over using Singularity pull?
• Does it complicate things if the Singularity Registry is external to the cluster (since requires docker).
• What level of control is a user given to update images in a Registry?
• Does some file system watcher (e.g., inofity) drive the notifications for changed images?
• Should a share be automatic, or manual? In both cases, how is the model created? Updated?

Please post your thoughts, or disregard the questions above and answer "if I had a way to easily share images it would look like this..."

Thanks!

[vsoch]

All interested - please review here https://github.com/singularityhub/sregistry/blob/add/globus/docs/plugins/globus.md#development

There is one final issue with regard to paths, but it's ok to start discussion on next steps and get feedback. The base case for what we've talked about for integration of Globus with a Singularity Registry is finished, on the branch linked above.

Please point your attention to the start of documentation (and brief walkthrough) of the changes:

https://github.com/singularityhub/sregistry/blob/add/globus/docs/plugins/globus.md

For a superficial overview look at the section on what the integration adds, and importantly, please give me your feedback on the development section. The remainder is notes about setting up a local endpoint that hopefully will be useful to someone setting up a registry in the future that has never used Globus.

I'd like to get the few bugs resolved, and then it will make sense to have some testing via a PR.

[vsoch]

From previous issue, question about search integration to be active only for registries with globus_auth activated

http://globus-search-docs.s3-website-us-east-1.amazonaws.com/stable/api/ingest.html

This PR seems to be dead, the issue was closed on the official Globus board, despite quick turn around on my part. :/

[vsoch]

The issue has been updated with an sdk method, and this will be the last thing I try --> https://github.com/globus/globus-sdk-python/issues/271#issuecomment-366830008

[victorsndvg]

Hi @vsoch , how are you? :)

I think I could do some tests with the globus branch. In which state is it? do you think is ready for playing?

Thanks!

[vsoch]

almost :) tomorrow / later today I'm going through one more round of testing myself, then will announce for wider testing!

[vsoch]

Also to be clear, the first we are testing is going to be with the Global Client Interface ("Tunel") and then I can finish the Globus integration here.

[victorsndvg]

Ok, thanks!

Let me know when it's ready. I will try to stress it :D

[victorsndvg]

I've seen some movements on things related with globus. Any news?

[vsoch]

Still underway, I'll post / alert when things are done. I work on multiple things at once, so I do my best.

[victorsndvg]

Yes, sorry.

It was only a ping to know the status because I read some comments somewhere.

Thanks! ;)

[vsoch]

If you want to test Tunel, that is ready to go :) --> https://singularityhub.github.io/interface

[vsoch]

@victorsndvg it's your lucky day! We are ready for testing! https://github.com/singularityhub/sregistry/pull/113

I added notes for how to get to the docs (in a few ways) at the link above. My brain is pooped out on Globus so please take your time :)

[victorsndvg]

Great @vsoch ! :))

Next week I will be doing some testing and reporting my experience.

Thanks for your hard work!

[victorsndvg]

I'm here again. I'm confused with SOCIAL_AUTH_GLOBUS_KEY, where can I get it?

[vsoch]

In the instructions page here --> https://singularityhub.github.io/sregistry//plugin-globus there is reference to getting the secrets (and the link to the page here) http://globus-sdk-python.readthedocs.io/en/stable/tutorial/#step-1-get-a-client. The key and secret are what you generate from those steps linked!

[vsoch]

Globus is a little weird I have to admit, I never liked using the various clients (although I think they are making them better) but it seems like every HPC center is hooked up, and once you have it configured it isn't so bad. But apologies if the setup for this is annoying / hard, it took me since October 2017 to just develop these!

[victorsndvg]

Hi @vsoch,

I do not why, but following the tutorial I can get the token (SOCIAL_AUTH_GLOBUS_KEY?). This is the link to the step: http://globus-sdk-python.readthedocs.io/en/stable/tutorial/#step-3-get-some-access-tokens

If I create my own organization and app (sregistry app with specified redirect URIS) and I run the python script with the CLIENT_ID it returns error:

Unable to Fulfill Your Request
Error processing OAuth2 request
Sorry, but we encountered a problem while servicing your request.

Mismatching redirect URI.
Occurred at time:
2018-05-30T08:35:09.037532+00:00
Error ID:
5c7fd698-63e4-11e8-8bf2-0e8f78c01df4
Error code:
OAUTH_AUTH_REQUEST_FAILURE
If this problem persists, please contact support and copy-paste the information above to help us resolve the problem

It sounds familiar for you?

[vsoch]

Yes the list of redirect Uris must match exactly, if it's different you get this error.

[victorsndvg]

Sorry I'm blocked with this.

If I define a "native" application I can obtain the token but not a secret. If the application is not "native" I can get the secret but not the token ...

I'm using a normal openid account. There is any restriction with the account? May I use the account of a globus datanode provider?

[vsoch]

It is not a native application, so you should have a key and secret.

[vsoch]

You were on the right track when you had the key and secret, you just need to debug the redirect URI. This is a common error - it HAS to match exactly. You can put all variations if it helps. Here is the ones that work for me:

If you still have trouble you can contact Globus Support to ask for help. They can look more closely at the error.

[victorsndvg]

Hi @vosch,

I was able to authenticate with the globus app. But I need to modify 2 hardcoded localhost here:

https://github.com/singularityhub/sregistry/blob/master/shub/apps/users/templates/users/_profile_integrations.html#L18 https://github.com/singularityhub/sregistry/blob/master/shub/plugins/globus/views.py#L88

it has sense for you?

[victorsndvg]

Hi @vsoch ,

I connect sregistry with an application owned by an OpenID account and now I'm using it with another user (different OpenID).

I don't see any endpoint, I suppose sRegistry endpoint should be there. Should I see any default endpoint?

Thanks in advance!

[vsoch]

hey @victorsndvg I don't totally understand the issue - could you write out steps to reproduce / take screen shots to help? Thanks!

[victorsndvg]

I setup the globus plugin with a dummy OpenID account for testing and a banana-blahblah-1234 endpoint appear. Then I reproduce the same steps with the production account. When a user login with his/her OpenID account no endpoints are shown for them. I expecto to have sregistry endpoint in the list, but no one appears. Is this the expected behaviour?

[vsoch]

You should only need to setup the Globus once - I'm not sure why you are doing it twice. Then second, "I expect to have an sregistry endpoint in the list" it's not clear where you are referring to this. When you create the endpoint it's a personal connect endpoint, so when it's active you would be able to transfer images from it to other Globus locations. I don't expect it to work the other way around (to transfer images TO the registry).

[victorsndvg]

Ok, sorry for bothering you ...

I think I'm starting to understand how it works. :tada:

I will report any issue I found.

[victorsndvg]

Hi @vsoch ,

I'm doing the first transfer tests. I get permission denied on globus activity.

First of all, I change this line: https://github.com/singularityhub/sregistry/blob/master/shub/plugins/globus/actions.py#L76 Because my images are stored in /var/www/images. But this change does not make it works.

Taking a look to the log, i see this:

{
"context": [
{
"operation": "Directory List / File Scan",
"path": "/var/www/images/blahblah.simg"
}
],
"error": {
"body": "500 Command failed : Path not allowed.\\r\\n",
"code": 500,
"endpoint": "sregistry-mso4sc-endpoint (60bcac8e-6d50-11e8-9305-0a6d4e044368)",
"server": "gsiftp://172.20.1.76:39722",
"type": "FTPServerError"
}
}

It sounds familiar to you?

[vsoch]

This looks like an issue with globus, I haven't used it on an FTPServer. I would contact support@globus.org and ask for advice.

[victorsndvg]

Hi @vsoch ,

I've identified the problem (I think).

1. /var/www/images is not in /root/.globusconfig/alt/config-path
2. Images are stored with this permissions/owner -rw------- 1 root root
3. transference is performed by tunel-user

I kindly suggest::

1. add settings.MEDIA_ROOT to /root/.globusconfig/alt/config-path
2. • change images permisions or owner of the images or
   • change the user who makes the globus transfer

Maybe I'm missing something in my configuration??

What do you think?

[vsoch]

I think maybe you didn't set it up correctly? Uncommenting a line in the Dockerfile should run this:

https://github.com/singularityhub/sregistry/blob/master/scripts/globus/globus-setup.sh

which copies this config-paths and does other setup stuffs:

https://github.com/singularityhub/sregistry/blob/master/scripts/globus/config-paths

[victorsndvg]

Yeah, @vsoch, I ran the globus-setup.sh script and then config-paths is in /root/.globusconfig/lta/config-paths. The problem is that the images is not one of the paths inside the file.

If I manually add the right path (/var/www/images) then globus is able to list the directory and show its content.

Then, I try to move from sregistry to other endpoint and the problem is with the owner/permissions. As tunel-user is the one performing the transfer, it has to able to read the images. By default my images are owned by root and with 600 permissions and tunel-user cannot read them.

more clear now?

[victorsndvg]

Hi @vsoch ,

I don't totally understand the workflow of sregistry while using globus plugin. In particular I see this line of code. It seems that if a user does not have an endpoint (shared, owned, etc.) he/she cannot search for an endpoint? https://github.com/singularityhub/sregistry/blob/master/shub/plugins/globus/templates/globus/transfer.html#L29 I'm right?

I would expect an end-user being able to search for an (not shared-with/owner) endpoint to transfer any container he/she has access from sregistry to a custom endpoint. Is this possible?

Let me know your thoughts about it!

[vsoch]

That if statement just renders the result of a search if there are endpoints. If there aren't, then we don't need to render them. Using the globus plugin, period, means that the registry is generated with a personal endpoint, so it's possible to transfer images from sregistry TO any globus endpoint with correct permissions.

[victorsndvg]

Ouch, you'r right!

I didn't saw the search form. It look weird in my browser:

[vsoch]

holy lord, that view looks terrible! Are there any bugs / issues in the console?

Note that for the current PR with chunked upload I found a few bugs for the globus integration that I'm fixing, but I'm having considerable trouble with the chunked upload from the terminal. The one from the browser is working. We have a few options:

• maintain the current command line (limited) upload, but add the browser upload for larger containers, merge the integration with this and put the command line as another issue
• hold off merging until both are figured out (will take much longer)

thoughts? I don't have much help doing this, so these are the choices

[victorsndvg]

@vsoch ,

maybe I'm being anxious, but my choice is the first option. I prefer to merge things that are working as soon as possible and then (if time and knowledge is enough) focus in new improvements.

But you can proceed as you prefer. I can also merge things from any development branch

[vsoch]

okay, in that case please go ahead and test the PR for the (in browser) upload, and ensure that the command line works as well. If it looks good to you, then we can add that feature (and work on the command line separately). I'm eager / excited to get the kubernetes integration done (andi it's next in the queue!) so I'm hoping to get through this one :) I also think it's one of those things I'll just figure out eventually, and doesn't make sense to hold up other things. Thanks!

[victorsndvg]

@vsoch ,

I cannot test before 2 weeks. I will do it ASAP.

Kubernetes? sounds interesting! how you are going to take advantage of it?

[jpunzel]

Resurrecting this thread because we had some issues with Globus integration with our non-Docker, standalone setup.

Attempting to connect a globus account in Settings > Integrations would always result in a failure:

Unable to Fulfill Your Request
Error processing OAuth2 request

Sorry, but we encountered a problem while servicing your request.

Invalid redirect URI.

Occurred at time:
    2018-08-09T20:42:53.356695+00:00
Error ID:
    c9ca71e6-9c14-11e8-9f8b-0a350ff3fd64
Error code:
    OAUTH_AUTH_REQUEST_FAILURE 

As it turns out, Globus requires the Redirect URIs given in the OAuth request url to match EXACTLY what is specified when you set up the app in the Globus developer portal. In our case, we specified full https URLs with our FQDN, when the path generated by globus_login was simply /globus/login/. In order to fix this we had to manually change this line in views.py:

#    redirect_uri = reverse('globus_login')
    redirect_uri = 'https://my.full.fqdn/globus/login/'

[vsoch]

I think from the contained version, this would work okay, but it's good to log this here for others that might run into the issue. Thanks!

[ghltshubh]

I keep getting the same error. I am trying to install Globus Data portal: https://github.com/globus/globus-sample-data-portal/blob/master/README.md for demo purposes but keep getting globus OAuth2 error: Error processing OAuth2 request Sorry, but we encountered a problem while servicing your request.

Mismatching redirect URI. . . .

[vsoch]

Did you see the post from above? The redirect uri has to match to a T so if you have some kind of proxy, or even use www (vs not) or a slightly different format, it will give you this error. Try the suggestion by @jpunzel :+1:

#    redirect_uri = reverse('globus_login')
    redirect_uri = 'https://my.full.fqdn/globus/login/'

[ghltshubh]

I used the demo app provided by the globus: https://github.com/globus/globus-sample-data-portal/blob/master/README.md. I am not sure their own demo has the same issue.

[vsoch]

Sorry that is different software, I’m not sure that’s relevant to using the client through sregistry. Please try the suggestion above.

[victorsndvg]

Hi @vsoch ,

I'm coming back to this issue to expose my experience with sregistry+globus and to suggest some improvements.

First of all, it's clear that the provided (install and stup ) globus scripts setup a personal endpoint. The con of this solution is that free globus-personal endpoints cannot be used by other users except the owner.

I successfully tried to move images from sregistry to other endpoints with my user (the personal endpoint owner), but other users don't have permission to do transfers from the personal endpoint.

In fact, the error I get is the following:

uwsgi_1   | Creating a new TransferData object
uwsgi_1   | [instance:140021531842992] TransferClient.get_submission_id({})
uwsgi_1   | [instance:140021531842992] GET to submission_id with params {}
uwsgi_1   | [instance:140021531842992] request will have authorization of type <class 'globus_sdk.authorizers.access_token.AccessTokenAuthorizer'>
uwsgi_1   | Setting AccessToken Authorization Header: "Bearer ...t0a07" (last 5 chars)
uwsgi_1   | [instance:140021531842992] request will hit URL:https://transfer.api.globus.org/v0.10/submission_id
uwsgi_1   | [instance:140021531842992] Request made to URL: https://transfer.api.globus.org/v0.10/submission_id
uwsgi_1   | [instance:140021531842992] request completed with response code: 200
uwsgi_1   | TransferData.submission_id = 1ed2ccd5-b7f8-11e8-8bfb-0a1d4c5c824a
uwsgi_1   | TransferData.source_endpoint = 5b52b35e-b5af-11e8-8241-0a3b7ca8ce66
uwsgi_1   | TransferData.destination_endpoint = d0efb760-3e40-11e8-ba17-0ac6873fc732
uwsgi_1   | TransferData.verify_checksum = False
uwsgi_1   | TransferData.preserve_timestamp = False
uwsgi_1   | TransferData.encrypt_data = False
uwsgi_1   | TransferData.recursive_symlinks = ignore
uwsgi_1   | TransferData.label = Singularity Registry Transfer
uwsgi_1   | TransferData.sync_level = 3 (checksum)
uwsgi_1   | TransferData[5b52b35e-b5af-11e8-8241-0a3b7ca8ce66, d0efb760-3e40-11e8-ba17-0ac6873fc732].add_item: "/code/images/mso4sc/zibaffinitylatest.simg"->"mso4sc/zibaffinitylatest.simg"
uwsgi_1   | [instance:140021531842992] TransferClient.submit_transfer(...)
uwsgi_1   | [instance:140021531842992] POST to /transfer with params None
uwsgi_1   | [instance:140021531842992] request will have authorization of type <class 'globus_sdk.authorizers.access_token.AccessTokenAuthorizer'>
uwsgi_1   | Setting AccessToken Authorization Header: "Bearer ...t0a07" (last 5 chars)
uwsgi_1   | [instance:140021531842992] request will hit URL:https://transfer.api.globus.org/v0.10/transfer
uwsgi_1   | [instance:140021531842992] Request made to URL: https://transfer.api.globus.org/v0.10/transfer
uwsgi_1   | [instance:140021531842992] request completed with (error) response code: 403
uwsgi_1   | Content-Type on error is application/json. Doing error load from JSON
uwsgi_1   | Internal Server Error: /globus/transfer/d0efb760-3e40-11e8-ba17-0ac6873fc732/container/17/
uwsgi_1   | Traceback (most recent call last):
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 41, in inner
uwsgi_1   |     response = get_response(request)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
uwsgi_1   |     response = self._get_response(request)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 187, in _get_response
uwsgi_1   |     response = self.process_exception_by_middleware(e, request)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 185, in _get_response
uwsgi_1   |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/contrib/auth/decorators.py", line 23, in _wrapped_view
uwsgi_1   |     return view_func(request, *args, **kwargs)
uwsgi_1   |   File "./shub/plugins/globus/decorators.py", line 38, in wrap
uwsgi_1   |     return function(request, *args, **kwargs)
uwsgi_1   |   File "./shub/plugins/globus/views.py", line 183, in submit_transfer
uwsgi_1   |     container=container)
uwsgi_1   |   File "./shub/plugins/globus/actions.py", line 78, in do_transfer
uwsgi_1   |     transfer_result = client.submit_transfer(tdata)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/globus_sdk/transfer/client.py", line 1099, in submit_transfer
uwsgi_1   |     return self.post('/transfer', data)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/globus_sdk/base.py", line 214, in post
uwsgi_1   |     retry_401=retry_401)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/globus_sdk/base.py", line 387, in _request
uwsgi_1   |     raise self.error_class(r)
uwsgi_1   | globus_sdk.exc.TransferAPIError: (403, 'PermissionDenied', 'Permission denied to the GCP endpoint sregistry-mso4sc-endpoint (5b52b35e-b5af-11e8-8241-0a3b7ca8ce66)', 'Vh91EIueW')

I can figure out that this error does not occur with globus plus users (https://www.globus.org/subscriptions) ,but it's not my case.

My first suggestion is to try to catch and manage this error. In particular I suggest,

• At least a message telling that the transfer is not going to be performed
• Include also a link to try to activate the origin endpoint (if not activated)

The second suggestion has not too much sense with free personal endpoints, but I'm thinking to try to integrate a globus connect server with Sregistry. In this case the user have to activate the origin and also the destiny endpoints. Have you already thought about it?

I think the could be possible with minor modifications on sregistry. What I understood from the current sregistry integration with globus is that the glue between both is:

• path to images
• endpoint id
• globus-cli/sdk

And I think that almost everything is ready except the activation of the origin endpoint.

What do you think? Does this origin endpoint should be also shown in the globus view?

[vsoch]

First of all, it's clear that the provided (install and stup ) globus scripts setup a personal endpoint. The con of this solution is that free globus-personal endpoints cannot be used by other users except the owner.

This is the level of access control I think that globus typically controls - for individual users.

I successfully tried to move images from sregistry to other endpoints with my user (the personal endpoint owner), but other users don't have permission to do transfers from the personal endpoint.

Yeah, this is how it should be. For the error - have you tried with other users using the sregistry provided endpoint to transfer to a (non personal) endpoint like a cluster?

At least a message telling that the transfer is not going to be performed Include also a link to try to activate the origin endpoint (if not activated)

The second suggestion has not too much sense with free personal endpoints, but I'm thinking to try to integrate a globus connect server with Sregistry. In this case the user have to activate the origin and also the destiny endpoints. Have you already thought about it?

The Globus API doesn't give me much power to actually get information about endpoint statuses. If you look at all my issues here https://github.com/globus/globus-sdk-python/issues?q=is%3Aissue+author%3Avsoch+is%3Aclosed you will see I struggled with these exact points and just did the best that I could.

And I think that almost everything is ready except the activation of the origin endpoint.

I think this happens when you originally build the image? Likely it didn't happen there.

What do you think? Does this origin endpoint should be also shown in the globus view?

I honestly don't think Globus works well enough or is useful enough to spend a lot of time on this. I was going to give it a lot of time to get better and revisit later. I would say that it would be more useful to invest in builders or storage plugins that connect to a small set of commonly used places instead.

[victorsndvg]

I successfully tried to move images from sregistry to other endpoints with my user (the personal endpoint owner), but other users don't have permission to do transfers from the personal endpoint.

Yeah, this is how it should be. For the error - have you tried with other users using the sregistry provided endpoint to transfer to a (non personal) endpoint like a cluster?

Exactly, trying to transfer an image from sregistry to a cluster DTN (globus server) with a different account than the globus personal owner account fails with no permission error.

This is what happen to me. Have you tried it? is this use case working for you??

Thanks!

PS: I think it's useful!

[vsoch]

Globus was added as a plugin, closing stale testing issue.