A draft of EUDAT authentication inside the Flask Server

[pdonorio]

What we need is a function to replace the internal Flask-Login authentication login which is implemented by default.

I didn't find a clear documentation on how to do this, but we can search online from other github repo with python flasks, i am sure there is plenty of code like that.

SubTask list:

• [x] Test a working snippet
• [x] Add oauth2 authorization inside the flask http-api-base server
• [x] Save the obtained token inside the server main db (graphdb in our case)
• [x] Ask for the proxy with the token and use it

[pdonorio]

The possible algorithm for the authentication function, based on the x509 certificates made so far would look like:

1. Receive credentials (username/passwords) into our /api/login endpoint
2. Ask B2ACCESS if credentials are valid
3. If valid, receive a proxy
4. Save the proxy into a shared dir (e.g. /opt/certificates)
5. Add the DN to iRODS server (Claudio should help us here)
6. Flask-Login will generate a token and save the entry inside the internal db (sqllite/postgres)

With a valid token, the user can request a protected endpoint.

Each authenticated request should be able to recover the iRODS username from the token.

In this way we may set the environment variable (something like X509_PROXY) in all commands for irods.

[pdonorio]

So we are now ready to implement this issue.

Thanks to @akrause2014 we have a working example for Oauth2 authentication in B2Access via flask server here.

I will try to work on the first integration and ask for help if something is wrong :)

[pdonorio]

My main idea is to create a new endpoint called for example

/api/b2access_login

which let the user open the browser and login to b2access, then confirm that they let the REST API have access to their certificates and store the token in session or similar.

[pdonorio]

First thing to do is to install libraries. e.g. flask-oauthlib and whatever else is needed.

[pdonorio]

Created a branch to test integration.

My first test is against github oauth2.

@akrause2014 where do you register the consumer key/secret on b2access? i thought i was going to find that inside the profile page once logged in, but i see nothing there.

[akrause2014]

Comment: What's the point of testing against github oauth2? The B2ACCESS oauth2 client needs some tweaks (different from github interaction) that are included in the working example. If it works with github it doesn't necessarily work with B2ACCESS - I've gone through a few iterations to create the client. You need to register your client with B2ACCESS as described here: https://eudat.eu/services/userdoc/b2access-service-integration#UserDocumentation-B2ACCESSServiceIntegration-HowtoregisteranOAuthclient It's a separate account, don't log in with your own credentials.

[pdonorio]

I tested github because i had no consumer key for b2access. And it's fine because i needed to integrate the oauth call inside the whole code, for b2access is just some different urls.

Now i need to store the access token somewhere associated to the user who is trying to login, then i am ready to be specific to b2access.

The only thing is that following the link instructions i registered the application. I received no email and login doesn't work with those new credentials. What should happen next? I don't see anything explained in that doc about it.

[akrause2014]

You won't receive an email. I don't know how long it takes them to add the new client but you can try and login to the b2access site with the client key and secret. Once the client is registered the login should work, and then the oauth2 client will work too.

[akrause2014]

In the meantime you could use the registered client key and secret that I was using, it just redirects to localhost and should be fine for test applications.

[pdonorio]

I don't have your key/secret :)

I just re-tested and the new credentials are valid now. But i have just username and password. Where are the key and secret? Is the username also the key? I don't see anything in the profile page related to this.

[akrause2014]

Well I did offer them to you (via email or skype) but you said you were going to create your own. :) The key is the username and the secret is the password.

[pdonorio]

Ok, tested with consumer key/secret since as you explained i already had! It forwards me to b2access and then i get a strange

{"errors": "Access denied: reason=server_error error=Unexpected server error"}

Uhm.

[akrause2014]

Right, I think I've seen that before. You have to log out as the registered oauth client from the b2access site. If you're still logged in with your consumer key and secret then it fails because this user is in the wrong group (no certificates). You have to log in with your "personal" username and password. The account that supports certificates i.e. a "normal" b2access account. I've raised that with the b2access guys and they're going to provide a more helpful error.

[pdonorio]

Perfect. I registered the personal account on the b2access secondary url (the one used for oauth2 login) and it works!

My callback (the second endpoint) fails after receiving the code, with:

urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)>

It seems that there is an handshake with SSL. I don't see this part covered in your authorize endpoint. Is there something wrong with my CN certificate i specified in the personal account registration?

[akrause2014]

Are you using the same code as in my client? Have you tested my client directly (standalone), with your key and secret? Do you get the error there? Line 165: decorate_http_request(b2access) This is needed to retrieve the token from b2access using the code. That's one of the tweaks I mentioned.

[akrause2014]

And there's no need to specify a CN certificate in your personal account. b2access create their own. If you're testing with the preproduction server you may have to ignore their host certificate (i.e. switch off host verification - not recommended for production!). This is done in lines 98-113 in the method http_request_no_verify_host()

[pdonorio]

I missed the decorator :)

Using it in my code i still get an error:

  File "/code/project/restapi/resources/checkauth.py", line 68, in new_http_request
    userpass = b64encode("%s:%s" % (client_id, client_secret)).decode("ascii")
  File "/usr/lib/python3.4/base64.py", line 62, in b64encode
    encoded = binascii.b2a_base64(s)[:-1]
TypeError: 'str' does not support the buffer interface

Did you test your code with python2 or 3?

Anyway i have to go now, but i will try my consumer key/secret with your code too, to see if it works there. I have to install openssl python lib first.

[pdonorio]

It was easy to install the ssl lib. Testing your code i get absolutely the same error above with my consumer key/secret.

[akrause2014]

Ah this might be an issue with the python version, I only tested with python 2.7. You should do a quick test with python 2 to make sure it's that.

[pdonorio]

Good to know.

I updated the docker image to use the b2access library: https://hub.docker.com/r/eudatb2safe/apiserver/builds/bhyi2fphflgy29kvfxugoek/

[pdonorio]

I fixed the python3 bug within the decorator. But now i am back to the SSL error i was getting before using the decorator...!

Digging more it looks like there is some issue with the unity.eudat site certificates and it could be that only in the python3 urlllib version the library complains about this.

I am looking now for what certificates are sent with the command:

openssl s_client -showcerts -connect unity.eudat-aai.fz-juelich.de:8443

If adding one of those can solve the problem at least we realized where the issue really is.

[akrause2014]

The certificate of the EUDAT preproduction site is not trusted so you either have to add it to your trusted sites, or you just ignore it by switching off verification of the host certificate (that's what I did). Have a look at the other decorator in lines 98-113. It shouldn't be necessary for the EUDAT production server.

[pdonorio]

Yeap, the http_request_no_verify_host is an option.

I wanted to check if the certificates were fine. In fact the strange thing is that my browser accepts the development b2access site certificates, while the plain ubuntu:16.04 doesn't.

So i downloaded three certificates, converted them to pem and imported everything in ubuntu. And i finally get the token to use. I will probably add the certificates either inside the docker image or as a volume from the github repo.

Next step is to save this token inside the flask user db, instead of using session or cookies. This will help us switch to the command line after using the browser to authorize the application.

[akrause2014]

That's certainly the proper way to do it. :)

[pdonorio]

I am testing again this issue's tasks.

We plugged the graphdb for storing data, so we can save an account for every b2access token obtained successfully.

After this we should check how to integrate the code on certificates.

[pdonorio]

Ready to get certificate requests.

Is the session the only way to save the b2access token for the flask-oauthlib?

[akrause2014]

The client is responsible for storing the token in whatever way it wishes - could be a file, a database or anything else.

[pdonorio]

Since we have a dockerized B2SAFE iRODS instance for development, but we have to link to an existing external B2SAFE instance in production, we must distinguish the two cases inside the code.

In the first case when the certificate is received we should check if the user exists otherwise add him with grid-proxy utilities.

In the second one we just act as if the certificate is good enough to let the user access his data, and do nothing.

[pdonorio]

@akrause2014 I can see here that you are somehow faking your host.

Can you give me some directions to help me make also the "production mode" request (e.g. when you have your host certificate)?

[pdonorio]

Thanks to @ccacciari we received a link to python scripts which may help in storing the user informations into the iRODS server when working on the dockerized version of the iCAT server.

[pdonorio]

We save the certificate proxy now. Let's see how to use it with the dockerized b2safe!

Note to self: I should also write the tests for those endpoints. Don't know how yet.

[muccix]

Some useful information to test the authentication.

• The list of B2ACCESS services is here.
• At the moment we are using the Devel user portal. So to test with a new user, it has to be created through that portal.
• To create a new EUDAT user, click on "Register a new account" -> "EUDAT staff", fill out the form and "SUBMIT".
• To manually request a temporary certificate and check the Certificate DN associated to the new user go the (devel) user certificate "portal", click on "Request an access token", login, click on "Request user certificate".
• In docker-compose.yml replace the B2ACCESS_APPKEY value with the real one: 8nAf.meh7TkoW8vk
• Run the irods server (./do irods_shell) and add the new user along with his DN: $ iadmin mkuser rmucci00 rodsuser $ iadmin aua rmucci00 "C=DE,L=Juelich,O=FZJ,OU=JSC,CN=e280fe69-3753-4061-9d9d-c52aa34324eb,CN=null"
• Run the server: $ ./do server_shell $ ./boot devel
• Authenticate the user; using the browser go to http://localhost:8080/auth/askauth. At the moment I receive an Internal Server Error.

 File "/code/restapi/resources/custom/oauth.py", line 38, in get
    return self.response(response)
AttributeError: 'OauthLogin' object has no attribute 'response'

[pdonorio]

At the moment I receive an Internal Server Error.

We will need some things to be fixed for that error:

• [x] Make use of the issue to apply all actual calls response inside oauth.py as a custom response, like the issue EUDAT-B2STAGE/http-api-base#23.

Move the graphdb code for storing the b2access informations into its class, and create an analogue for the sqllite db (see #25). Use the proxy saved with the B2ACCESS token.

[pdonorio]

Ok, response is done. I am testing back the access to b2access.

When I'll make it work I have to save it inside the sqllite db.

[pdonorio]

There are ongoing problems with B2ACCESS CA on unity domain (the b2access dev website). While the access token works on port 8443, in 8445 the CA has "pool timeout"

[pdonorio]

The B2ACCESS team solved quickly after our feedback on the unity CA. Now we can retrieve and store the proxy certificate into the /tmp dir, both verified by me and Roberto.

Things to go:

• [x] verify if the proxy certificate allows us to use iRODS on the client side, by storing the proxy into /opt/certificates and setting the right env variables.
• [x] store the b2access token and the other b2access infos into sqllite, instead of graphdb like it used to be #25
• [x] when a user authenticated through b2access is accessing with our token, we must verify the b2access token expiration time. if it's expired, drop the current token and require a new one. if you can't retrieve a new one, point the user to repeat the protocol

[pdonorio]

verify if the proxy certificate allows us to use iRODS on the client side, by storing the proxy into /opt/certificates and setting the right env variables.

@muccix can you try this manually? copy the proxy from tmp into the shared /opt/certificates folder and set the client env vars to see if the ils icommand works

[pdonorio]

when a user authenticated through b2access is accessing with our token

I prepared instead a refresh proxy, which will signal the user at every call if it's expired.

If refreshing the b2access token is expired too, we will instead redirect to /auth/askauth