OIDC Error message: "Not authenticated. Use the key on the server index page."}

[rudyzhou2]

Hi there,

Ive been trying to deploy ga4gh with oidc auth enabled through our own OIDC provider. However, I am encountering the following error after linking ga4gh server with our OIDC auth service

ga4gh server message upon successful oidc auth

{"errorCode": 1533469862, "message": "Not authenticated. Use the key on the server index page."}

OIDC discovery document

{"issuer":"http://172.17.0.57:5556","authorization_endpoint":"http://172.17.0.57:5556/auth","token_endpoint":"http://172.17.0.57:5556/token","jwks_uri":"http://172.17.0.57:5556/keys","response_types_supported":["code"],"grant_types_supported":["authorization_code","client_credentials"],"subject_types_supported":["public"],"id_token_signing_alg_values_supported":["RS256"],"token_endpoint_auth_methods_supported":["client_secret_basic"]}

OIDC debug log

INFO: 172.17.0.52 - - [12/Jul/2016:18:46:59 +0000] "GET /.well-known/openid-configuration HTTP/1.1" 200 438
INFO: 172.17.0.52 - - [12/Jul/2016:18:46:59 +0000] "GET /auth?nonce=Ez7EubVRRExLDNP9ldEh6TQp&state=D986zyuU4VBa8Tp7o7DPIkQJ&redirect_uri=http%3A%2F%2F172.17.0.52%2Fga4gh%2Foauth2callback&response_type=code&client_id=znLZ6aw5CylEOOd3iCfXo56qn5UJY66vZpa0lp0q1aY%3D%40172.17.0.52&scope=openid+profile HTTP/1.1" 200 25283
INFO: 172.17.42.1 - - [12/Jul/2016:18:46:59 +0000] "GET /auth?nonce=Ez7EubVRRExLDNP9ldEh6TQp&state=D986zyuU4VBa8Tp7o7DPIkQJ&redirect_uri=http%3A%2F%2F172.17.0.52%2Fga4gh%2Foauth2callback&response_type=code&client_id=znLZ6aw5CylEOOd3iCfXo56qn5UJY66vZpa0lp0q1aY%3D%40172.17.0.52&scope=openid+profile HTTP/1.1" 200 25283
INFO: Session W5MqzJzhXDM= created: clientID=znLZ6aw5CylEOOd3iCfXo56qn5UJY66vZpa0lp0q1aY=@172.17.0.52 clientState=D986zyuU4VBa8Tp7o7DPIkQJ
INFO: 172.17.42.1 - - [12/Jul/2016:18:47:01 +0000] "GET /auth?client_id=znLZ6aw5CylEOOd3iCfXo56qn5UJY66vZpa0lp0q1aY%3D%40172.17.0.52&connector_id=ldap&nonce=Ez7EubVRRExLDNP9ldEh6TQp&redirect_uri=http%3A%2F%2F172.17.0.52%2Fga4gh%2Foauth2callback&response_type=code&scope=openid+profile&state=D986zyuU4VBa8Tp7o7DPIkQJ HTTP/1.1" 302 0
INFO: 172.17.42.1 - - [12/Jul/2016:18:47:01 +0000] "GET /auth/ldap/callback/login?prompt=select_account&session_key=WC0sFL8eDOo%3D HTTP/1.1" 200 25396
INFO: Session W5MqzJzhXDM= remote identity attached: clientID=znLZ6aw5CylEOOd3iCfXo56qn5UJY66vZpa0lp0q1aY=@172.17.0.52 identity=oidc.Identity{ID:"uid=rzhou,ou=users,dc=dex,dc=ca", Name:"Rudy Zhou", Email:"rzhou@xxx.ca", ExpiresAt:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}
INFO: Session W5MqzJzhXDM= user identified: clientID=znLZ6aw5CylEOOd3iCfXo56qn5UJY66vZpa0lp0q1aY=@172.17.0.52 user=user.User{ID:"97c3e6d9-9a0f-49b3-a068-d38ca6aadf0d", DisplayName:"", Email:"rzhou@xxx.ca", EmailVerified:true, Admin:false, Disabled:false, CreatedAt:time.Time{sec:63603595496, nsec:0, loc:(*time.Location)(0xfd84c0)}}
INFO: 172.17.42.1 - - [12/Jul/2016:18:47:07 +0000] "POST /auth/ldap/callback/login?prompt=select_account&session_key=WC0sFL8eDOo%3D HTTP/1.1" 302 0

The callback url used this this case is: http://172.17.0.52/ga4gh/oauth2callback. I have tested this deployment against another sample web app and the authentication functions as expected.

Can anyone give me an idea where exactly is ga4gh giving me trouble regarding auth? Does ga4gh absolutely require https for oauth to work?

[rudyzhou2]

ga4gh access log

172.17.42.1 - - [12/Jul/2016:18:43:37 +0000] "GET /ga4gh/oauth2callback?code=j7_uPKYu994%3D&state=Ksw4ThxiRG3D2YFP5vFe96Xn HTTP/1.1" 403 299 "http://172.17.0.57:5556/auth/ldap/callback/login?prompt=&session_key=agOmIyQcfV0%3D" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36"
172.17.42.1 - - [12/Jul/2016:18:46:59 +0000] "GET /ga4gh HTTP/1.1" 302 1469 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36"
172.17.42.1 - - [12/Jul/2016:18:47:07 +0000] "GET /ga4gh/oauth2callback?code=QE6yyd7E7P4%3D&state=D986zyuU4VBa8Tp7o7DPIkQJ HTTP/1.1" 403 299 "http://172.17.0.57:5556/auth/ldap/callback/login?prompt=select_account&session_key=WC0sFL8eDOo%3D" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36"
172.17.42.1 - - [12/Jul/2016:18:51:46 +0000] "GET /ga4gh HTTP/1.1" 302 1469 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36"
172.17.42.1 - - [12/Jul/2016:21:05:20 +0000] "GET /ga4gh HTTP/1.1" 302 1472 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36"
172.17.42.1 - - [12/Jul/2016:21:05:28 +0000] "GET /ga4gh/oauth2callback?code=BKcIGgy251k%3D&state=QYqFZoa6uDcMZTh3juLppDSp HTTP/1.1" 403 299 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36"

Seems like I am constantly getting 403d even though my OIDC server has sent the auth code...

ga4gh error log

[Tue Jul 12 18:35:48.975616 2016] [wsgi:warn] [pid 6:tid 140632977897344] mod_wsgi: Compiled for Python/2.7.11.
[Tue Jul 12 18:35:48.975621 2016] [wsgi:warn] [pid 6:tid 140632977897344] mod_wsgi: Runtime using Python/2.7.11+.
[Tue Jul 12 18:35:48.977661 2016] [mpm_event:notice] [pid 6:tid 140632977897344] AH00489: Apache/2.4.18 (Ubuntu) mod_wsgi/4.3.0 Python/2.7.11+ configured -- resuming normal operations
[Tue Jul 12 18:35:48.977682 2016] [core:notice] [pid 6:tid 140632977897344] AH00094: Command line: '/usr/sbin/apache2 -D FOREGROUND'

[jeromekelleher]

Thanks for the bug report @rudyzhou2. Is there a particular reason for not using https for the callback URL? I would have thought that using https from end-to-end would be a wise security precaution for any deployment using authentication. Otherwise, isn't it possible for an attacker to steal credentials?

[rudyzhou2]

Hi Jerome,

Oauth2 only requires https on the authentication server end, which I have enabled. I could use https for the callback url but Id have to make some changes as to how the containers are exposed to each other and have to setup apache redirect, this is in my plans once I can get basic authentication working.

Upon examining the frontend code, it seems like I am getting issues at oidcCallback function where flask session state does not match session state responded by the auth server. Looks like I may have to try a different OIDC provider to see if this issue persists. Will report back.

[jeromekelleher]

Oauth2 only requires https on the authentication server end, which I have enabled. I could use https for the callback url but Id have to make some changes as to how the containers are exposed to each other and have to setup apache redirect, this is in my plans once I can get basic authentication working.

This sounds good, but I'm curious: why not have it HTTPS only the whole way through? This would seem like the simpler (and safer) option.

[rudyzhou2]

Hi Jerome,

Mainly because for dev purposes, I just wanted it to work without worrying about tls and certificates yet. Would be great if you can point me to some documentation on how to add tls to ga4gh server, is it a simple apache redirect from 443 to 80?

Cheers,

Rudy

[jeromekelleher]

I think it'll be easier to tackle the https now rather than later. First you need to set up Apache to support https. I think I followed this guide to use a self-signed cert. This is a terrible idea for production use, but it's enough to get started with. You might want to look into let's encrypt as a better alternative (I haven't tested this).

Once you have a HTTPS site up and running, you just edit the config file (e.g. /etc/apache2/sites-enabled/default-ssl.conf) to include the same WSGIDaemonProcess stuff as the standard config. Once that's done, everything should be running over HTTPS.

[rudyzhou2]

o i see, sounds like a great idea! I will implement https and see if resolves the issue.

Upon more debug, it seems like the issue originates from flask session state != state generated by OIDC provider, raising the not authenticated exception... Unfortunately I have no idea as to why the 2 states different though I could see the first login request from flask generates a random 24 character code., and access token request with oic. oauth2 successful by manually POSTing the oidc server. Gonna dig into it a bit more tomorrow.

[rudyzhou2]

Hi Jerome,

So applying HTTPS did not fix the issue; but I finally ended up finding out the issue after digging through the code and the OIDC providers I have used...

It seems that when everyone says that they are implementing OpenID connect compatible workflows, there is always things that is "non-standard" and causes incompatbility...

I tried 2 OIDC providers so far, Dex from CoreOS (Access token API not yet implemented) and UAA from Cloudfoundry (OpenID token appends number sign (#) tag right infront of POST result, which is non Werkzeug and HTTP GET request will ignore anything after (#) sign, so the flask server cant even receive this information after the login redirect...). Neither works, however, google does work while facebooks fail as testing (same issue as Cloundfoundry UAA).

My colleage told me he has a working OIDC provider using reference OpenID standard which has been tested working but he doesnt like it (MitreID Connect). I may likely just end up rewriting the authentication bit in the frontend server to make authentication Oauth2 compliant at least...

You can close this ticket in the main time!

Cheers,

Rudy

[jeromekelleher]

Thanks for the update @rudyzhou2. This is all a bit disappointing I have to say... do you think we would need to have to explicitly support different OIDC providers and have different code paths for each? Thanks very much for looking in to this and telling us about it!

[rudyzhou2]

Hi Jerome,

The current authentication workflow in ga4gh is via OpenID connect, which is giving us issues. Pyoidc seems to support Oauth2 as well so Im probably gonna see if I can rewrite the OIDC ga4gh authentication workflow to use Oauth2 standard instead of OpenID Connect. This should make it compatible with most IdP providers and tools out there since Oauth2 is simpler.

The end result should be the same since a random session token generated by ga4gh server will be associated with an access token and be used to check for authentication success by ga4gh. The good thing is at least authentication is fairly modular in ga4gh so hopefully the changes I make wont affect the API and the client tool...

If pyoidc doesnt work, I may try using Flask-Rauth or Flask-Oauth packages instead and see if that can help with the issue.

Cheers,

Rudy

[rudyzhou2]

Hi Jerome,

I am very happy to report that I finally made some significant progress with regards to ga4gh authentication.

I ended up implementing OAuth2 instead of OpenID Connect workflow with rauth, and it seems to be working on my test flask server, the next task for me is probably to just rewrite the entire ga4gh authentication code to use rauth instead of pyoidc/oic.oauth2. Since the authentication code is quite independent, thank god it probably wont affect the API itself.

I still dont fully understand why oic.oauth2 does not work, but it could be related to the fact that it seems to use 'state' as part of its workflow during access token request, causing no grant found for state 'xxx' issue.

Will report back once I rewrite the thing and finally get ga4gh authentication working!

[jeromekelleher]

Great, thanks @rudyzhou2! Do you think it would be possible to structure what you're doing as working in parallel with OpenID Connect support rather than replacing it? For example, by having a different callback URL for OAuth2? It would be great if we could merge your work back in so that we can support OAuth2 properly, given that OIDC is such a pain to get working with providers other than Google!

[rudyzhou2]

Hi Jerome,

This is a great idea! Will make my work modifying the flow significantly easier as well! Will report back once I have a working prototype!

Cheers,

Rudy