About feature requests and providers

[qay21]

Hi,

I couldn't find any notes regarding feature requests in your Contributing or Code of Conduct documents, so here I am : are you accepting feature requests ?

I am trying to authenticate my company's Redmine users through either Google or Keycloak. Neither of these are supported by the current version of your plugin. It seems that your plugin is the last one standing regarding modern auth processes. Most OIDC plugins are dead or dying, mostly not compatible with Redmine v5+, so you are kinda my last hope ^^'

I do not have skills in Ruby on Rails but I am considering learning so I can help maintain some crucial plugins. If you are not accepting feature requests, would you consider pull requests if I succeed in adding support for other providers ? I have spotted various forks of your work that seems to add some providers, but are apparently not opening pull requests to you, which seems strange to me.

[picman]

I certainly accept feature request and appreciate pull requests. Adding support for another provider won't be a problem. A problem is testing as for a new provider you would need an account which is usually paid one. You obviously have got such accounts so we can cooperate on realization.

[qay21]

That's awesome news ! Should I create a dedicated feature request about Google integration, as it is the one requiring paid accounts (that I can provide) ?

[picman]

I've added a support for Google OAuth provider into google branch. Would you be able to test the google branch and provide me with a feedback? I don't know how to test it.

[picman]

I've added a support for Keycloak too. At this phase I'd need to test both new providers. Either by you or if you provide me access I can test it myself too. Especially I'd need to verify that I entered correct information for authorization:

1. Authorization URL Google: https://accounts.google.com/o/oauth2/v2/auth Keycloak: https://someDomain/auth/realms/[myRealm]/protocol/openid-connect/auth
2. Scope Google: "openid profile email" Keycloak "openid email"
3. Obtained token's format

[picman]

I've successfully tested Keylock as a OAuth provider. What left is Google. Google requires your own domain even for a trial. It costs min €12. If you have a Google company account with admin rights, you can test it yourself. I can't.

[qay21]

Sorry for the delay and happy new year !

I indeed have access to admin rights of my company's Google Workspace organization. I will do the testing and keep you posted !

[qay21]

I'm back with some news !

I was able to install and configure the plugin with only one minor issue : when configuring Google as a provider for the first time, the "Tenant ID / Realm" field is hidden (as it should, if I understand your commit correctly), but as soon as parameters are saved, the field is displayed again and is never hidden after that. It does not seem to be a problem other than that.

Once configured, the login form is properly edited to show the "Continue with Google" button. Upon clicking it, I'm correctly redirected to Google's login interface, where I can auth, and be redirected to Redmine. However, problems start here. The login does not seems to go well, and I'm redirected to the login form again, with a weird looking Google error :

I initially suspected that this was due to a bad config on my side, but I have been searching since this morning and can't find what I'm doing wrong. I don't have much logs to hand out to you. Here is what I could find :

INFO -- : Started GET "/login?back_url=https%3A%2F%2Fredmine-dev.priv.atolcd.com%2F" for 172.16.60.3 at 2024-01-10 10:15:35 +0100
INFO -- : Processing by AccountController#login as HTML
INFO -- :   Parameters: {"back_url"=>"https://redmine-dev.priv.atolcd.com/"}
INFO -- :   Current user: anonymous
INFO -- :   Rendered account/login.html.erb within layouts/base (Duration: 3.1ms | Allocations: 1110)
INFO -- :   Rendered layout layouts/base.html.erb (Duration: 10.3ms | Allocations: 4086)
INFO -- : Completed 200 OK in 21ms (Views: 11.0ms | ActiveRecord: 2.4ms | Allocations: 5183)
INFO -- : Started GET "/oauth?utf8=%E2%9C%93&back_url=%2F&login-oauth=" for 172.16.60.3 at 2024-01-10 10:15:40 +0100
INFO -- : Processing by RedmineOauthController#oauth as HTML
INFO -- :   Parameters: {"utf8"=>"✓", "back_url"=>"/", "login-oauth"=>""}
INFO -- :   Current user: anonymous
INFO -- : Redirected to https://accounts.google.com/o/oauth2/v2/auth?client_id=MY_CLIENT_ID&redirect_uri=https%3A%2F%2Fredmine-dev.priv.atolcd.com%2Foauth2callback&response_type=code&scope=openid+profile+email&state=kK91zi989rECslMHr38rMD8J4enkeD2sXMJctmnF9bk%3D
INFO -- : Completed 302 Found in 15ms (ActiveRecord: 2.1ms | Allocations: 1325)
INFO -- : Started GET "/oauth2callback?state=kK91zi989rECslMHr38rMD8J4enkeD2sXMJctmnF9bk%3D&code=SOME_TOKEN&scope=email+profile+openid+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.profile+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email&authuser=0&hd=atolcd.com&prompt=consent" for 172.16.60.3 at 2024-01-10 10:15:43 +0100
INFO -- : Processing by RedmineOauthController#oauth_callback as HTML
INFO -- :   Parameters: {"state"=>"kK91zi989rECslMHr38rMD8J4enkeD2sXMJctmnF9bk=", "code"=>"SOME_TOKEN", "scope"=>"email profile openid https://www.googleapis.com/auth/userinfo.profile https://www.googleapis.com/auth/userinfo.email", "authuser"=>"0", "hd"=>"atolcd.com", "prompt"=>"consent"}
INFO -- :   Current user: anonymous
ERROR -- : <html lang=en><meta charset=utf-8><meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width"><title>Error 400 (Bad Request)!!1</title><style nonce="tHYzg2cxq-aC0vDYXmjCFg">*{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{color:#222;text-align:unset;margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px;}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}pre{white-space:pre-wrap;}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}</style><main id="af-error-container" role="main"><a href=//www.google.com><span id=logo aria-label=Google role=img></span></a><p><b>400.</b> <ins>That’s an error.</ins><p>The server cannot process the request because it is malformed. It should not be retried. <ins>That’s all we know.</ins></main>
INFO -- : Redirected to https://redmine-dev.priv.atolcd.com/login
INFO -- : Completed 302 Found in 188ms (ActiveRecord: 2.4ms | Allocations: 3218)

[qay21]

I might have an idea :

when 'Google'
        OAuth2::Client.new(
          Setting.plugin_redmine_oauth[:client_id],
          Setting.plugin_redmine_oauth[:client_secret],
          site: site,
          authorize_url: '/o/oauth2/v2/auth',
          token_url: '/o/oauth2/v2/auth'
        )

I think "token_url" here might be wrong. Google's OIDC discovery file seem to use "https://oauth2.googleapis.com/token" instead of "https://accounts.google.com/o/oauth2/v2/auth". I'm trying to see if I can test that locally

[picman]

I've fixed the problem with "Tenant ID / Realm".

Concerning URLs, it's strange that the domain for authorization and token differs.

authorization_endpoint "https://**accounts.google.com**/o/oauth2/v2/auth" token_endpoint "https://**oauth2.googleapis.com**/token"

But you can certainly try

when 'Google'
        OAuth2::Client.new(
          Setting.plugin_redmine_oauth[:client_id],
          Setting.plugin_redmine_oauth[:client_secret],
          site: site,
          authorize_url: '/o/oauth2/v2/auth',
          token_url: '/token'
        )

[picman]

https://github.com/googleapis/google-api-ruby-client/blob/main/docs/auth-guide.md

"auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://accounts.google.com/o/oauth2/token"

[qay21]

When creating credentials for my redmine instance in the Google Cloud Console, the generated client_secret JSON file states otherwise :

{
    "web": {
        "client_id": "MY_CLIENT_ID",
        "project_id": "redmine-409609",
        "auth_uri": "https://accounts.google.com/o/oauth2/auth",
        "token_uri": "https://oauth2.googleapis.com/token",
        "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
        "client_secret": "MY_CLIENT_SECRET",
        "redirect_uris": [
            "https://redmine-dev.priv.atolcd.com/oauth2callback"
        ],
        "javascript_origins": [
            "https://redmine-dev.priv.atolcd.com"
        ]
    }
}

I think there is indeed two different domains nowadays, which does not seem to be supported by the oauth gem. I will try with endpoints stated by Google in its google-api-ruby-client project, it is possible that there is some level of internal redirections on their side.

Working with Google for several years now, this type of inconsistencies are not uncommon, and it is quite annoying.

[qay21]

Also, it seems your commit is not using the values found in Google's repo : /o/oauth2/v2/{auth|token} vs /o/oauth2/{auth|token}. This might be a slight copy/paste issue

[qay21]

Okay, this is starting to look better :

INFO -- : Started GET "/login?back_url=https%3A%2F%2Fredmine-dev.priv.atolcd.com%2F" for 172.16.60.3 at 2024-01-10 14:31:04 +0100
INFO -- : Processing by AccountController#login as HTML
INFO -- :   Parameters: {"back_url"=>"https://redmine-dev.priv.atolcd.com/"}
INFO -- :   Current user: anonymous
INFO -- :   Rendered account/login.html.erb within layouts/base (Duration: 10.9ms | Allocations: 2676)
INFO -- :   Rendered layout layouts/base.html.erb (Duration: 19.5ms | Allocations: 5989)
INFO -- : Completed 200 OK in 40ms (Views: 24.8ms | ActiveRecord: 2.1ms | Allocations: 8875)
INFO -- : Started GET "/oauth?utf8=%E2%9C%93&back_url=%2F&login-oauth=" for 172.16.60.3 at 2024-01-10 14:31:07 +0100
INFO -- : Processing by RedmineOauthController#oauth as HTML
INFO -- :   Parameters: {"utf8"=>"✓", "back_url"=>"/", "login-oauth"=>""}
INFO -- :   Current user: anonymous
INFO -- : Redirected to https://accounts.google.com/o/oauth2/auth?client_id=MY_CLIENT_ID&redirect_uri=https%3A%2F%2Fredmine-dev.priv.atolcd.com%2Foauth2callback&response_type=code&scope=openid+profile+email&state=DtpJJbPpJHHkr8bNqRmRIkyjJlVED8NLJns5hcWHyZ8%3D
INFO -- : Completed 302 Found in 12ms (ActiveRecord: 1.5ms | Allocations: 1863)
INFO -- : Started GET "/oauth2callback?state=DtpJJbPpJHHkr8bNqRmRIkyjJlVED8NLJns5hcWHyZ8%3D&code=SOME_CODE&scope=email+profile+openid+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.profile+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email&authuser=0&hd=atolcd.com&prompt=consent" for 172.16.60.3 at 2024-01-10 14:31:11 +0100
INFO -- : Processing by RedmineOauthController#oauth_callback as HTML
INFO -- :   Parameters: {"state"=>"DtpJJbPpJHHkr8bNqRmRIkyjJlVED8NLJns5hcWHyZ8=", "code"=>"SOME_CODE", "scope"=>"email profile openid https://www.googleapis.com/auth/userinfo.profile https://www.googleapis.com/auth/userinfo.email", "authuser"=>"0", "hd"=>"atolcd.com", "prompt"=>"consent"}
INFO -- :   Current user: anonymous
ERROR -- : Invalid segment encoding
INFO -- : Redirected to https://redmine-dev.priv.atolcd.com/login
INFO -- : Completed 302 Found in 182ms (ActiveRecord: 1.8ms | Allocations: 4215)

this Invalid segment encoding thing does not seems to be a Google error, it is displayed in the normal Redmine theme, pretty much exactly like would be displayed a "Wrong credentials" error.

This test is using the following config :

when 'Google'
        OAuth2::Client.new(
          Setting.plugin_redmine_oauth[:client_id],
          Setting.plugin_redmine_oauth[:client_secret],
          site: site,
          authorize_url: '/o/oauth2/auth',
          token_url: '/o/oauth2/token'
        )

(that is to say the one stated in Google's repo, not the one currently commited to yours)

[picman]

I think that the token processing was wrong. There has to be another step to get user's profile from the obtained token. I've also tried to put an absolute URL into _tokenurl param. There are some debug output.

[picman]

URL changed to /oauth2/v2/userinfo

[qay21]

I think you modified Gitlab's conf, this is probably not correct

[picman]

oops, you are right. BTW, oauth gem contains a test for google

describe 'via 2-legged JWT assertion' do
    let(:client) do
      OAuth2::Client.new(
        '',
        '',
        site: 'https://accounts.google.com',
        authorize_url: '/o/oauth2/auth',
        token_url: '/o/oauth2/token',
        auth_scheme: :request_body
      )
    end

[qay21]

I can't get your Rails.logger.debug to show in my logs and I don't know why yet. I'll keep you posted once I succeed -_-

[picman]

You have to modify your configuration as follows:

_config/additionalenvironment.rb

config.log_level = :debug

[qay21]

Yep this is already in place and logs are filled with debug information including SQL requests and stuff, but still no sign of yours. I think I did something wrong building my last docker image, I am rebuilding/restarting at the moment to see if it goes better. Might have accidentally grabbed an old version of the code

[qay21]

There we go, it was an obscure problem of docker cache.

Now I can see your debug calls. Code and token are properly output, but the next step seems to send back a 404 from Google. That would be userinfo_response = token.get('/oauth2/v2/userinfo', headers: { 'Accept' => 'application/json' }). Consequently, next debug calls for response, login, and eail (email ?) are not displayed.

I guess this is not the right endpoint

[qay21]

According to Google's doc, user's informations are to be requested through yet another endpoint : userinfo_endpoint "https://openidconnect.googleapis.com/v1/userinfo".

[qay21]

I have the feeling that oauth gem is not at all expecting identity providers to use various endpoints like Google do. The token.get method is only able to make requests to customized path,but on the host configured for its client's site attribute.

However, since you are claiming openid profile email in the scope, the token should normally already contain pretty much everything you need, without having to make requests to Google's specific userinfo endpoint. Adding the name claim to the scope might even give access to family_name and given_name, in order to facilitate self registration (which is NOT something we are using, since we are syncing internal users from our LDAP server, so I will need to set up an additional instance to be able to test such a feature).

[picman]

Finally, I've bought a domain and registered a Google Workspace trial account for 14 days. I registered my web application, got Client ID and Client secret, registered Authorized redirect URI: http://localhost:3000. And, I can't get over Error 400 - redirect_uri_mismatch :-(

[j-goodwin]

Finally, I've bought a domain and registered a Google Workspace trial account for 14 days. I registered my web application, got Client ID and Client secret, registered Authorized redirect URI: http://localhost:3000. And, I can't get over Error 400 - redirect_uri_mismatch :-(

If the Google OAuth is the same as Microsoft then the redirect_uri protocol has to be https rather than http and the hostname has to be what is presented to the external world by the server i.e. you can't use localhost. If I remember, the hostname does not need to be resolvable by google but has to be what the server presents to google.

[qay21]

j-goodwin is right, redirection to localhost is permitted for development purposes but only with HTTPS.

Meanwhile, I also got some things to work :

when 'Google'
      Rails.logger.debug ">>> Test !"
      Rails.logger.debug ">>> code: #{params['code']}"
      token = oauth_client.auth_code.get_token(params['code'], redirect_uri: oauth_callback_url)
      Rails.logger.debug ">>> token: #{token.to_hash}"
      user_info = JWT.decode(token.to_hash['id_token'], nil, false).first
      user_info['login'] = user_info['email']
      Rails.logger.debug ">>> login: #{user_info['login']}"
      email = user_info['email']
      Rails.logger.debug ">>> eail: #{user_info['email']}"

This makes possible to log in to Redmine without error. I'm a bit unsatisfied with this strange to_hash['id_token'] thingy, but at least it contains the email of the authenticated account.

However, no name is to be found, so self registration might need some manual setting by the user here.

[qay21]

Plus, it seems weirder and weirder to me that the oauth gem would limit auth/token URLs to a single host. I can't find anything in Oauth or OIDC specs stating such a thing. Google might be the only provider to use multiple hosts for various endpoints, but they are apparently absolutely compliant doing so. The fact that Oauth gem can't get along with it is weird.

Side question, what lead you to name the plugin "Redmine Oauth" when you are essentially re-doing OIDC workflow with Oauth ? Are you planning more Oauth features that are not related to OIDC ?

[picman]

1. localhost I think that you should be able to authenticate via an OAuth provider also with your internal application. It's mentioned in many Google related forums that localhost works. Practically, it works by all other implemented providers including Microsoft. To eliminate this possibility I tested also a public domain. Unfortunately, with the same result Error 400.
2. If you get email from the token, we're pretty done I think. For basic authentication, email is sufficient. Can't be the reason for missing name a missing scope profile in the app registration? Have you added './auth/userinfo.profile' into application's scopes? Is your working client configured as follows?

   OAuth2::Client.new(
         Setting.plugin_redmine_oauth[:client_id],
         Setting.plugin_redmine_oauth[:client_secret],
         site: site,
         authorize_url: '/o/oauth2/auth',
         token_url: '/o/oauth2/token'
       )

3. OAuth. Except login to Redmine via an OAuth provider is there also implemented access to IMAP via OAuth to process incoming emails, e.g. issue updates, ...

[picman]

I got it working. Name is missing but email is essential. Please re-test the last commit with your app.

[qay21]

   Is your working client configured as follows?

OAuth2::Client.new(
          Setting.plugin_redmine_oauth[:client_id],
          Setting.plugin_redmine_oauth[:client_secret],
          site: site,
          authorize_url: '/o/oauth2/auth',
          token_url: '/o/oauth2/token'
        )

Yes it is. I'm a bit worried that these are sort of deprecated, since not specified in the well-known discovery file. So far, everything seems to work fine. I will do a full rebuild and test from scratch (my current image is a bit of a mess due to me trying to debug Google, understand Oauth and learn Ruby at the same time)

[picman]

One more update. Now it gets name too.

[qay21]

Oooooh you can actually use full URLs instead of just paths ! This is great, current oauth gem's documentation mislead me on this one.

[qay21]

That being said I'm unsure if Google's name should be mapped to Redmine's login. I'll see what my Workspace organization is using as a name, but it might actually be a full name with spaces

[picman]

You're right. We can put email there instead.

[qay21]

Another good thing : oauth gem is in fact using Faraday to build its URLs, so we should use OIDC discovery values instead of 4 years old Google docs values I think.

That would be :

when 'Google'
        OAuth2::Client.new(
          Setting.plugin_redmine_oauth[:client_id],
          Setting.plugin_redmine_oauth[:client_secret],
          site: site,
          authorize_url: '/o/oauth2/v2/auth',
          token_url: 'https://oauth2.googleapis.com/token'
        )

Faraday will understand that token_url is a full URL and not just a path, and will consequently not append it to the site value

[picman]

Okay

[qay21]

Beware, auth URL is now on v2 : /o/oauth2/v2/auth. There is a good chance that /o/oauth2/auth will cease to answer requests without warning at some point

[picman]

I don't understand. What is better to use then? /o/oauth2/v2/auth or /o/oauth2/auth ?

[qay21]

The one stated in Google's OIDC well-known is https://accounts.google.com/o/oauth2/v2/auth, hence my comment changing both auth AND token URL. But for the auth URL you simply have missed the added "v2", which is hard to notice :smiley:

[picman]

Okay

[qay21]

Everything is working mighty fine !

[picman]

So we can probably close it. I will release it soon.

[qay21]

Many thanks for your work and reactivity !

[picman]

You contributed a lot too!