[HTTP/401 on repo.jenkins-ci.org] Fix LDAP user configurations in Artifactory to avoid unexpected HTTP/401 when logging in

dduportal commented 1 year ago

Service(s)

Artifactory

Summary

We recently (past 2 weeks) had a lot of contribuotr users failing to release their plugin due to an authentication problem in Artifactory (repo.jenkins-ci.org): HTTP/401 answered.

All accounts are valid LDAP accounts (all users have successfully logged-in on accounts.jenkins.io and/or issues.jenkins.io as a technical proof)
The HTTP/401 happens when logging-in in the Artifactory UI, when using the Artifactory API and also when releasing manually a plugin (with the LDAP credentials set up in the Maven's settings.xml).
- No HTTP/401 happens when using the "CD" process from JEP-299 as it uses short-lived access tokens

The solution is to update user configuration in Artifactory itself (a "proxy" user is created inside Artifactory database when a LDAP account logs in the first time) by setting the boolean attribute "Disable Internal Password" to true to force Artifactory considering only the LDAP system for password instead of checking the "proxy" user local password (??) as described by https://github.com/jenkins-infra/helpdesk/issues/3680#issuecomment-1651716208.

The following cases where treated with this problem:

This issue is to track the actions required to fix all users:

Contacting JFrog about this behavior to have their recommendation and diagnostic
And/or doing a batch script using the Artifactory API to force this attribute to true to all users
And finding a solution to force this setting to "true" (otherwise we'll keep having it over time)

Reproduction steps

No response

dtbaum commented 1 year ago

I have the same issue:

Login at https://accounts.jenkins.io/ works
login at https://repo.jenkins-ci.org/ui/login/ failed with "Login has failed. Due to Incorrect username/password or locked user."
grypescanner plugin release failed with: " status code: 401, reason phrase: (401)"

@dduportal: I'm not sure: Should I delete this comment here and create a separate ticket for my issue instead?

olamy commented 1 year ago

same here for olamy user. Not sure if we need to create some new issues for that.

olamy commented 1 year ago

All good. Thanks @daniel-beck to fix my issue.

dtbaum commented 1 year ago

All good. Thanks @daniel-beck to fix my issue.

I would be happy to thank as well, but my issue is still unresolved :-)

dduportal commented 1 year ago

@dtbaum Currently looking. You did good to comment here (no problem to have a separated issue if it is ok for you , both are ok 👍 )

dduportal commented 1 year ago

@dtbaum your user account tbaum has been updated: can you retry?

dtbaum commented 1 year ago

My issue is fixed! Thank you!

dduportal commented 1 year ago

Sent an email to Jfrog about this behavioral change, before trying any other action

timja commented 1 year ago

@smerle33 is there a reason you unpinned this? it's affecting a lot of people so shouldn't it be highlighted?

KostyaSha commented 1 year ago

integer has issue, could you check?

lemeurherve commented 1 year ago

@smerle33 is there a reason you unpinned this? it's affecting a lot of people so shouldn't it be highlighted?

Stéphane mentioned today that he unpinned an issue by mistake but closed the tab and didn't remember which one it was, thanks for the comment it I've repinned the issue.

lemeurherve commented 1 year ago

integer has issue, could you check?

@KostyaSha I don't have the necessary permissions on the Artifactory for that but it will be taken care of quickly by someone who does.

daniel-beck commented 1 year ago

@KostyaSha Your account should be fixed.

dduportal commented 1 year ago

More cases:

dduportal commented 1 year ago

Update: with the help of JFrog user, we were able to tackle the challenge of the new users.

Since a few days, any new Jenkins LDAP user, when logging in to repo.jenkins-ci.org for the first time, are created with the option "Disable Internal password" to avoid ther problem.

We confirmed that behavior with 2 new users we created in parallel and tested to be sure (cleaned up since through my admin account).

Next step is to batch update all existing users. JFrog support is checking if they have this tooling internally and will come back to us. If they don't, we'll have to write and run a script using the API (https://jfrog.com/help/r/jfrog-rest-apis/update-a-user-partial-update) to to it ourselves.

nfalco79 commented 1 year ago

I have the same issue. Login to https://accounts.jenkins.io/myself/ works but fail to login in repo.jenkins-ci.org and when I run mvn release:perform (HTTP 401)

I'm the maintainer of nodejs, dependency-check and xunit plugins. I get failure on plugin release. Anyone can help?

LDAP username nfalco GitHub username nfalco79 nfalco79(at)hotmail.com

dduportal commented 1 year ago

I have the same issue. Login to https://accounts.jenkins.io/myself/ works but fail to login in repo.jenkins-ci.org and when I run mvn release:perform (HTTP 401)

I'm the maintainer of nodejs, dependency-check and xunit plugins. I get failure on plugin release. Anyone can help?

LDAP username nfalco GitHub username nfalco79 nfalco79(at)hotmail.com

Hi @nfalco79 , your account has been updated (and also the account nfalco79 which is using the same email). Can you retry?

(Edit I've seen youe initial message in https://github.com/jenkins-infra/helpdesk/issues/3702 but marked it as off-topic, as well as my answer, as your message here is easier to track and gives more information)

nfalco79 commented 1 year ago

I'm able to login thanks very much!

dduportal commented 1 year ago

For info: new problem with Artifactory today when updating "faulty" user profiles: https://github.com/jenkins-infra/helpdesk/issues/3724#issuecomment-1689490695

Gotta check the API

rsandell commented 1 year ago

me too

dduportal commented 1 year ago

Good news, the APi still works 🎉

Just ran:

curl -X PATCH -v -H "Authorization: Bearer $TOKEN" https://repo.jenkins-ci.org/access/api/v2/users/rsandell -H 'Content-Type: application/json' -H 'Accept: application/json' -d '{"internal_password_disabled": true}'

and now @rsandell 's profile looks good. @rsandell can you retry?

rsandell commented 1 year ago

I am able to login to the ui, but I still get a 403 error when trying to deploy an artifact

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:3.1.1:deploy (default-deploy) on project active-directory: Failed to deploy artifacts: Could not transfer artifact org.jenkins-ci.plugins:active-directory:hpi:2.32 from/to maven.jenkins-ci.org (https://repo.jenkins-ci.org/releases/): status code: 403, reason phrase: (403)

sarah-witt commented 1 year ago

Hello, we are getting 401 errors releasing the datadog plugin using the datadog user. Could you update the account? Thanks!

dduportal commented 1 year ago

Hello, we are getting 401 errors releasing the datadog plugin using the datadog user. Could you update the account? Thanks!

Hello @sarah-witt thanks for raising the concern. The datadog account in Artifactory has been updated and you should be able to log-in with the Jenkins password (and proceed with the plugin release).

dduportal commented 1 year ago

I am able to login to the ui, but I still get a 403 error when trying to deploy an artifact

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:3.1.1:deploy (default-deploy) on project active-directory: Failed to deploy artifacts: Could not transfer artifact org.jenkins-ci.plugins:active-directory:hpi:2.32 from/to maven.jenkins-ci.org (https://repo.jenkins-ci.org/releases/): status code: 403, reason phrase: (403)

Thanks @rsandell . The change HTTP/401 -> HTTP/403 shows that the applied change works as expected. The new problem is related to permissions. Ping @daniel-beck were you able to find the cause for this case (or was it a partial release) with @rsandell ?

daniel-beck commented 1 year ago

The 403 is what motivated me asking about better logs elsewhere 😉 Increasing the version number and retrying worked for @rsandell.

sarah-witt commented 1 year ago

Thanks @dduportal, we were able to release!

Vlatombe commented 1 year ago

Hi @dduportal, I also have login issues (vlatombe) on Artifactory (but not on accounts or jira). Can you fix my account?

dduportal commented 1 year ago

@Vlatombe done (you're lucky I am working with the batch update of all users and just see your message ;) )

Vlatombe commented 1 year ago

Thanks @dduportal !

dduportal commented 1 year ago

Update: working on the batch updated. A few numbers with my current script:

We have a total of 2723 users (all kinds) including 2703 LDAP users
We have (right after Vincent's profile updated) 1286 users with the attribute internal_password_disabled not set to true
It takes ~3s to retrieve the list of all users (requires 3 requests) but we only know their username, their realm kind (LDAP, admin, other) and if they are enabled or not
- The Artifactory API does not allow to filter user based on other attributes => we have to batch update all users which takes some time
It takes ~ 23 minutes to retrieve ALL users with detailled atrributes

=> I expect to be running the new batch later today. It should take 1 hour:

10 min testing
12 min of execution
23 min of "retrieving updated users"
15 min of post checks

dduportal commented 1 year ago

Update: Currently running the following API call on each LDAP enabled user:

curl --fail --location --silent --show-error --header "Authorization: Bearer <super secret token not stored in clear and valid only 1 day>" \
    --header 'Content-Type: application/json' --header 'Accept: application/json' --data '{"internal_password_disabled": true}' \
    --request PATCH \
      "https://repo.jenkins-ci.org/access/api/v2/users/${ldap_user}"

Result in ~1 hour

dduportal commented 1 year ago

Update: all users have been updated to a proper configuration with succes. Running one last check to ensure there are no edge cases.

dduportal commented 1 year ago

Last check:

$ bash ./check-artifactory-users.sh                                        
Total # of users: 2724
Total # of LDAP enabled users: 2704
Total # of misconfigured users: 0

=> We can close the issue! Feel free to reopen if you still see HTTP/401 errors

jenkins-infra / helpdesk