Unable to login to Grafana

aktech commented 3 months ago

Context

This is what I see while trying to login to Grafana:

Logs:

➜  journalctl -u grafana-server.service -f
Jul 25 09:54:20 jhub-login grafana[986907]: logger=context userId=0 orgId=0 uname= t=2024-07-25T09:54:20.768233638-05:00 level=info msg="OAuth auto login enabled. Redirecting to /monitoring/login/generic_oauth"
Jul 25 09:54:20 jhub-login grafana[986907]: logger=context userId=0 orgId=0 uname= t=2024-07-25T09:54:20.768539919-05:00 level=info msg="Request Completed" method=GET path=/login status=307 remote_addr=192.166.211.218 time_ms=3 duration=3.14035ms size=67 referer= handler=/login status_source=server
Jul 25 09:54:20 jhub-login grafana[986907]: logger=context userId=0 orgId=0 uname= t=2024-07-25T09:54:20.873726215-05:00 level=info msg="Request Completed" method=GET path=/login/generic_oauth status=302 remote_addr=192.166.211.218 time_ms=0 duration=82.624µs size=306 referer= handler=/login/:name status_source=server
Jul 25 09:54:21 jhub-login grafana[986907]: logger=user.sync t=2024-07-25T09:54:21.10607955-05:00 level=error msg="Failed to create user" error="user already exists" auth_module=oauth_generic_oauth auth_id=4d4a9bc3-efe9-4e95-b1b9-3a54c89796fb
Jul 25 09:54:21 jhub-login grafana[986907]: logger=authn.service t=2024-07-25T09:54:21.106350655-05:00 level=error msg="Failed to run post auth hook" client=auth.client.generic_oauth id=: error="[user.sync.internal] unable to create user: user already exists"
Jul 25 09:54:21 jhub-login grafana[986907]: logger=context userId=0 orgId=0 uname= t=2024-07-25T09:54:21.112687295-05:00 level=info msg="Request Completed" method=GET path=/login/generic_oauth status=302 remote_addr=192.166.211.218 time_ms=20 duration=20.169282ms size=40 referer= handler=/login/:name status_source=server
Jul 25 09:54:33 jhub-login grafana[986907]: logger=context userId=0 orgId=0 uname= t=2024-07-25T09:54:33.319209421-05:00 level=info msg="Request Completed" method=GET path=/login/generic_oauth status=302 remote_addr=192.166.211.218 time_ms=0 duration=99.04µs size=306 referer=https://<NEBARI-SLURM-URL-REMOVED>/monitoring/login handler=/login/:name status_source=server
Jul 25 09:54:33 jhub-login grafana[986907]: logger=user.sync t=2024-07-25T09:54:33.550794803-05:00 level=error msg="Failed to create user" error="user already exists" auth_module=oauth_generic_oauth auth_id=4d4a9bc3-efe9-4e95-b1b9-3a54c89796fb
Jul 25 09:54:33 jhub-login grafana[986907]: logger=authn.service t=2024-07-25T09:54:33.550848423-05:00 level=error msg="Failed to run post auth hook" client=auth.client.generic_oauth id=: error="[user.sync.internal] unable to create user: user already exists"
Jul 25 09:54:33 jhub-login grafana[986907]: logger=context userId=0 orgId=0 uname= t=2024-07-25T09:54:33.557529344-05:00 level=info msg="Request Completed" method=GET path=/login/generic_oauth status=302 remote_addr=192.166.211.218 time_ms=19 duration=19.997124ms size=40 referer= handler=/login/:name status_source=server

Potentially relevant upstream issues:

Potential solution: https://github.com/grafana/grafana/issues/70203#issuecomment-1612823390 (or upgrade grafana)

Value and/or benefit

Being able to monitor nebari-slurm deployment.

Anything else?

No response

lcapka commented 1 month ago

Hi @aktech, were you able to solve this issue? We are having the same problem.

aktech commented 1 month ago

What version are you on @lcapka ?

Heavybullets8 commented 1 month ago

I encountered a sign-in issue with Grafana due to a conflict between Grafana's default admin user and my LDAP admin user. (LDAP is used by my OAuth provider, Authelia)

My LDAP User: admin
My LDAP Group: admins

Grafana creates a default admin user upon initial setup, which conflicted with my LDAP admin user.

Note: Changing my LDAP username was an option, but I chose to adjust Grafana's configuration instead. Honestly not sure which is the correct method for solving this.

The Steps I Personally Took

1. Disable Grafana's Default `admin` User Creation

To prevent Grafana from creating the default admin user, I disabled the initial admin creation in the Grafana configuration.

Environment Variable (what I used):

GF_SECURITY_DISABLE_INITIAL_ADMIN_CREATION: true

Equivalent grafana.ini Configuration:

#################################### Security ############################
[security]
# Disable creation of admin user on first start of Grafana
disable_initial_admin_creation = true

2. Configure LDAP Admin User

Since my LDAP user admin is part of the admins group, I configured Grafana to assign the Admin role to users in the admins group.

Environment Variables (what I used):

GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH: contains(groups, 'admins') && 'Admin' || 'Viewer'
GF_AUTH_GENERIC_OAUTH_ALLOW_ASSIGN_GRAFANA_ADMIN: true

Equivalent grafana.ini Configuration:

[auth.generic_oauth]
role_attribute_path = contains(groups, 'admins') && 'Admin' || 'Viewer'
allow_assign_grafana_admin = true

3. Reinitialize Grafana Database

To apply the changes, I needed to remove Grafana and reset the database.

Steps:
1. Remove Grafana: Stop and remove the Grafana instance.
2. Drop the Grafana Database: Remove the existing Grafana database from CNPG (Cloud Native PostgreSQL).
3. Re-add Grafana: Deploy Grafana again with the updated configuration.

Afterwards, everything came up for me. May or may not work for you depending on your setup.

Alternative Solutions Attempted

Some users have reported success with the following settings, but they did not work for me.

Environment Variables:

GF_AUTH_GENERIC_OAUTH_SKIP_ORG_ROLE_SYNC: true
GF_AUTH_OAUTH_ALLOW_INSECURE_EMAIL_LOOKUP: true

Equivalent grafana.ini Configuration:

[auth]
oauth_allow_insecure_email_lookup = true

[auth.generic_oauth]
skip_org_role_sync = true

nebari-dev / nebari-slurm