nebari-dev / nebari-slurm

An opinionated open source deployment of jupyterhub based on an Slurm job scheduler.
BSD 3-Clause "New" or "Revised" License
28 stars 10 forks source link

Unable to login to Grafana #165

Open aktech opened 4 months ago

aktech commented 4 months ago

Context

This is what I see while trying to login to Grafana:

Screenshot 2024-07-25 at 3 59 43 pm

Logs:

➜  journalctl -u grafana-server.service -f
Jul 25 09:54:20 jhub-login grafana[986907]: logger=context userId=0 orgId=0 uname= t=2024-07-25T09:54:20.768233638-05:00 level=info msg="OAuth auto login enabled. Redirecting to /monitoring/login/generic_oauth"
Jul 25 09:54:20 jhub-login grafana[986907]: logger=context userId=0 orgId=0 uname= t=2024-07-25T09:54:20.768539919-05:00 level=info msg="Request Completed" method=GET path=/login status=307 remote_addr=192.166.211.218 time_ms=3 duration=3.14035ms size=67 referer= handler=/login status_source=server
Jul 25 09:54:20 jhub-login grafana[986907]: logger=context userId=0 orgId=0 uname= t=2024-07-25T09:54:20.873726215-05:00 level=info msg="Request Completed" method=GET path=/login/generic_oauth status=302 remote_addr=192.166.211.218 time_ms=0 duration=82.624µs size=306 referer= handler=/login/:name status_source=server
Jul 25 09:54:21 jhub-login grafana[986907]: logger=user.sync t=2024-07-25T09:54:21.10607955-05:00 level=error msg="Failed to create user" error="user already exists" auth_module=oauth_generic_oauth auth_id=4d4a9bc3-efe9-4e95-b1b9-3a54c89796fb
Jul 25 09:54:21 jhub-login grafana[986907]: logger=authn.service t=2024-07-25T09:54:21.106350655-05:00 level=error msg="Failed to run post auth hook" client=auth.client.generic_oauth id=: error="[user.sync.internal] unable to create user: user already exists"
Jul 25 09:54:21 jhub-login grafana[986907]: logger=context userId=0 orgId=0 uname= t=2024-07-25T09:54:21.112687295-05:00 level=info msg="Request Completed" method=GET path=/login/generic_oauth status=302 remote_addr=192.166.211.218 time_ms=20 duration=20.169282ms size=40 referer= handler=/login/:name status_source=server
Jul 25 09:54:33 jhub-login grafana[986907]: logger=context userId=0 orgId=0 uname= t=2024-07-25T09:54:33.319209421-05:00 level=info msg="Request Completed" method=GET path=/login/generic_oauth status=302 remote_addr=192.166.211.218 time_ms=0 duration=99.04µs size=306 referer=https://<NEBARI-SLURM-URL-REMOVED>/monitoring/login handler=/login/:name status_source=server
Jul 25 09:54:33 jhub-login grafana[986907]: logger=user.sync t=2024-07-25T09:54:33.550794803-05:00 level=error msg="Failed to create user" error="user already exists" auth_module=oauth_generic_oauth auth_id=4d4a9bc3-efe9-4e95-b1b9-3a54c89796fb
Jul 25 09:54:33 jhub-login grafana[986907]: logger=authn.service t=2024-07-25T09:54:33.550848423-05:00 level=error msg="Failed to run post auth hook" client=auth.client.generic_oauth id=: error="[user.sync.internal] unable to create user: user already exists"
Jul 25 09:54:33 jhub-login grafana[986907]: logger=context userId=0 orgId=0 uname= t=2024-07-25T09:54:33.557529344-05:00 level=info msg="Request Completed" method=GET path=/login/generic_oauth status=302 remote_addr=192.166.211.218 time_ms=19 duration=19.997124ms size=40 referer= handler=/login/:name status_source=server

Potentially relevant upstream issues:

Potential solution: https://github.com/grafana/grafana/issues/70203#issuecomment-1612823390 (or upgrade grafana)

Value and/or benefit

Being able to monitor nebari-slurm deployment.

Anything else?

No response

lcapka commented 1 month ago

Hi @aktech, were you able to solve this issue? We are having the same problem.

aktech commented 1 month ago

What version are you on @lcapka ?

Heavybullets8 commented 1 month ago

I encountered a sign-in issue with Grafana due to a conflict between Grafana's default admin user and my LDAP admin user. (LDAP is used by my OAuth provider, Authelia)

Grafana creates a default admin user upon initial setup, which conflicted with my LDAP admin user.

Note: Changing my LDAP username was an option, but I chose to adjust Grafana's configuration instead. Honestly not sure which is the correct method for solving this.


The Steps I Personally Took

1. Disable Grafana's Default admin User Creation

To prevent Grafana from creating the default admin user, I disabled the initial admin creation in the Grafana configuration.

Environment Variable (what I used):

GF_SECURITY_DISABLE_INITIAL_ADMIN_CREATION: true

Equivalent grafana.ini Configuration:

#################################### Security ############################
[security]
# Disable creation of admin user on first start of Grafana
disable_initial_admin_creation = true

2. Configure LDAP Admin User

Since my LDAP user admin is part of the admins group, I configured Grafana to assign the Admin role to users in the admins group.

Environment Variables (what I used):

GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH: contains(groups, 'admins') && 'Admin' || 'Viewer'
GF_AUTH_GENERIC_OAUTH_ALLOW_ASSIGN_GRAFANA_ADMIN: true

Equivalent grafana.ini Configuration:

[auth.generic_oauth]
role_attribute_path = contains(groups, 'admins') && 'Admin' || 'Viewer'
allow_assign_grafana_admin = true

3. Reinitialize Grafana Database

To apply the changes, I needed to remove Grafana and reset the database.

Afterwards, everything came up for me. May or may not work for you depending on your setup.


Alternative Solutions Attempted

Some users have reported success with the following settings, but they did not work for me.

Environment Variables:

GF_AUTH_GENERIC_OAUTH_SKIP_ORG_ROLE_SYNC: true
GF_AUTH_OAUTH_ALLOW_INSECURE_EMAIL_LOOKUP: true

Equivalent grafana.ini Configuration:

[auth]
oauth_allow_insecure_email_lookup = true

[auth.generic_oauth]
skip_org_role_sync = true