Closed Red-M closed 1 year ago
Hello! Could you share your Kubernetes config files? We tested several scenarios before releasing 3.7.0 and did not notice any issues.
This is the grafana.ini in the Kube CM:
[alerting]
enabled = false
[analytics]
check_for_updates = true
[auth.azuread]
allow_assign_grafana_admin = true
allow_sign_up = false
auth_url = https://login.microsoftonline.com/tenantID/oauth2/v2.0/authorize
auto_login = false
enabled = true
name = AzureAD
oauth_skip_org_role_update_sync = true
role_attribute_strict = false
scopes = openid email profile
skip_org_role_sync = true
token_url = https://login.microsoftonline.com/tenantID/oauth2/v2.0/token
[auth.generic_oauth]
allow_signup = false
api_url = https://Oauth-provider/userinfo
auth_url = https://Oauth-provider/authorize
enabled = true
name = OAuth
scopes = openid profile email
token_url = https://Oauth-provider/oauth/token
[feature_toggles]
enable = tempoSearch tempoBackendSearch traceqlEditor panelTitleSearch
publicDashboards autoMigrateGraphPanels
[grafana_net]
url = https://grafana.net
[live]
max_connections = 0
[log]
level = warn
mode = console
[panels]
enable_alpha = true
[paths]
data = /var/lib/grafana/
logs = /var/log/grafana
plugins = /var/lib/grafana/plugins
provisioning = /etc/grafana/provisioning
[plugins]
enable_alpha = true
[rendering]
callback_url = http://grafana:80/
server_url = http://grafana-image-renderer:8081/render
[server]
domain = grafana.uat
root_url = https://grafana.uat/
[unified_alerting]
enabled = true
ha_peers = grafana-headless:9094
[unified_alerting.screenshots]
capture = true
capture_timeout = 30s
Here is the env variables that the grafana pod has:
env:
- name: GF_AUTH_GENERIC_OAUTH_CLIENT_ID
valueFrom:
secretKeyRef:
key: client-id
name: grafana-auth
- name: GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET
valueFrom:
secretKeyRef:
key: client-secret
name: grafana-auth
- name: GF_AUTH_AZUREAD_CLIENT_ID
valueFrom:
secretKeyRef:
key: aad-client-id
name: grafana-auth
- name: GF_AUTH_AZUREAD_CLIENT_SECRET
valueFrom:
secretKeyRef:
key: aad-client-secret
name: grafana-auth
- name: GF_DATABASE_USER
valueFrom:
secretKeyRef:
key: username
name: grafana-db
- name: GF_DATABASE_NAME
valueFrom:
secretKeyRef:
key: database
name: grafana-db
- name: GF_DATABASE_PASSWORD
valueFrom:
secretKeyRef:
key: password
name: grafana-db
- name: POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: GF_SECURITY_ADMIN_USER
valueFrom:
secretKeyRef:
key: admin-user
name: grafana-admin
- name: GF_SECURITY_ADMIN_PASSWORD
valueFrom:
secretKeyRef:
key: admin-password
name: grafana-admin
- name: GF_SMTP_USER
valueFrom:
secretKeyRef:
key: user
name: grafana-smtp
- name: GF_SMTP_PASSWORD
valueFrom:
secretKeyRef:
key: password
name: grafana-smtp
- name: GF_RENDERING_SERVER_URL
value: 'http://grafana-image-renderer.monitoring:8081/render'
- name: GF_RENDERING_CALLBACK_URL
value: 'http://grafana.monitoring:80/'
- name: GF_PATHS_DATA
value: /var/lib/grafana/
- name: GF_PATHS_LOGS
value: /var/log/grafana
- name: GF_PATHS_PLUGINS
value: /var/lib/grafana/plugins
- name: GF_PATHS_PROVISIONING
value: /etc/grafana/provisioning
- name: GF_AUTH_BASIC_ENABLED
value: 'false'
- name: GF_AUTH_DISABLE_LOGIN_FORM
value: 'true'
- name: GF_AUTH_PROXY_AUTO_SIGN_UP
value: 'false'
- name: GF_AUTH_PROXY_ENABLED
value: 'true'
- name: GF_AUTH_PROXY_HEADERS
value: 'Name:X-WEBAUTH-NAME Email:X-WEBAUTH-EMAIL'
- name: GF_AUTH_PROXY_HEADER_NAME
value: X-WEBAUTH-USERNAME
- name: GF_AUTH_PROXY_HEADER_PROPERTY
value: username
- name: GF_AUTH_PROXY_WHITELIST
value: 'CENSORED'
- name: GF_DATABASE_HOST
value: CENSORED:3306
- name: GF_DATABASE_MAX_IDLE_CONN
value: '20'
- name: GF_DATABASE_MAX_OPEN_CONN
value: '100'
- name: GF_DATABASE_SSL_MODE
value: 'false'
- name: GF_DATABASE_TYPE
value: mysql
- name: GF_DATAPROXY_TIMEOUT
value: '600'
- name: GF_INSTALL_PLUGINS
value: >-
grafana-piechart-panel,raintank-worldping-app,jdbranham-diagram-panel,goshposh-metaqueries-datasource,vonage-status-panel,petrslavotinek-carpetplot-panel,grafana-polystat-panel,snuids-radar-panel,yesoreyeram-boomtheme-panel,digrich-bubblechart-panel,agenty-flowcharting-panel,citilogics-geoloop-panel,michaeldmoore-multistat-panel,corpglory-progresslist-panel,marcuscalidus-svg-panel,fetzerch-sunandmoon-datasource,snuids-trafficlights-panel,http://grafana-streamer-plugin.default.svc.cluster.local:8080/streamer.zip;streamer,marcusolsson-treemap-panel,grafana-sentry-datasource
- name: GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS
value: >-
grafana-piechart-panel,raintank-worldping-app,jdbranham-diagram-panel,goshposh-metaqueries-datasource,vonage-status-panel,petrslavotinek-carpetplot-panel,grafana-polystat-panel,snuids-radar-panel,yesoreyeram-boomtheme-panel,digrich-bubblechart-panel,agenty-flowcharting-panel,citilogics-geoloop-panel,michaeldmoore-multistat-panel,corpglory-progresslist-panel,marcuscalidus-svg-panel,fetzerch-sunandmoon-datasource,snuids-trafficlights-panel,streamer,marcusolsson-treemap-panel
- name: GF_RENDERING_RENDERER_TOKEN
value: >-
map[valueFrom:map[secretKeyRef:map[key:token
name:grafana-renderer]]]
- name: GF_SECURITY_ALLOW_EMBEDDING
value: 'true'
- name: GF_SECURITY_COOKIE_SECURE
value: 'true'
- name: GF_SERVER_ROOT_URL
value: 'https://grafana.uat/'
- name: GF_SESSION_COOKIE_NAME
value: session
- name: GF_SESSION_COOKIE_SECURE
value: 'true'
- name: GF_SESSION_PROVIDER
value: mysql
- name: GF_SESSION_PROVIDER_CONFIG
value: >-
CENSORED@tcp(CENSORED:3306)/grafana
- name: GF_SMTP_ENABLED
value: 'true'
- name: GF_SMTP_FROM_ADDRESS
value: noreply@CENSORED
- name: GF_SMTP_FROM_NAME
value: Entain Grafana
- name: GF_SMTP_HOST
value: 'smtp.CENSORED:587'
- name: GF_USERS_VIEWERS_CAN_EDIT
value: 'true'
here is the env values the renderer has:
env:
- name: HTTP_PORT
value: '8081'
- name: AUTH_TOKEN
value: >-
map[valueFrom:map[secretKeyRef:map[key:token
name:grafana-renderer]]]
- name: HTTP_HOST
value: 0.0.0.0
- name: IGNORE_HTTPS_ERRORS
value: 'true'
- name: RENDERING_ARGS
value: >-
--no-sandbox,--disable-setuid-sandbox,--disable-dev-shm-usage,--disable-accelerated-2d-canvas,--disable-gpu,--window-size=1280x758
- name: RENDERING_CLUSTERING_MAX_CONCURRENCY
value: '5'
- name: RENDERING_CLUSTERING_MODE
value: context
- name: RENDERING_MODE
value: clustered
I don't have a lot of knowledge around Kubernetes but after a few tests, I would say this is incorrect:
- name: AUTH_TOKEN
value: >-
map[valueFrom:map[secretKeyRef:map[key:token
name:grafana-renderer]]]
And should be replaced with:
- name: AUTH_TOKEN
valueFrom:
secretKeyRef:
key: token
name: grafana-renderer
With the image renderer 3.6.4, you didn't use your Kubernetes secret but the string:
map[valueFrom:map[secretKeyRef:map[key:token
name:grafana-renderer]]]
In 3.7.0, it broke because I had to chose a separator when allowing multiple tokens and decided to use the whitespace character as it's usually not used in authentication tokens.
Thanks, I'll let the rest of my team know, I may provide a patch so that this works correctly in the helm chart at least.
What happened: An unrelated change pushed the version of grafana-image-renderer in my environment to 3.7.0 which is ok but suddenly it started to receive 401 unauth responses from grafana. A rollback to 3.6.4 fixed this.
I did try hardcoding the token to
test
on grafana and grafana-image-renderer and it was still getting the same thing.What you expected to happen: 3.7.0 worked with existing functioning configuration.
How to reproduce it (as minimally and precisely as possible): Upgrade to 3.7.0 as far as I can see. Happy to provide more detail if required.
Anything else we need to know?: Running on kubernetes using the grafana provided chart which defaults to
latest
.Environment: