search

secretflow / kuscia

Kuscia(Kubernetes-based Secure Collaborative InfrA) is a K8s-based privacy-preserving computing task orchestration framework.
https://www.secretflow.org.cn/docs/kuscia/latest/zh-Hans
Apache License 2.0
73 stars 56 forks source link

Empty grafana dashboard #390

Closed henrysun007 closed 3 months ago

henrysun007 commented 4 months ago

Issue Type

Running

Search for existing issues similar to yours

Yes

OS Platform and Distribution

ubuntu 22.04

Kuscia Version

0.10.0

Deployment

k8s

deployment Version

1.28

App Running type

others

App Running version

NA

Configuration file used to run kuscia.

Not relevant

What happend and What you expected to happen.

The dashboard is empty.

Kuscia log output.

grafana error output:
logger=datasources t=2024-07-22T18:30:00.302745188+08:00 level=error msg="Failed getting data source" err="context canceled" uid=PAF26937B51086004 id=0 name= orgId=1
logger=context userId=1 orgId=1 uname=admin t=2024-07-22T18:30:00.302993029+08:00 level=error msg="Client closed request" error="[api.requestCanceled] response: request canceled: context canceled" remote_addr=11.163.85.164 traceID=
logger=datasources t=2024-07-22T18:30:00.303420208+08:00 level=error msg="Failed getting data source" err="context canceled" uid=PAF26937B51086004 id=0 name= orgId=1
logger=context userId=1 orgId=1 uname=admin t=2024-07-22T18:30:00.305943708+08:00 level=info msg="Request Completed" method=POST path=/api/ds/query status=499 remote_addr=11.163.85.164 time_ms=36 duration=36.047636ms size=94 referer="http://11.163.85.164:30608/d/1552593593532704-17695-1-1/machine-center?editview=dashboard_json&orgId=1&refresh=5s" handler=/api/ds/query
logger=context userId=1 orgId=1 uname=admin t=2024-07-22T18:30:00.306197545+08:00 level=error msg="Client closed request" error="[api.requestCanceled] response: request canceled: context canceled" remote_addr=11.163.85.164 traceID=
logger=context userId=1 orgId=1 uname=admin t=2024-07-22T18:30:00.306326629+08:00 level=info msg="Request Completed" method=POST path=/api/ds/query status=499 remote_addr=11.163.85.164 time_ms=36 duration=36.082798ms size=94 referer="http://11.163.85.164:30608/d/1552593593532704-17695-1-1/machine-center?editview=dashboard_json&orgId=1&refresh=5s" handler=/api/ds/query
logger=datasources t=2024-07-22T18:30:00.325777935+08:00 level=error msg="Failed getting data source" err="context canceled" uid=PAF26937B51086004 id=0 name= orgId=1
logger=context userId=1 orgId=1 uname=admin t=2024-07-22T18:30:00.325934916+08:00 level=error msg="Client closed request" error="[api.requestCanceled] response: request canceled: context canceled" remote_addr=11.163.85.164 traceID=
logger=context userId=1 orgId=1 uname=admin t=2024-07-22T18:30:00.326043152+08:00 level=info msg="Request Completed" method=POST path=/api/ds/query status=499 remote_addr=11.163.85.164 time_ms=55 duration=55.766045ms size=94 referer="http://11.163.85.164:30608/d/1552593593532704-17695-1-1/machine-center?editview=dashboard_json&orgId=1&refresh=5s" handler=/api/ds/query
logger=context userId=1 orgId=1 uname=admin t=2024-07-22T18:30:00.428513892+08:00 level=error msg="Invalid dashboard UID in annotation request" error="context canceled" remote_addr=11.163.85.164 traceID=
logger=context userId=1 orgId=1 uname=admin t=2024-07-22T18:30:00.428650986+08:00 level=info msg="Request Completed" method=GET path=/api/annotations status=400 remote_addr=11.163.85.164 time_ms=160 duration=160.916194ms size=70 referer="http://11.163.85.164:30608/d/1552593593532704-17695-1-1/machine-center?editview=dashboard_json&orgId=1&refresh=5s" handler=/api/annotations
logger=datasources t=2024-07-22T18:30:00.470956035+08:00 level=error msg="Failed getting data source" err="context canceled" uid=PAF26937B51086004 id=0 name= orgId=1
logger=context userId=1 orgId=1 uname=admin t=2024-07-22T18:30:00.471135717+08:00 level=error msg="Client closed request" error="[api.requestCanceled] response: request canceled: context canceled" remote_addr=11.163.85.164 traceID=
logger=context userId=1 orgId=1 uname=admin t=2024-07-22T18:30:00.471227516+08:00 level=info msg="Request Completed" method=POST path=/api/ds/query status=499 remote_addr=11.163.85.164 time_ms=203 duration=203.286602ms size=94 referer="http://11.163.85.164:30608/d/1552593593532704-17695-1-1/machine-center?editview=dashboard_json&orgId=1&refresh=5s" handler=/api/ds/query
logger=oauthtoken t=2024-07-22T18:31:47.327147911+08:00 level=error msg="Failed to fetch oauth token for user" userId=1 username= error="context canceled"
logger=oauth_token.sync t=2024-07-22T18:31:47.327499366+08:00 level=error msg="Failed to fetch oauth entry" id=user:1 error="context canceled"
logger=authn.service t=2024-07-22T18:31:47.327631597+08:00 level=error msg="Failed to run post auth hook" client=auth.client.session id=user:1 error="[user.sync.fetch] failed to resolve user: context canceled"
logger=context userId=0 orgId=0 uname= t=2024-07-22T18:31:47.32786692+08:00 level=error msg= error="[user.sync.fetch] failed to resolve user: context canceled" remote_addr=11.163.85.164 traceID=
logger=context userId=0 orgId=0 uname= t=2024-07-22T18:31:47.328029438+08:00 level=error msg="Request Completed" method=GET path=/api/annotations status=500 remote_addr=11.163.85.164 time_ms=2 duration=2.721913ms size=133 referer="http://11.163.85.164:30608/d/1552593593532704-17695-1-1/machine-center?editview=dashboard_json&orgId=1&refresh=5s" handler=/api/annotations
logger=datasources t=2024-07-22T18:31:47.334768887+08:00 level=error msg="Failed getting data source" err="context canceled" uid=PAF26937B51086004 id=0 name= orgId=1
logger=context userId=1 orgId=1 uname=admin t=2024-07-22T18:31:47.335001521+08:00 level=error msg="Client closed request" error="[api.requestCanceled] response: request canceled: context canceled" remote_addr=11.163.85.164 traceID=
logger=context userId=1 orgId=1 uname=admin t=2024-07-22T18:31:47.335132888+08:00 level=info msg="Request Completed" method=POST path=/api/ds/query status=499 remote_addr=11.163.85.164 time_ms=9 duration=9.515949ms size=94 referer="http://11.163.85.164:30608/d/1552593593532704-17695-1-1/machine-center?editview=dashboard_json&orgId=1&refresh=5s" handler=/api/ds/query
logger=datasources t=2024-07-22T18:31:47.335910815+08:00 level=error msg="Failed getting data source" err="context canceled" uid=PAF26937B51086004 id=0 name= orgId=1
logger=context userId=1 orgId=1 uname=admin t=2024-07-22T18:31:47.336095726+08:00 level=error msg="Client closed request" error="[api.requestCanceled] response: request canceled: context canceled" remote_addr=11.163.85.164 traceID=
logger=context userId=1 orgId=1 uname=admin t=2024-07-22T18:31:47.336199272+08:00 level=info msg="Request Completed" method=POST path=/api/ds/query status=499 remote_addr=11.163.85.164 time_ms=8 duration=8.95018ms size=94 referer="http://11.163.85.164:30608/d/1552593593532704-17695-1-1/machine-center?editview=dashboard_json&orgId=1&refresh=5s" handler=/api/ds/query
logger=datasources t=2024-07-22T18:31:47.346779467+08:00 level=error msg="Failed getting data source" err="context canceled" uid=PAF26937B51086004 id=0 name= orgId=1
logger=datasources t=2024-07-22T18:31:47.346878956+08:00 level=error msg="Failed getting data source" err="context canceled" uid=PAF26937B51086004 id=0 name= orgId=1
henrysun007 commented 4 months ago

Addup: I deployed the kuscia monitor according to https://www.secretflow.org.cn/en/docs/kuscia/v0.10.0b0/deployment/kuscia_monitor. The promethus can be successfully explored but grafana is empty.

1299756312 commented 4 months ago

Have the data sources in your Grafana template file been replaced

henrysun007 commented 4 months ago

The prometheus type is

          "datasource": {
            "type": "prometheus",
            "uid": "PAF26937B51086004"
          }

The datasource type is

        "datasource": {
          "type": "datasource",
          "uid": "grafana"
        }

Is it right?

henrysun007 commented 4 months ago

The original file is machine_center-1721645255119.json.

aokaokd commented 3 months ago

Enter curl -s http://admin:admin@localhost:3000/api/datasources , check what is returned

henrysun007 commented 3 months ago 
[{"id":1,"uid":"PAF26937B51086004","orgId":1,"name":"Kuscia-monitor","type":"prometheus","typeName":"Prometheus","typeLogoUrl":"/public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://localhost:9090","user":"","database":"","basicAuth":false,"isDefault":false,"jsonData":{"cacheLevel":"High","disableRecordingRules":false,"httpMethod":"POST","incrementalQueryOverlapWindow":"10m","manageAlerts":true,"prometheusType":"Prometheus","prometheusVersion":"2.45.2"},"readOnly":true},{"id":2,"uid":"fd1664ab-33a4-423b-afcc-76addc85de80","orgId":1,"name":"prometheus","type":"prometheus","typeName":"Prometheus","typeLogoUrl":"/public/app/plugins/datasource/prometheus/img/prometheus_logo.svg","access":"proxy","url":"http://localhost:9090","user":"","database":"","basicAuth":false,"isDefault":false,"jsonData":{"httpMethod":"POST"},"readOnly":false}]
aokaokd commented 3 months ago

We would like to reproduce it, please provide your Grafana version

henrysun007 commented 3 months ago

我是通过 https://github.com/secretflow/kuscia/blob/e8666fe31e464eb425b136f3979afe232030e14c/build/dockerfile/kuscia-monitor.Dockerfile 制作的镜像

aokaokd commented 3 months ago

好的， 目前正在复现

aokaokd commented 3 months ago

参考官网的步骤，没有复现出来。 看日志是你的uid有问题，再检查下你的环境和版本 image

henrysun007 commented 3 months ago

能把你的容器镜像发我吗？

aokaokd commented 3 months ago

好的，registry.cn-hangzhou.aliyuncs.com/wushijie/kuscia-monitor

henrysun007 commented 3 months ago

dashboard界面中的instance我手动输入后，可以正常显示数据了，谢谢