deepflow grafana Dashboards Deepflow-Template/Application-开箱即用，查询请求 No Data

[LemonSqi]

Search before asking

• [X] I had searched in the issues and found no similar feature requirement.

Description

在使用deepflow grafana的Dashboards去测试Deepflow-Template/Application-开箱即用面板时遇到查询请求No Data的情况：

观察到后台所有服务均处于Running状态：

查看deepflow-server的日志看到如下报错：

请问这个是什么原因引起的？？

Use case

No response

Related issues

No response

Are you willing to submit a PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[1473371932]

Hello, may I ask which version of Deepflow you are using? Currently, is it only the 'Deepflow-Template/Application-Out-of-the-Box' dashboard that has no data, or do all the panels lack data?

[LemonSqi]

Hello, may I ask which version of Deepflow you are using? Currently, is it only the 'Deepflow-Template/Application-Out-of-the-Box' dashboard that has no data, or do all the panels lack data?

Deepflow 6.5.8 all the panels lack data

kubernetes.go:165 get az controller connection (node_ip:172.21.227.41) from db failed: record not found what does this error mean？？？

How can I access the Deepflow data source on the Pod of deepflowserver? If there are any instructions, please let me know. I can check the specific table information inside the Deepflow data source.

I try to query data source of Deepflow to use sql api, but value is null ...

curl -XPOST "http://172.21.227.41:32238/v1/query/" \
    --data-urlencode "db=flow_log" \
    --data-urlencode "sql=SELECT pod_ns_0, toString(_id), pod_ns_id_0 FROM l7_flow_log WHERE time>=1719381554 AND time<=1719385154 LIMIT 100"

{ "OPT_STATUS": "SUCCESS", "DESCRIPTION": "", "result": { "columns": ["pod_ns_0", "toString(_id)", "pod_ns_id_0"], "schemas": [{ "label_type": "", "pre_as": "", "type": 0, "unit": "", "value_type": "" }, { "label_type": "", "pre_as": "", "type": 0, "unit": "", "value_type": "" }, { "label_type": "", "pre_as": "", "type": 0, "unit": "", "value_type": "" }], "values": null }, "debug": null }

It seems that the deepflow data source itself is no data...

Looking forward to your answer！！！

[LemonSqi]

Today，another strange phenomenon occurred. The pods of the deepflow cluster deepflow-server began to be rebuilt, and many pods were in the Evicted state.

[1473371932]

Hello, when encountering an "Evicted" status in the pod, please investigate whether it is due to issues with the corresponding k8s node itself, such as disk insufficiency causing the node to become tainted.

Regarding the aforementioned data issue, could you first upgrade to our v6.5 LTS version? You can change the image tag of the deepflow-related components, such as server, agent, and app, from v6.5.8 to v6.5. After updating to the LTS version, check the Dashboard panel again to see if the data issue persists.

[LemonSqi]

Hello, when encountering an "Evicted" status in the pod, please investigate whether it is due to issues with the corresponding k8s node itself, such as disk insufficiency causing the node to become tainted.

Regarding the aforementioned data issue, could you first upgrade to our v6.5 LTS version? You can change the image tag of the deepflow-related components, such as server, agent, and app, from v6.5.8 to v6.5. After updating to the LTS version, check the Dashboard panel again to see if the data issue persists.

Sorry, I installed it through the intranet. The directly downloaded deepflow-charts warehouse package was installed and started locally through helm. I checked again and the version of Chart.yaml was 6.5.009. I noticed that the 6.5.9 version of the chart only has Source code and does not have a compiled tgz package. Can you provide a compiled package?

[LemonSqi]

I found the v6.5.9 tgz installation package on the chart's release tag page, but when I tried to install it through helm, I got an error: missing in charts/ directory: deepflow-agent, grafana. I tried to check the contents of this tgz and found that deepflow-agent and grafana were indeed not included, and complete dependencies were provided in previous versions of charts. What considerations did the 6.5.9 LTS version take to change the release structure of the package? We are expecting a reply!

[LemonSqi]

I tried to put the package deepflow-agent in deedflow/charts, and put grafana in the previous 6.5.009 package in the same location. All images used v6.5 of the latest tag. After starting, grafana Still getting error:

Now an error is reported directly. XX does not exist in the database. Maybe it is because the version is incompatible? I don’t know where the problem lies. Can you provide a complete version of the deepflow v6.5 chart package that has been tested internally for stability (LTS includes all components). We have invested a lot of time in the installation and use of components. A stable version of the chart Bags are really important to us, thank you so much for supporting us! ! !

[1473371932]

I found the v6.5.9 tgz installation package on the chart's release tag page, but when I tried to install it through helm, I got an error: missing in charts/ directory: deepflow-agent, grafana. I tried to check the contents of this tgz and found that deepflow-agent and grafana were indeed not included, and complete dependencies were provided in previous versions of charts. What considerations did the 6.5.9 LTS version take to change the release structure of the package? We are expecting a reply!

Hello, have you tried using helm repo update to refresh the helm repository? I noticed you mentioned the downloaded tar package is named v6.5.9 tgz, while our latest chart version is already 6.5.010. Alternatively, you can update by simply replacing the image tag, as the format in the chart package has not changed; only the image tag has been updated from v6.5.9 to v6.5.

If you are using an internal network and cannot directly connect to GitHub to download or update the latest chart package, you can first download the chart package and the corresponding image through an external network, then upload them to your intranet host for updating. Based on the error message in your current screenshot, it appears that ClickHouse is missing some tables. However, I still recommend updating using the aforementioned method and then checking if there are any data issues.

[LemonSqi]

I found the v6.5.9 tgz installation package on the chart's release tag page, but when I tried to install it through helm, I got an error: missing in charts/ directory: deepflow-agent, grafana. I tried to check the contents of this tgz and found that deepflow-agent and grafana were indeed not included, and complete dependencies were provided in previous versions of charts. What considerations did the 6.5.9 LTS version take to change the release structure of the package? We are expecting a reply!

Hello, have you tried using helm repo update to refresh the helm repository? I noticed you mentioned the downloaded tar package is named v6.5.9 tgz, while our latest chart version is already 6.5.010. Alternatively, you can update by simply replacing the image tag, as the format in the chart package has not changed; only the image tag has been updated from v6.5.9 to v6.5.

If you are using an internal network and cannot directly connect to GitHub to download or update the latest chart package, you can first download the chart package and the corresponding image through an external network, then upload them to your intranet host for updating. Based on the error message in your current screenshot, it appears that ClickHouse is missing some tables. However, I still recommend updating using the aforementioned method and then checking if there are any data issues.

I deployed in an offline environment, so I cannot update my chart package through the helm update command. I need to download the specified version of the chart package and image on a networked machine and import it into the offline environment. I have downloaded the 6.5.010 chart package and redeployed it as you said, but there is still a problem with the display of grafana, No data. This confuses us. Because it is a black box, we cannot determine the cause of the problem.

We query the Deepflow table through the SQL API and check whether the view is empty, but we find that this time, it is no longer empty, but has a value: curl -XPOST "http://172.21.227.41:32062/v1/query/" --data-urlencode "db=flow_log" --data-urlencode "sql=show tag pod_ns values from l7_flow_log"

{"OPT_STATUS":"SUCCESS","DESCRIPTION":"","result":{"columns":["value","display_name"],"schemas":[],"values":[[1,"kube-public"],[2,"kube-node-lease"],[3,"deepflow"],[4,"kube-system"],[5,"sqwang"],[6,"default"]]},"debug":null}

However, we cannot view these namespaces in grafana's explore panel, which makes us very confused. View deepflow-server log:

POST /v1/query displays 200, indicating that the query is OK, but traces export: Post "http://deepflow-agent/api/v1/otel/trace" failed connect refused

So, We cannot directly complete the test of the open source deepflow component through containerized deployment. We look forward to your further reply.

[1473371932]

I tried to put the package deepflow-agent in deedflow/charts, and put grafana in the previous 6.5.009 package in the same location. All images used v6.5 of the latest tag. After starting, grafana Still getting error:

Now an error is reported directly. XX does not exist in the database. Maybe it is because the version is incompatible? I don’t know where the problem lies. Can you provide a complete version of the deepflow v6.5 chart package that has been tested internally for stability (LTS includes all components). We have invested a lot of time in the installation and use of components. A stable version of the chart Bags are really important to us, thank you so much for supporting us! ! !

After the update, are there still any errors? Could you select "Data" at the screenshot's indicated position to check for any content, and also try querying the controller's status in MySQL with SELECT state FROM deepflow.controller; to rule out issues caused by the controller?

[LemonSqi]

I tried to put the package deepflow-agent in deedflow/charts, and put grafana in the previous 6.5.009 package in the same location. All images used v6.5 of the latest tag. After starting, grafana Still getting error: Now an error is reported directly. XX does not exist in the database. Maybe it is because the version is incompatible? I don’t know where the problem lies. Can you provide a complete version of the deepflow v6.5 chart package that has been tested internally for stability (LTS includes all components). We have invested a lot of time in the installation and use of components. A stable version of the chart Bags are really important to us, thank you so much for supporting us! ! !

After the update, are there still any errors? Could you select "Data" at the screenshot's indicated position to check for any content, and also try querying the controller's status in MySQL with SELECT state FROM deepflow.controller; to rule out issues caused by the controller?

I don’t understand the test method you mentioned above. When it comes to mysql you mentioned above, does this instruction mean that I access it directly through the url api, or does it mean that I log in to the deepflow-mysql service and select it? The other screenshot is the log of deepflow-server. How to get the data in the log? Do not understand what you mean

I checked through the sql api that there is data in the deepflow data source. At least I found all the namespace data of the ks cluster through tag=pod_ns, but what I saw in grafana was groups of no data.

[LemonSqi]

I try to understand your test method: I entered the pod of deepflow-mysql, and the result of the query through select state from deepflow.controller is 2.what does this status mean?

[LemonSqi]

通过进一步的跟踪定位，我们发现ck的日志里存在一些比较奇怪的报错： 跟着这个报错，我们进入了pod内部，定位到该路径下确实没有证书： 查看config.yaml, openSSL确实配置了证书路径：

请问，grafana查询不到数据是否和ck的这个异常有关，如果有关系，为什么成功启动的ck在该路径下没有生成可用的证书，期待你们的答复！！！

[liqian1834]

你好，对于 Grafana 中 No Data 的情况，整个 Issue 里面目前看到问题如下：

• auto create domain failed：record not found，try again after 3s 报错，这个是 6.5 版本新增多租户功能引入的一个 Bug，通过升级到 6.5 LTS 版本可解决
• deepflow-server pod 的状态不对，这个状态是否已经正常？
• cannot view these namespaces in grafana's explore panel 这里应该是 K8s 对接存在问题，然后引入了后续的一系列问题
• the result of the query through select state from deepflow.controller is 2 这里看 deepflow-server 的 controller 有问题，也需要继续看下问题

上面的问题，持续在 Issue 跟进可能效率并不高，可以加我们微信，语音下快速解决

[LemonSqi]

感谢支持~！！！问题已经及时被解决~！！