search

IBM / FfDL

Fabric for Deep Learning (FfDL, pronounced fiddle) is a Deep Learning Platform offering TensorFlow, Caffe, PyTorch etc. as a Service on Kubernetes
https://developer.ibm.com/code/patterns/deploy-and-use-a-multi-framework-deep-learning-platform-on-kubernetes/
Apache License 2.0
690 stars 185 forks source link

Grafana charts shows no data points #151

Open Fly-Luck opened 5 years ago

Fly-Luck commented 5 years ago

Hi, I've installed FfDL in a completely offline kubernetes cluster:

  1. Imported all the necessary docker images to each cluster node.
  2. Inited tiller with specified image so it won't pull from the Internet.
  3. Installed FfDL using helm.
  4. Trained the example model according to your instructions.

Everything worked fine, and I've got the training results, but Grafana showed nothing but mostly a 'no data points' hint on its panel. Four dashboards: 1 2 3 4

And I can't find any useful Prometheus or Grafana logs.

BTW, I've commented out the env variable 'GF_INSTALL_PLUGINS' in spec of container 'grafana' in templates/monitoring/prometheus-deployment.yml, for it would try to download from the Internet.

Any hint on what's missing? Thanks!

animeshsingh commented 5 years ago

@Fly-Luck Grafana usage has been deprecated for us, and we aren`t testing it. We should remove it from the docs as well. Any valuable input you think you want to get out of Grafana here?