Grafana timeouts retrieving the grafana-piechart-panel plugin

[niels-s]

Description

When deploying the prometheus-operator on a new Lokomotive cluster, the grafana pod is stuck in crashloop. It won't start because it's unable to retrieve the grafana-piechart-panel from Grafana

Error: ✗ Failed to send request: Get "https://grafana.com/api/plugins/repo/grafana-piechart-panel": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Impact

Deploying of the component never finishes and lokoctl get's stuck. And we can't use Grafana since it doesn't come up.

Environment and steps to reproduce

1. Set-up:

   
   cluster "packet" {
   asset_dir = "./assets"
   
   auth_token   = var.packet_token
   project_id   = var.packet_project_id
   facility     = "ny5"
   
   cluster_name     = "lokomotive-kv-benchmarks"
   
   controller_type  = "c3.small.x86"
   controller_count = 1
   
   dns {
   zone     = "onesignal.io"
   provider = "cloudflare"
   }
   
   ssh_pubkeys = [ "xxx" ]
   management_cidrs  = ["0.0.0.0/0"]
   node_private_cidr = "10.0.0.0/8"
   
   worker_pool "default" {
   count = 3
   node_type = "c3.small.x86"
   disable_bgp = true
   }
   }
   component "prometheus-operator" {
   prometheus {
   # All service monitors and prometheus rules in the cluster are watched by this prometheus.
   watch_labeled_service_monitors = "false"
   watch_labeled_prometheus_rules = "false"
   }
   
   storage_class = "rook-ceph-block"
   }

component "rook" { enable_monitoring = true }

component "rook-ceph" { monitor_count = 3

enable_toolbox = true

storage_class { enable = true } }

2. **Task**: 

3. **Action(s)**: 
- lokoctl cluster apply -v --skip-components
- lokoctl component apply rook
- lokoctl component apply rook-ceph
- Wait for rook pods to be running before continuing
- lokoctl component apply prometheus-operator
4. **Error**: 

kubectl -n monitoring logs prometheus-operator-grafana-c845fd4cc-vbdgg grafana Error: ✗ Failed to send request: Get "https://grafana.com/api/plugins/repo/grafana-piechart-panel": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

**Expected behavior**
I expect the Grafana pod to come up and have the grafana-piechart-panel installed

**Additional information**
Deploying to Equinix Metal with Lokoctl 0.5.0

[niels-s]

CoreDNS pods aren't up so there is no DNS resolution available on the cluster, this isn't an issue with Grafana.