Failed to fetch images: prometheus-config-reloader and grafana-agent

[sojjan1337]

Hello!

Problem fetching the grafana-agent and prometheus-config-reloader, when installing loki. I'm in isolated env. I have pointed other image's to our own registry and it works fine. But how can i point those images that won't work to our own registry? I can't find the endpoints in the chart's. The endpoints that loki-logs "pod" is trying to fetch is = "https://quay.io/v2/" + "https://registry-1.docker.io/v2/" but failing as you can see below in the logs.

The grafana-operator starts but when the loki-logs "Pod" is trying to start it cannot fetch. Below is a "Describe" of one of the loki-logs "pod" Error: ErrImagePull

Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  102s                default-scheduler  Successfully assigned loki/loki-logs-948r4 to test
  Warning  Failed     87s                 kubelet            Failed to pull image "quay.io/prometheus-operator/prometheus-config-reloader:v0.67.1": rpc error: code = Unknown desc = Error response from daemon: Get "https://quay.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Warning  Failed     56s                 kubelet            Failed to pull image "quay.io/prometheus-operator/prometheus-config-reloader:v0.67.1": rpc error: code = Unknown desc = Error response from daemon: Get "https://quay.io/v2/": dial tcp xx.xx.xx.xx:443: i/o timeout
  Warning  Failed     41s (x2 over 72s)   kubelet            Error: ErrImagePull
  Warning  Failed     41s                 kubelet            Error: ImagePullBackOff
  Warning  Failed     41s (x2 over 72s)   kubelet            Failed to pull image "grafana/agent:v0.38.1": rpc error: code = Unknown desc = Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Normal   BackOff    41s                 kubelet            Back-off pulling image "grafana/agent:v0.38.1"
  Warning  Failed     41s                 kubelet            Error: ImagePullBackOff
  Normal   BackOff    41s                 kubelet            Back-off pulling image "quay.io/prometheus-operator/prometheus-config-reloader:v0.67.1"                                                                                                   
  Normal   Pulling    28s (x3 over 102s)  kubelet            Pulling image "quay.io/prometheus-operator/prometheus-config-reloader:v0.67.1"                                                                                                            
  Normal   Pulling    13s (x3 over 87s)   kubelet            Pulling image "grafana/agent:v0.38.1"
  Warning  Failed     13s (x3 over 87s)   kubelet            Error: ErrImagePull
  Warning  Failed     13s                 kubelet            Failed to pull image "quay.io/prometheus-operator/prometheus-config-reloader:v0.67.1": rpc error: code = Unknown desc = Error response from daemon: Get "https://quay.io/v2/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Here below you can see that the grafana-agent-operator is running fine, it can fetch the img from our own registry. But the loki-logs "pod" having problem pulling the IMG as error-log above.

loki-canary-wp2jw                              1/1     Running             0          37s
loki-gateway-74d74f9984-l9m8j                  1/1     Running             0          37s
loki-grafana-agent-operator-54bd4b9674-59lns   1/1     Running             0          37s
loki-logs-2nsrb                                0/2     ErrImagePull        0          34s

The following block below is working for the grafana-agent-operator. I want to do the same thing for the agent and the config-reloader IMG, if it is the correct way of doing it. It overwrites and points to our own registry.

grafana-agent-operator:
  image:
    # -- Image registry
    registry: harbor.test.com
    # -- Image repo
    repository: docker_registry.hub.docker.com-r/grafana/agent-operator
    # -- Image tag
    tag: main

Chart=5.41.4

[nbmahsyar]

anyway I got the same issue with the loki-5.39.0 chart version

[JStickler]

Questions have a better chance of being answered if you ask them on the community forums.

[volker-raschek]

Hi, I opened a thread in the community forum.

[sicoyle]

any update?