image-renderer timeout for rendering full dashboard with multiple panels

[infa-mlagad]

Getting stack":"TimeoutError: Navigation timeout of 60000 ms exceeded\n at /usr/src/app/node_modules/puppeteer/lib/LifecycleWatcher.js:142:21\ while rendering full dashboard

What you expected to happen: Should have create image after rendering the full dashboard panels

How to reproduce it (as minimally and precisely as possible): try creating image of full dashboard with multiple panels (Kubernetes Cluster (Prometheus)).

Please see below deployment file apiVersion: extensions/v1beta1 kind: Deployment metadata: labels: app.kubernetes.io/instance: grafanatest app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: grafana app.kubernetes.io/version: 7.0.3 helm.sh/chart: grafana-5.3.0 name: grafanatest namespace: grafana

spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app.kubernetes.io/instance: grafanatest app.kubernetes.io/name: grafana strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata:

  labels:
    app.kubernetes.io/instance: grafanatest
    app.kubernetes.io/name: grafana
spec:
  containers:
  - env:
    - name: METHOD
    - name: LABEL
      value: grafana_dashboard
    - name: FOLDER
      value: /tmp/dashboards
    - name: RESOURCE
      value: both
    image: kiwigrid/k8s-sidecar:0.1.151
    imagePullPolicy: IfNotPresent
    name: grafana-sc-dashboard
    resources:
      limits:
        cpu: 200m
        memory: 256Mi
      requests:
        cpu: 100m
        memory: 128Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /tmp/dashboards
      name: sc-dashboard-volume
  - env:
    - name: GF_SECURITY_ADMIN_USER
      valueFrom:
        secretKeyRef:
          key: admin-user
          name: grafana
    - name: GF_SECURITY_ADMIN_PASSWORD
      valueFrom:
        secretKeyRef:
          key: admin-password
          name: grafana
    - name: GF_INSTALL_PLUGINS
      valueFrom:
        configMapKeyRef:
          key: plugins
          name: grafanatest
    - name: GF_RENDERING_CALLBACK_URL
      value: http://localhost:3000/
    - name: GF_RENDERING_SERVER_URL
      value: http://localhost:8081/render
    image: grafana/grafana:7.0.3
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 10
      httpGet:
        path: /api/health
        port: 3000
        scheme: HTTP
      initialDelaySeconds: 60
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 30
    name: grafana
    ports:
    - containerPort: 80
      name: service
      protocol: TCP
    - containerPort: 3000
      name: grafana
      protocol: TCP
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /api/health
        port: 3000
        scheme: HTTP
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits:
        cpu: 500m
        memory: 1Gi
      requests:
        cpu: 250m
        memory: 512Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /etc/grafana/grafana.ini
      name: config
      subPath: grafana.ini
    - mountPath: /var/lib/grafana
      name: storage
    - mountPath: /tmp/dashboards
      name: sc-dashboard-volume
    - mountPath: /etc/grafana/provisioning/dashboards/sc-dashboardproviders.yaml
      name: sc-dashboard-provider
      subPath: provider.yaml
    - mountPath: /etc/grafana/provisioning/datasources
      name: sc-datasources-volume
  - image: grafana/grafana-image-renderer:latest
    imagePullPolicy: Always
    name: renderer
    resources:
      limits:
        cpu: "1"
        memory: 2Gi
      requests:
        cpu: 500m
        memory: 1Gi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
  dnsPolicy: ClusterFirst
  initContainers:
  - command:
    - chown
    - -R
    - 472:472
    - /var/lib/grafana
    image: quay.io/prometheus/busybox
    imagePullPolicy: IfNotPresent
    name: init-chown-data
    resources:
      limits:
        cpu: 200m
        memory: 256Mi
      requests:
        cpu: 100m
        memory: 128Mi
    securityContext:
      runAsUser: 0
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/lib/grafana
      name: storage
  - env:
    - name: METHOD
      value: LIST
    - name: LABEL
      value: grafana_datasource
    - name: FOLDER
      value: /etc/grafana/provisioning/datasources
    - name: RESOURCE
      value: both
    image: kiwigrid/k8s-sidecar:0.1.151
    imagePullPolicy: IfNotPresent
    name: grafana-sc-datasources
    resources:
      limits:
        cpu: 200m
        memory: 256Mi
      requests:
        cpu: 100m
        memory: 128Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /etc/grafana/provisioning/datasources
      name: sc-datasources-volume
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 472
    runAsGroup: 472
    runAsUser: 472
  serviceAccount: grafana-account
  serviceAccountName: grafana-account
  terminationGracePeriodSeconds: 30
  volumes:
  - configMap:
      defaultMode: 420
      name: grafanatest
    name: config
  - name: storage
    persistentVolumeClaim:
      claimName: grafanatest
  - emptyDir: {}
    name: sc-dashboard-volume
  - configMap:
      defaultMode: 420
      name: grafanatest-config-dashboards
    name: sc-dashboard-provider
  - emptyDir: {}
    name: sc-datasources-volume

Anything else we need to know?: Mine is Kubernetes set up for grafana where image-renderer has been added as an extraContainer in it. I also tried giving more memory to renderer thinking it might have happened because if low resources. But still the issue is same. Limits: cpu: 1 memory: 2Gi Requests: cpu: 500m memory: 1Gi

Environment:

• Grafana Image Renderer version: grafana/grafana-image-renderer:latest
• Grafana version: grafana/grafana:7.0.3
• Installed plugin or remote renderer service: remote service, added as an extracontainer in grafana
• OS Grafana Image Renderer is installed on: Docker image grafana/grafana-image-renderer
• User OS & Browser: mac / firefox
• Others:

[infa-mlagad]

Team, can someone please check this once?

[jkupidura14]

Bump, having the same issue. Is there a way to set the timeout for this? I have a report that runs and will timeout since it is set to 10 seconds when renderering a PDF report.