search

mvisonneau / gitlab-ci-pipelines-exporter

Prometheus / OpenMetrics exporter for GitLab CI pipelines insights
Apache License 2.0
1.25k stars 237 forks source link

Seeing 500 errors in logs - #690

Open TJM opened 1 year ago

TJM commented 1 year ago

We are seeing a string of 500 errors in the logs that may be related to gcpe's thrashing of the gitlab server since we enabled job metrics. 

jsonPayload: {
  error: "GET https://gitlab.company.com/api/v4/projects/group/gke/jobs: 500 {error: Internal Server Error}, {status: failed to parse unexpected error type: float64}"
  level: "warning"
  msg: "pulling ref metrics"
  project-name: "group/gke"
  ref: "feat/master-upgrade"
}

Sometimes it gives more details...

jsonPayload: {
  error: "could not read content of pipeline group/group-program - ybondarchuk-master-patch-56877 | GET https://gitlab.company.com/api/v4/projects/group/group-program/pipelines/73895: 500 {error: Internal Server Error}, {status: failed to parse unexpected error type: float64}"
  level: "warning"
  msg: "pulling ref metrics"
  project-name: "group/group-program"
  ref: "ybondarchuk-master-patch-56877"
}

Either way, it seems to fall on: status: failed to parse unexpected error type: float64

Not sure if this is just "expected" when gitlab returns a 500 error, or if there is something else wrong? Note that the 500 errors (and some 502s) are related to #691 .. this specific issue is about the "failed to parse unexpected error type: float64"