search

thoth-station / thoth-application

Thoth-Station ArgoCD Applications
GNU General Public License v3.0
12 stars 22 forks source link

[3pt] PrometheusApiClientException: HTTP Status Code 403 #2618

Closed sentry-io[bot] closed 2 years ago

sentry-io[bot] commented 2 years ago

Sentry Issue: THOTH-1SVE

PrometheusApiClientException: HTTP Status Code 403 (b'\n<!DOCTYPE html>\n<html lang="en" charset="utf-8">\n<head>\n  <title>Log In</title>\n  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">\n  <style>\n    @font-face {\n      font-family: "Open Sans";\n      src: url(data:application/x-font-woff;charset=utf-8;base64,d09GRgABAAAAAFeoABMAAAAAlkQAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAABGRlRNAAABqAAAABwAAAAcavCZq0dERUYAAAHEAAAAHQAAAB4AJwD1R1BPUwAAAeQAAASjAAAJni1yF0JHU1VCAAAGiAAAAIEAAACooGKIn...
  File "wsgi.py", line 87, in func_wrapper
    job()
  File "decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
  File "/opt/app-root/src/thoth/metrics_exporter/jobs/base.py", line 42, in register_metric_job
    method(*args, **kwargs)
  File "/opt/app-root/src/thoth/metrics_exporter/jobs/db.py", line 94, in check_is_schema_up2date_for_components
    metrics_retrieved = Configuration.PROM.custom_query(query=query)
  File "prometheus_api_client/prometheus_connect.py", line 340, in custom_query
    raise PrometheusApiClientException(

Failed to run metrics gathering job DBMetrics.check_is_schema_up2date_for_components

Cluster: Prod

Acceptance criteria:

goern commented 2 years ago

/sig devsecops /kind bug /priority critical-urgent /triage accepted /assign @harshad16 /assign @Gregory-Pereira

goern commented 2 years ago

this is Prometheus being down? a flake? shall we handle this exception and just incr an error counter metrics?

Gregory-Pereira commented 2 years ago

Taking a look. /cc @Gregory-Pereira

goern commented 2 years ago

any update on this?

harshad16 commented 2 years ago

With the updated the Prometheus reader token https://github.com/thoth-station/thoth-application/commit/3e5229b215cd979442e65e9ca03aec8d0998a0b2 this issue is now fixed. closing the issue.