search

mvisonneau / gitlab-ci-pipelines-exporter

Prometheus / OpenMetrics exporter for GitLab CI pipelines insights
Apache License 2.0
1.29k stars 242 forks source link

"context canceled" error only with redis #620

Closed LGhoull closed 1 year ago

LGhoull commented 1 year ago

Hey, every time I trigger a pipeline with the webhook setup, i get a "context canceled" error. I'm not using ngrok since this is a self-hosted gitlab instance.

docker compose logs -f returns:

webhooks-redis-1                         | 1:C 08 Mar 2023 12:55:54.308 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
webhooks-redis-1                         | 1:C 08 Mar 2023 12:55:54.313 # Redis version=7.0.7, bits=64, commit=00000000, modified=0, pid=1, just started
webhooks-redis-1                         | 1:C 08 Mar 2023 12:55:54.313 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
webhooks-redis-1                         | 1:M 08 Mar 2023 12:55:54.314 * monotonic clock: POSIX clock_gettime
webhooks-redis-1                         | 1:M 08 Mar 2023 12:55:54.315 * Running mode=standalone, port=6379.
webhooks-redis-1                         | 1:M 08 Mar 2023 12:55:54.315 # Server initialized
webhooks-redis-1                         | 1:M 08 Mar 2023 12:55:54.315 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
webhooks-redis-1                         | 1:M 08 Mar 2023 12:55:54.316 * Loading RDB produced by version 7.0.7
webhooks-redis-1                         | 1:M 08 Mar 2023 12:55:54.316 * RDB age 5 seconds
webhooks-redis-1                         | 1:M 08 Mar 2023 12:55:54.316 * RDB memory usage when created 1.34 Mb
webhooks-redis-1                         | 1:M 08 Mar 2023 12:55:54.318 * Done loading RDB, keys loaded: 3, keys expired: 1.
webhooks-redis-1                         | 1:M 08 Mar 2023 12:55:54.318 * DB loaded from disk: 0.002 seconds
webhooks-redis-1                         | 1:M 08 Mar 2023 12:55:54.318 * Ready to accept connections
webhooks-gitlab-ci-pipelines-exporter-1  | time="2023-03-08T12:55:54Z" level=info msg=configured gitlab-endpoint="https://[redacted]" gitlab-rate-limit=1rps
webhooks-gitlab-ci-pipelines-exporter-1  | time="2023-03-08T12:55:54Z" level=info msg="pull projects from wildcards" on-init=no scheduled=no
webhooks-gitlab-ci-pipelines-exporter-1  | time="2023-03-08T12:55:54Z" level=info msg="pull environments from projects" on-init=no scheduled=no
webhooks-gitlab-ci-pipelines-exporter-1  | time="2023-03-08T12:55:54Z" level=info msg="pull refs from projects" on-init=no scheduled=no
webhooks-gitlab-ci-pipelines-exporter-1  | time="2023-03-08T12:55:54Z" level=info msg="pull metrics" on-init=no scheduled=no
webhooks-gitlab-ci-pipelines-exporter-1  | time="2023-03-08T12:55:54Z" level=info msg="garbage collect projects" on-init=no scheduled="every 14400s"
webhooks-gitlab-ci-pipelines-exporter-1  | time="2023-03-08T12:55:54Z" level=info msg="garbage collect environments" on-init=no scheduled="every 14400s"
webhooks-gitlab-ci-pipelines-exporter-1  | time="2023-03-08T12:55:54Z" level=info msg="garbage collect refs" on-init=no scheduled="every 1800s"
webhooks-gitlab-ci-pipelines-exporter-1  | time="2023-03-08T12:55:54Z" level=info msg="garbage collect metrics" on-init=no scheduled="every 600s"
webhooks-gitlab-ci-pipelines-exporter-1  | time="2023-03-08T12:55:54Z" level=debug msg="opentelemetry.grpc_endpoint is not configured, skipping open telemetry support"
webhooks-gitlab-ci-pipelines-exporter-1  | time="2023-03-08T12:55:54Z" level=info msg="redis url configured, initializing connection.."
webhooks-gitlab-ci-pipelines-exporter-1  | time="2023-03-08T12:55:54Z" level=info msg="connected to redis"
webhooks-gitlab-ci-pipelines-exporter-1  | time="2023-03-08T12:55:54Z" level=debug msg="task scheduled" interval_seconds=14400 task=GarbageCollectProjects
webhooks-gitlab-ci-pipelines-exporter-1  | time="2023-03-08T12:55:54Z" level=debug msg="task scheduled" interval_seconds=14400 task=GarbageCollectEnvironments
webhooks-gitlab-ci-pipelines-exporter-1  | time="2023-03-08T12:55:54Z" level=debug msg="task scheduled" interval_seconds=1800 task=GarbageCollectRefs
webhooks-gitlab-ci-pipelines-exporter-1  | time="2023-03-08T12:55:54Z" level=debug msg="task scheduled" interval_seconds=600 task=GarbageCollectMetrics
webhooks-gitlab-ci-pipelines-exporter-1  | time="2023-03-08T12:55:54Z" level=info msg="internal monitoring listener set" host="[redacted]:8082" path= scheme=tcp
webhooks-gitlab-ci-pipelines-exporter-1  | time="2023-03-08T12:55:54Z" level=info msg="http server started" controller-uuid=59910a23-e1f1-4bcf-931f-c4e5f860d678 listen-address=":8080" metrics-endpoint-enabled=true openmetrics-encoding-enabled=false pprof-endpoint-enabled=false webhook-endpoint-enabled=true
webhooks-gitlab-ci-pipelines-exporter-1  | time="2023-03-08T12:56:20Z" level=debug msg="webhook request" ip-address="[redacted]" user-agent=GitLab/15.8.3
webhooks-gitlab-ci-pipelines-exporter-1  | time="2023-03-08T12:56:20Z" level=error msg="reading ref from the store" error="context canceled" project-name=test/test-project ref=main ref-kind=branch
webhooks-gitlab-ci-pipelines-exporter-1  | time="2023-03-08T12:56:28Z" level=debug msg="webhook request" ip-address="[redacted]" user-agent=GitLab/15.8.3
webhooks-gitlab-ci-pipelines-exporter-1  | time="2023-03-08T12:56:28Z" level=error msg="reading ref from the store" error="context canceled" project-name=test/test-project ref=main ref-kind=branch

docker-compose.yml:

---
version: '3.8'
services:
  redis:
    image: redis:7.0.7-bullseye@sha256:8184cfe57f205ab34c62bd0e9552dffeb885d2a7f82ce4295c0df344cb6f0007
    ports:
      - 6379:6379
    environment:
      ALLOW_EMPTY_PASSWORD: 'yes'
    volumes:
      - redis_data:/data

  gitlab-ci-pipelines-exporter:
    image: quay.io/mvisonneau/gitlab-ci-pipelines-exporter:v0.5.4
    # You can comment out the image name and use the following statement
    # to build the image against the current version of the repository
    # build: ../..
    ports:
      - 8080:8080
    links:
      - redis
    environment:
      GCPE_CONFIG: /etc/gitlab-ci-pipelines-exporter.yml
      GCPE_INTERNAL_MONITORING_LISTENER_ADDRESS: tcp://127.0.0.1:8082
    volumes:
      - type: bind
        source: ./gitlab-ci-pipelines-exporter.yml
        target: /etc/gitlab-ci-pipelines-exporter.yml

networks:
  default:
    driver: bridge

volumes:
  redis_data:
    driver: local

gitlab-ci-pipelines-exporter.yml:

---
log:
  level: debug

gitlab:
  url: https://[redacted]
  token: [redacted]

redis:
  url: redis://redis:6379

server:
  webhook:
    enabled: true
    secret_token: [redacted]

pull:
  projects_from_wildcards:
    on_init: false
    scheduled: false

  environments_from_projects:
    on_init: false
    scheduled: false

  refs_from_projects:
    on_init: false
    scheduled: false

  metrics:
    on_init: false
    scheduled: false

projects:
  # Configure a project on which you are authorized to configure webhooks
  - name: test/test-project
    pull:
      refs:
        branches:
          regexp: ".*"

exported metrics:

# HELP gcpe_currently_queued_tasks_count Number of tasks in the queue
# TYPE gcpe_currently_queued_tasks_count gauge
gcpe_currently_queued_tasks_count 0
# HELP gcpe_environments_count Number of GitLab environments being exported
# TYPE gcpe_environments_count gauge
gcpe_environments_count 0
# HELP gcpe_executed_tasks_count Number of tasks executed
# TYPE gcpe_executed_tasks_count gauge
gcpe_executed_tasks_count 15
# HELP gcpe_gitlab_api_requests_count GitLab API requests count
# TYPE gcpe_gitlab_api_requests_count gauge
gcpe_gitlab_api_requests_count 0
# HELP gcpe_gitlab_api_requests_limit GitLab API requests available in the api limit
# TYPE gcpe_gitlab_api_requests_limit gauge
gcpe_gitlab_api_requests_limit 0
# HELP gcpe_gitlab_api_requests_remaining GitLab API requests remaining in the api limit
# TYPE gcpe_gitlab_api_requests_remaining gauge
gcpe_gitlab_api_requests_remaining 0
# HELP gcpe_metrics_count Number of GitLab pipelines metrics being exported
# TYPE gcpe_metrics_count gauge
gcpe_metrics_count 0
# HELP gcpe_projects_count Number of GitLab projects being exported
# TYPE gcpe_projects_count gauge
gcpe_projects_count 1
# HELP gcpe_refs_count Number of GitLab refs being exported
# TYPE gcpe_refs_count gauge
gcpe_refs_count 0
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0

Everything still worked when I wasn't using redis. At that time, the docker-compose.yml was the same as this one, with the "redis" service removed and without "links: redis" in the gitlab-ci-pipelines-exporter service. The gitlab-ci-pipelines-exporter.yml was also the same, just with "redis: url: redis://redis:6379" removed.

tomsozolins commented 1 year ago

Same error happening when using webhooks with redis enabled.

coibib commented 1 year ago

Same here

karpanin commented 1 year ago

same error when using only webhooks with push model and redis

jrblanno commented 1 year ago

same error ever here :)