search

mvisonneau / gitlab-ci-pipelines-exporter

Prometheus / OpenMetrics exporter for GitLab CI pipelines insights
Apache License 2.0
1.27k stars 240 forks source link

MR pipelines sent through webhooks are rejected in `webhook` way. #889

Open ZhaoUncle opened 2 days ago

ZhaoUncle commented 2 days ago

hello @mvisonneau ,

issues:

I through webhook to push metrics into exporter,as the project use main branch to ci pipeline, it is ok from exporter logs. but if I use ci-test branch to test , it is failed. I can't get metrics about ci-test branch ci pipeline

version: gitlab-ci-pipelines-exporter:v0.5.8

logs:

{"gitlab-endpoint":"https://gitlab.test.com","gitlab-rate-limit":"1rps","level":"info","msg":"configured","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"pull projects from wildcards","on-init":"yes","scheduled":"no","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"pull environments from projects","on-init":"yes","scheduled":"no","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"pull refs from projects","on-init":"yes","scheduled":"no","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"pull metrics","on-init":"yes","scheduled":"no","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"garbage collect projects","on-init":"no","scheduled":"every 14400s","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"garbage collect environments","on-init":"no","scheduled":"every 14400s","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"garbage collect refs","on-init":"no","scheduled":"every 1800s","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"garbage collect metrics","on-init":"no","scheduled":"every 600s","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"redis url configured, initializing connection..","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"connected to redis","time":"2024-09-24T08:24:46Z"}
{"host":"127.0.0.1:8082","level":"info","msg":"internal monitoring listener set","path":"","scheme":"tcp","time":"2024-09-24T08:24:46Z"}
{"controller-uuid":"652728c2-3aa6-4f94-94bd-50e95ffdfebe","level":"info","listen-address":":8080","metrics-endpoint-enabled":true,"msg":"http server started","openmetrics-encoding-enabled":false,"pprof-endpoint-enabled":false,"time":"2024-09-24T08:24:46Z","webhook-endpoint-enabled":true}
{"level":"info","msg":"received a pipeline webhook from GitLab for a ref, triggering metrics pull","project-name":"test/devops/poc/ci-test","ref":"main","ref-kind":"branch","time":"2024-09-24T08:25:16Z"}
{"level":"info","msg":"received a pipeline webhook from GitLab for a ref, triggering metrics pull","project-name":"test/devops/poc/ci-test","ref":"main","ref-kind":"branch","time":"2024-09-24T08:25:17Z"}
{"level":"info","msg":"received a pipeline webhook from GitLab for a ref, triggering metrics pull","project-name":"test/devops/poc/ci-test","ref":"main","ref-kind":"branch","time":"2024-09-24T08:25:17Z"}
{"level":"info","msg":"ref not configured in the exporter, ignoring pipeline webhook","project-name":"test/devops/poc/ci-test","ref":"ci-test","ref-kind":"branch","time":"2024-09-24T08:25:20Z"}
{"level":"info","msg":"ref not configured in the exporter, ignoring pipeline webhook","project-name":"test/devops/poc/ci-test","ref":"ci-test","ref-kind":"branch","time":"2024-09-24T08:25:20Z"}
{"level":"info","msg":"ref not configured in the exporter, ignoring pipeline webhook","project-name":"test/devops/poc/ci-test","ref":"ci-test","ref-kind":"branch","time":"2024-09-24T08:25:20Z"}

config:

config.yml: |
    log:
      format: json
    server:
      webhook:
        enabled: true
    gitlab:
      url: https://gitlab.test.com
      enable_health_check: true
      enable_tls_verify: true
      maximum_jobs_queue_size: 5000
    pull: 
      environments_from_projects:
        on_init: true
        scheduled: false
      metrics:
        on_init: true
        scheduled: false
      projects_from_wildcards:
        on_init: true
        scheduled: false
      refs_from_projects:
        on_init: true
        scheduled: false
    project_defaults: 
      output_sparse_status_metrics: true
      pull:
        environments:
          enabled: false
          exclude_stopped: true
          regexp: .*
        pipeline:
          jobs:
            enabled: true
            from_child_pipelines:
              enabled: true
            runner_description:
              aggregation_regexp: shared-runners-manager-(\d*)\.gitlab\.com
              enabled: true
          test_reports:
            enabled: true
            test_cases:
              enabled: true
          variables:
            enabled: true
            regexp: .*
        refs:
          branches:
            enabled: true
            exclude_deleted: true
            max_age_seconds: 0
            most_recent: 0
            regexp: .*
          merge_requests:
            enabled: true
            max_age_seconds: 0
            most_recent: 0
          tags:
            enabled: true
            exclude_deleted: true
            max_age_seconds: 0
            most_recent: 0
            regexp: .*
    projects: 
      - name: test/devops/poc/ci-test
jose-sky commented 2 days ago

Same problem. I tried with all the options I can think of (project_defaults, wildcards, projects configuration). I cannot get the branch trunk to work.

project_defaults:
  pull:
    pipeline:
      jobs:
        enabled: true
    refs:
      branches:
        enabled: true
        regexp: .*
        most_recent: 500
        max_age_seconds: 0
      merge_requests:
        enabled: true
        most_recent: 500

I get errors for trunk and MRs and doesn't matter what I change I cannot make it work, I tried many combinations.

Sep 24 16:23:19 gitlab-ci-pipelines-exporter[236447]: time="2024-09-24T16:23:19Z" level=info msg="ref not configured in the exporter, ignoring pipeline webhook" project-name=group-name/apps/core-app ref=refs/merge-requests/4945/merge ref-kind=branch

Sep 24 16:24:17 gitlab-ci-pipelines-exporter[236447]: time="2024-09-24T16:24:17Z" level=info msg="ref not configured in the exporter, ignoring pipeline webhook" project-name=group-name/apps/core-app ref=trunk ref-kind=branch

StefanOid commented 1 day ago

hello @mvisonneau ,

issues:

I through webhook to push metrics into exporter,as the project use main branch to ci pipeline, it is ok from exporter logs. but if I use ci-test branch to test , it is failed. I can't get metrics about ci-test branch ci pipeline

version: gitlab-ci-pipelines-exporter:v0.5.8

logs:

{"gitlab-endpoint":"https://gitlab.test.com","gitlab-rate-limit":"1rps","level":"info","msg":"configured","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"pull projects from wildcards","on-init":"yes","scheduled":"no","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"pull environments from projects","on-init":"yes","scheduled":"no","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"pull refs from projects","on-init":"yes","scheduled":"no","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"pull metrics","on-init":"yes","scheduled":"no","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"garbage collect projects","on-init":"no","scheduled":"every 14400s","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"garbage collect environments","on-init":"no","scheduled":"every 14400s","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"garbage collect refs","on-init":"no","scheduled":"every 1800s","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"garbage collect metrics","on-init":"no","scheduled":"every 600s","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"redis url configured, initializing connection..","time":"2024-09-24T08:24:46Z"}
{"level":"info","msg":"connected to redis","time":"2024-09-24T08:24:46Z"}
{"host":"127.0.0.1:8082","level":"info","msg":"internal monitoring listener set","path":"","scheme":"tcp","time":"2024-09-24T08:24:46Z"}
{"controller-uuid":"652728c2-3aa6-4f94-94bd-50e95ffdfebe","level":"info","listen-address":":8080","metrics-endpoint-enabled":true,"msg":"http server started","openmetrics-encoding-enabled":false,"pprof-endpoint-enabled":false,"time":"2024-09-24T08:24:46Z","webhook-endpoint-enabled":true}
{"level":"info","msg":"received a pipeline webhook from GitLab for a ref, triggering metrics pull","project-name":"test/devops/poc/ci-test","ref":"main","ref-kind":"branch","time":"2024-09-24T08:25:16Z"}
{"level":"info","msg":"received a pipeline webhook from GitLab for a ref, triggering metrics pull","project-name":"test/devops/poc/ci-test","ref":"main","ref-kind":"branch","time":"2024-09-24T08:25:17Z"}
{"level":"info","msg":"received a pipeline webhook from GitLab for a ref, triggering metrics pull","project-name":"test/devops/poc/ci-test","ref":"main","ref-kind":"branch","time":"2024-09-24T08:25:17Z"}
{"level":"info","msg":"ref not configured in the exporter, ignoring pipeline webhook","project-name":"test/devops/poc/ci-test","ref":"ci-test","ref-kind":"branch","time":"2024-09-24T08:25:20Z"}
{"level":"info","msg":"ref not configured in the exporter, ignoring pipeline webhook","project-name":"test/devops/poc/ci-test","ref":"ci-test","ref-kind":"branch","time":"2024-09-24T08:25:20Z"}
{"level":"info","msg":"ref not configured in the exporter, ignoring pipeline webhook","project-name":"test/devops/poc/ci-test","ref":"ci-test","ref-kind":"branch","time":"2024-09-24T08:25:20Z"}

config:

config.yml: |
    log:
      format: json
    server:
      webhook:
        enabled: true
    gitlab:
      url: https://gitlab.test.com
      enable_health_check: true
      enable_tls_verify: true
      maximum_jobs_queue_size: 5000
    pull: 
      environments_from_projects:
        on_init: true
        scheduled: false
      metrics:
        on_init: true
        scheduled: false
      projects_from_wildcards:
        on_init: true
        scheduled: false
      refs_from_projects:
        on_init: true
        scheduled: false
    project_defaults: 
      output_sparse_status_metrics: true
      pull:
        environments:
          enabled: false
          exclude_stopped: true
          regexp: .*
        pipeline:
          jobs:
            enabled: true
            from_child_pipelines:
              enabled: true
            runner_description:
              aggregation_regexp: shared-runners-manager-(\d*)\.gitlab\.com
              enabled: true
          test_reports:
            enabled: true
            test_cases:
              enabled: true
          variables:
            enabled: true
            regexp: .*
        refs:
          branches:
            enabled: true
            exclude_deleted: true
            max_age_seconds: 0
            most_recent: 0
            regexp: .*
          merge_requests:
            enabled: true
            max_age_seconds: 0
            most_recent: 0
          tags:
            enabled: true
            exclude_deleted: true
            max_age_seconds: 0
            most_recent: 0
            regexp: .*
    projects: 
      - name: test/devops/poc/ci-test

Looks like your regex is not working... Have you tried looking at that?? Maybe quote the regex expression?? Apologies if it's dumb 😆

ZhaoUncle commented 1 day ago

@StefanOid sure, regexp: .*,you can see in projects_defaults settting. And I test in projects add this part regex rule regexp: ci-test, ci-test is branch name, but it is not ok.

 refs:
          branches:
            enabled: true
            exclude_deleted: true
            max_age_seconds: 0
            most_recent: 0
            regexp: .*
StefanOid commented 1 day ago

@StefanOid sure, regexp: .*,you can see in projects_defaults settting. And I test in projects add this part regex rule regexp: ci-test, ci-test is branch name, but it is not ok.

 refs:
          branches:
            enabled: true
            exclude_deleted: true
            max_age_seconds: 0
            most_recent: 0
            regexp: .*

ci-test is not a valid regex expression. ^ci-test$ would match what you have. Also, what i meant was, try "dotstar" instead of just .* Or "^ci-test$"

Basically .* in "" (github is removing star symbol for some reason)

ZhaoUncle commented 1 day ago

@StefanOid yes, I already add it into regex, but it is failed, same problem

^(?:ci-test)$
StefanOid commented 1 day ago

@StefanOid yes, I already add it into regex, but it is failed, same problem

^(?:ci-test)$

can you show me the working configuration you had with main?

ZhaoUncle commented 1 day ago

@StefanOid yes, I already add it into regex, but it is failed, same problem

^(?:ci-test)$

can you show me the working configuration you had with main?

don't use main, only ci-test.

StefanOid commented 1 day ago

You said it is working for main branch. Is it the same config?

Kind Regards, Stefan Birca

Tel: 07761916718

On Wed, 25 Sep 2024 at 12:52, ZhaoUncle @.***> wrote:

@StefanOid https://github.com/StefanOid yes, I already add it into regex, but it is failed, same problem

^(?:ci-test)$

can you show me the working configuration you had with main?

don't use main, only ci-test.

— Reply to this email directly, view it on GitHub https://github.com/mvisonneau/gitlab-ci-pipelines-exporter/issues/889#issuecomment-2373867231, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOIBF6KDKAMLKA6GVK5Y5DZYKPZLAVCNFSM6AAAAABOXYZ3C2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNZTHA3DOMRTGE . You are receiving this because you were mentioned.Message ID: @.***>

ZhaoUncle commented 1 day ago

You said it is working for main branch. Is it the same config?

yes, only this config, no another

ZhaoUncle commented 17 hours ago

@StefanOid 

config.yml: |
    log:
      level: info
      format: json
    server:
      webhook:
        enabled: true
    gitlab:
      url: https://gitlab.test.com
      enable_health_check: true
      enable_tls_verify: true
      maximum_jobs_queue_size: 5000
    pull: 
      environments_from_projects:
        on_init: false
        scheduled: false
      metrics:
        on_init: false
        scheduled: false
      projects_from_wildcards:
        on_init: false
        scheduled: false
      refs_from_projects:
        on_init: false
        scheduled: false
    project_defaults: 
      output_sparse_status_metrics: true
      pull:
        environments:
          enabled: false
          exclude_stopped: true
          regexp: .*
        pipeline:
          jobs:
            enabled: true
            from_child_pipelines:
              enabled: true
            runner_description:
              aggregation_regexp: shared-runners-manager-(\d*)\.gitlab\.com
              enabled: true
          test_reports:
            enabled: true
            test_cases:
              enabled: true
          variables:
            enabled: true
            regexp: .*
        refs:
          branches:
            enabled: true
            exclude_deleted: true
            max_age_seconds: 0
            most_recent: 0
            regexp: .*
          merge_requests:
            enabled: true
            max_age_seconds: 0
            most_recent: 0
          tags:
            enabled: true
            exclude_deleted: true
            max_age_seconds: 0
            most_recent: 0
            regexp: .*
    projects: 
      - name: test/devops/poc/ci-test
        pull:
          environments:
            enabled: false
            exclude_stopped: true
            regexp: .*
          pipeline:
            jobs:
              enabled: true
              from_child_pipelines:
                enabled: true
              runner_description:
                aggregation_regexp: shared-runners-manager-(\d*)\.gitlab\.com
                enabled: true
            test_reports:
              enabled: true
              test_cases:
                enabled: true
            variables:
              enabled: true
              regexp: .*
          refs:
            branches:
              enabled: true
              exclude_deleted: true
              max_age_seconds: 0
              most_recent: 0
              regexp: ^(?:ci-test)$
            merge_requests:
              enabled: true
              max_age_seconds: 0
              most_recent: 0
            tags:
              enabled: true
              exclude_deleted: true
              max_age_seconds: 0
              most_recent: 0
              regexp: .*

use this config, same problem, main branch is ok, ci-test branch is failed, same err log