Option to aggregate PR-builds from same place

[macetw]

What feature do you want to see added?

With projects that are large, with many PRs, metrics are potentially too large, giving too much detail between different PRs, where differences between PRs is not helpful for metrics given the extra storage and bandwidth required.

As an SRE managing metrics and dashboards for Jenkins builds, I would like to simplify and simplify the aggregation across PRs of the same build, so that I don't have to store and transmit so much data and Prometheus doesn't have to work so hard to aggregate it at query time.

For example, if I have a build with 2 PRs, and it shows statistics like this:

     default_jenkins_builds_duration_milliseconds_summary_count{jenkins_job="mybuild/PR-481"} 1.0
     default_jenkins_builds_duration_milliseconds_summary_sum{jenkins_job="mybuild/PR-481"} 2000.0
     default_jenkins_builds_duration_milliseconds_summary_count{jenkins_job="mybuild/PR-81"} 3.0
     default_jenkins_builds_duration_milliseconds_summary_sum{jenkins_job="mybuild/PR-81"} 4000.0

with this option enabled, it would aggregate like so:

     default_jenkins_builds_duration_milliseconds_summary_count{jenkins_job="mybuild"} 4.0
     default_jenkins_builds_duration_milliseconds_summary_sum{jenkins_job="mybuild"} 6000.0

(and for all other per-build metrics)

We will want to continue to differentiate on the other labels. "buildable" and "repo"

Upstream changes

No response

[macetw]

Appears highly related to: https://github.com/jenkinsci/prometheus-plugin/issues/343 (I'm not the only one who suggested this)

[Waschndolos]

Hi, thank you for the issue. I'll have to investigate how this is possible. Sounds like a valid use case. So should this only be applied to PullRequests? Or should it be more like a general grouping by.. I don't know regex or something?

[macetw]

Maybe it could regex on PR builds, but it might be more simple to generalize it. We might want metrics for all folders of builds, to show aggregation at the directory level. The use case is slightly different, but the logic might be more simple to implement. Like for builds:

     foo/bar/PR-1 job
     foo/bar/PR-2 job
     foo/foobar job

We could want to:

• see additional metrics for /foo/bar and /foo and / (this is in addition to what we already see)
• see those additional metrics only and NOT the individual jobs (to reduce wasted space and TSDB compute)

The directory statistics could be very powerful for groups of teams. Our specific use case is for multibranch pipeline jobs, but this might also work to generalize outside that specific use case.

[Waschndolos]

I see. I cannot promise that I'll be able to implement it fast but I'll see what I can do. Think it could be useful :)

[github-actions[bot]]

Stale issue message

[Waschndolos]

"activity"

[seguins]

I have the same issue when I'm using this plugin with the workflow-multibranch plugin. A quick win could be to detect builds attached to a multibranch workflow and aggregate them together.

@Waschndolos, I have some free time to work on it if you want.

[Waschndolos]

@seguins. Hey thank you for reaching out. My plan was to implement this feature when I'm done testing my refactoring in the refactoring branch (https://github.com/jenkinsci/prometheus-plugin/tree/refactoring). But I guess I'm able to merge it when you've already have an idea to implement this in the current code. Otherwise I'll try to finish testing this week on my companies Jenkins instance and after that feel free to help :)

[github-actions[bot]]

Stale issue message

[Waschndolos]

Quick update. It's not forgotten :) I'm working on it. Didn't have a good idea yet but I might be making progress right now.

[github-actions[bot]]

Stale issue message

[github-actions[bot]]

Stale issue message

[Waschndolos]

My idea didn't work. Don't have a good idea on how to implement it. Does anybody have one?

[drewandersen]

@Waschndolos what was your idea? I might be able to take on this work.

[Waschndolos]

@drewandersen. Don't have a specific one but probably it's best to let the plugin calculate all individual metrics and then before writing it out to group the relevant metrics somehow. Maybe it's possible to write a custom CollectorRegistry which can do that?

[github-actions[bot]]

Stale issue message