Add prometheus metrics collecting

[DiegoTavares]

Adds a new optional module that runs every 60 seconds to collect metrics about the state of thread pools. Also, it collects basic time measurements and counts from dispatchDaoJdbc, which helps assess performance degradation on the system.

Metrics are exposed under /metrics for being collected by a Prometheus instance (the Prometheus server setup is not part of this PR).

A future PR is coming that will include Grafana charts to consume the exposed metrics.

Exposed metrics:

BookingQueue

• cue_booking_waiting_total: Booking Queue number of waiting tasks
• cue_booking_remaining_capacity_total: Booking Queue remaining capacity
• cue_booking_threads_total: Booking Queue number of active threads
• cue_booking_executed_total: Booking Queue number of executed tasks
• cue_booking_rejected_total: Booking Queue number of rejected tasks

Manage Queue

• cue_manage_waiting_total: Manage Queue number of waiting tasks
• cue_manage_remaining_capacity_total: Manage Queue remaining capacity
• cue_manage_threads_total: Manage Queue number of active threads
• cue_manage_executed_total: Manage Queue number of executed tasks
• cue_manage_rejected_total: Manage Queue number of rejected tasks

DispatchQueue

• cue_dispatch_waiting_total: Dispatch Queue number of waiting tasks
• cue_dispatch_remaining_capacity_total: Dispatch Queue remaining capacity
• cue_dispatch_threads_total: Dispatch Queue number of active threads
• cue_dispatch_executed_total: Dispatch Queue number of executed tasks
• cue_dispatch_rejected_total: Dispatch Queue number of rejected tasks

HostReportQueue

• cue_report_waiting_total: Report Queue number of waiting tasks
• cue_report_remaining_capacity_total: Report Queue remaining capacity
• cue_report_threads_total: Report Queue number of active threads
• cue_report_executed_total: Report Queue number of executed tasks
• cue_report_rejected_total: Report Queue number of rejected tasks

General Performance Metrics:

• cue_find_jobs_by_show_count: Count the occurrences of the query FIND_JOBS_BY_SHOW.
• cue_booking_durations_in_millis: Register the duration of booking steps in milliseconds.
• cue_booking_durations_histogram_in_millis: Register a summary of the duration of booking steps in milliseconds.
• cue_frame_killed_counter: Number of frames kill requests processed
• cue_frame_kill_failure_counter: Number of frames that failed to be killed after FRAME_KILL_RETRY_LIMIT tries

[DiegoTavares]

(In reply to Jimmy Christensen's question that seem to have vanished from the PR)

Looking from the application perspective, the module is collecting metrics from the threadpools and exporting them to prometheus.

So for prometheus it is an exporter, for opencue it is a collector!? I confess I was having problems with the terminology, I'll gladly change if you think it is confusing the way it is now.

---

Diego Tavares da Silva @. @.

On Wed, Jul 10, 2024 at 11:58 PM Jimmy Christensen @.***> wrote:

Not to go too much into semantics here, but isn't this more of an exporter and not a collector?

— Reply to this email directly, view it on GitHub https://github.com/AcademySoftwareFoundation/OpenCue/pull/1408#issuecomment-2222179147, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGXQ24VM6FBZONPCTLZTXTZLYUIRAVCNFSM6AAAAABKV4IC36VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRSGE3TSMJUG4 . You are receiving this because you authored the thread.Message ID: @.***>

[lithorus]

I removed it after I checked up on it. You're absolutely correct 🙂 Just ignore it 😅 But great to see Prometheus support as a priority!

[DiegoTavares]

@bcipriano FYI