search

flux-framework / flux-core

core services for the Flux resource management framework
GNU Lesser General Public License v3.0
167 stars 50 forks source link

Request: job stats info based on timeframe #6387

Open kkier opened 2 hours ago

kkier commented 2 hours ago

The general case of this just about querying the KVS, but I've been handed the task of collecting day to day info about jobs, specifically: jobs succeeded jobs failed jobs canceled

Ideally it needs to be system- and configuration-agnostic, so I can't rely on flux-accounting. The output of flux jobs --stats-only isn't going to work because the counts are only of unpurged jobs, otherwise I could just do a day to day difference.

My need would be met by something that just queries based on a given day like flux_jobs_report 20241022 for all jobs that finished on that day, but I can see use cases for more granularity a la flux_jobs_report --timeframe=-12h or similar.

garlick commented 2 hours ago

Do the requirements imply that flux needs to keep data on purged jobs that is discarding now?

kkier commented 2 hours ago

For my needs, no. I just need to know what happened yesterday/last week/whatever. One way to get that info would be running totals, but that was just me wanting to be hacky and use flux jobs --stats-only instead of actually querying the info. Tell me if this isn't making sense.

garlick commented 2 hours ago

I was responding to this:

The output of flux jobs --stats-only isn't going to work because the counts are only of unpurged jobs

Once jobs are purged they are not reflected anywhere. They are removed from the KVS and unavailable via the job-list service.

kkier commented 1 hour ago

Right, which makes sense. You can't be keeping a running tally of jobs until the heat death of the universe.

garlick commented 40 minutes ago

So... not a problem if your queries can only only reach back to the...event horizon?