Integrate Job Client By Status computation into API

[ChaiWithJai]

Proposal

The Nomad SysBatch UI created a client-side computation for Job Client By Status. We'd like to see this status become an API endpoint that the client can consume from the server.

The response schema of the read job summary endpoint is perfect for the data visualization pattern we're following.

Background

To determine the status of a job in a particular client will require grouping all the allocations of the job by client ID, and then aggregating their desired and client statuses, deriving a single job status value.

It's possible for a system or sysbatch job to not have allocations placed in some of the clients. This could happen due to constraints or the client not being available at the time the job was registered.

The possible job statuses in a client will be based on the existing allocation summary values, with the addition of two new ones: degraded and not scheduled.

• queued is the initial status and indicates that the job was registered but not scheduled yet.
• not scheduled represents clients that were not considered for any allocation placement for this job. The reasons for this status can be nebulous, so a popup with a help message will be available.
• starting indicates that at least one allocation of the job has the client status of pending.
• running indicates that at least one allocation of the job has the client status of running.
• complete is set when all allocations reach the client status of complete.
• degraded can happen when some allocations do not complete successfully.
• failed is set when all allocations reach the client status of failed.
• lost is set when all allocations reach the client status of lost.

[ChaiWithJai]

@tgross LMK if this needs more description for you to get up-to-speed on how we implemented this.

[tgross]

I've got a couple of questions on this, for my clarity:

• Where does the "Job Status for Client" box get surfaced today? I see the job-client-status-summary.hbs file but I'm not sure what page that gets rendered on. I just looked at the /ui/jobs and /ui/jobs/$myjob pages and I only see the allocation status box on the job index page and the summary bar on the jobs list page (which is also allocation status).
• I'm not sure I understand how the sysbatch "degraded" state contributes to this issue... it adds a couple of branches to the code in job-client-status.js, but it doesn't add additional API calls currently, right?

As far as the idea of adding this information to the Job Summary API, that seems like a very reasonable spot for it to me as well. We don't have the data in the state store's job_summary table but its trivial to get from that spot in the code by querying allocations for the job, and doesn't require any new ACLs.

One small item that comes to mind is that the data will be degraded during the cluster upgrade that bridges onto this feature. Ex. if I'm hitting the web UI on a server that's a follower that has the UI change but the leader doesn't, the data will temporarily disappear from the UI. I don't think we normally worry about this kind of thing, but I've been running a cluster where the UI is accessible only from one of the clients lately and so forwarding of the API has been on my mind 😀