observability, jobs: multiple progress phases

[shermanCRL]

Problem

When observing progress of a job, we only have a facility for one “percent complete”. Jobs often have multiple phases, but they are lumped under one number. The user might see rapid progress to (say) 80% and then the last 20% goes more slowly -- because it’s different work!

Desired solution

A facility to record & display multiple (arbitrarily labelled) progress metrics. In the case of restore, it might be the various phases:

• Reading manifests
• Moving the main bytes
• Moving non-main bytes, whatever they may be (like SQL stats)
• Other finalization

It would be great to let the user see which ones are how far along, and which are upcoming.

Each phase could have its own definition of progress:

• https://github.com/cockroachdb/cockroach/issues/67667

Jira issue: CRDB-8697

Epic CRDB-32144

[shermanCRL]

Some similarity in spirit to https://pkg.go.dev/testing#B.ReportMetric

[amruss]

Here is how we currently show progress:

Some points from discussion:

• Time left is an estimate and maybe not the best thing to show users since it can be misleading
• When was the last time we made real progress? Might be a better metric