jobs: add markers to categorize user created jobs and background service jobs

[adityamaru]

As suggested by @dt we should add a few more bits of information to be able to categorize jobs into:

• System task/background job
• User-created job

Within the System task category we should further subdivide into:

• Forever running job
• One-time task that runs and completes

The primary motivation for this categorization is to improve the UX around how users can track currently running jobs in the DB console. Today, the console has an ALL jobs tab, and then each job type has a separate tab. Confusingly, the ALL jobs table does not show all jobs for example all the automatic jobs such as key viz, create stats, span config reconciliation etc. If we were to categorize jobs then we could split the System tasks from the User-created jobs and make it easier for customers to identify all the processes that are currently running in their cluster.

Furthermore, forever running background services should probably not be incrementing and decrementing job-related metrics such as jobs_running_non_idle. This metric is currently used by serverless to decide whether or not to scale down a SQL pod. In the future, this is going to be used to annotate Prometheus graphs with information about processes that are doing work in the cluster. It would be useful to separate jobs from services by introducing another system_service_running_non_idle. This can be toggled when the background services are actually doing work rather than just idling waiting for their next run.

Jira issue: CRDB-27113

[dt]

it's possible system task is just creation user == node or not

[irfansharif]

Once this gets fixed, I might give https://github.com/cockroachdb/cockroach/pull/103215#issuecomment-1546237647 another shot.