[Feature Request] Add temporal_worker_task_slots_total metric

[ghaskins]

Is your feature request related to a problem? Please describe.

It's harder to build meaningful visualizations on temporal_worker_task_slots_available alone when the total amount is unknown.

Most of the time, the graphs look like this

It's particularly hard to see nodes with smaller task slot allocations....and these are precisely the candidates likely to run into resource exhaustion.

Describe the solution you'd like

Providing a means to retrieve the total possible slots would make it trivial to graph the utilization percent using an 'available/total' approach. This would make it easier to observe the cases that require the most attention by better visual emphasis and alerting targets.

[cretz]

when the total amount is unknown.

But since you set this value, it is known. You can expose this fixed value as a metric if you'd like.

[Quinn-With-Two-Ns]

I think the SDK exposing this as a metric makes sense given these feature requests https://github.com/temporalio/features/issues/334 https://github.com/temporalio/features/issues/388

[ghaskins]

But since you set this value, it is known.

Well, it's known somewhere. But it's not conveniently accessible to metrics/dashboard consumers, making developing things like dashboards more fragile instead of generic.