[FEATURE REQUEST] Expose metrics via an engine

[nicholasmhughes]

Ported from https://github.com/saltstack/salt/issues/49939

Describe the solution you'd like Prometheus is quickly becoming de-facto for monitoring and alerting. The model of pull instead of push would fit well in the salt-master/salt-minion topology. Exposing metrics in salt-minion and salt-master would allow more flexibility in monitoring large salt environments.

Metrics suggested by @gnalsa

salt_minion_last_run_epoch
salt_minion_last_run_state_completion_time
salt_minion_last_run_state_executed
salt_minion_last_run_state_error
salt_minion_last_run_state_success

Metrics suggested by @justindesilets

salt_master_keys{key_state="accepted"}
salt_master_gitfs_lock
salt_master_number_of_scheduled_jobs
salt_master_number_of_threads
salt_master_number_of_jobs_active
salt_master_number_of_minions_return
salt_master_running_process
salt_syndic_running_process
salt_syndic_master_sync

Describe alternatives you've considered https://github.com/BonnierNews/saltstack_exporter rebuttal: https://github.com/saltstack/salt/issues/49939#issuecomment-465041644

Additional context An internal project is in process to provide this functionality, and should be upstreamed soon.

[MartinEmrich]

https://github.com/BonnierNews/saltstack_exporter seems abandoned (last commit 7 years ago, README mentions Python 2.7).