terascope / teraslice

Scalable data processing pipelines in JavaScript
https://terascope.github.io/teraslice/
Apache License 2.0
50 stars 13 forks source link

Extract a `metrics_api` from the Job Metrics API processor in standard-assets #3360

Closed godber closed 6 months ago

godber commented 1 year ago

In standard assets we added the following job_metric_api:

https://github.com/terascope/standard-assets/blob/master/asset/src/job_metric_api/api.ts

We should extract out the API into Teraslice itself so the metrics can be used elsewhere

See registerApis here:

https://github.com/terascope/teraslice/blob/master/packages/job-components/src/register-apis.ts#L113

... and the APIs that get registered could be good examples.

Note that when this issue is done we need to do the following:

godber commented 7 months ago

Play with prom-client a bit:

https://github.com/siimon/prom-client

busma13 commented 6 months ago

We decided that the proper place for this to live is in the terafoundation foundation APIS. https://github.com/terascope/teraslice/blob/master/packages/terafoundation/src/api/index.ts

busma13 commented 6 months ago

The registerApis function within terafoundation contains the foundationApis object. It will now have a promMetrics key that contains all the functions related to the prom-metrics-api.

There was a lot of discussion about when the API and the associated exporter server should be created. I've experimented with creating the API at the terafoundation level vs the teraslice master, execution_controller, and worker level.