Caliper can send metrics to a prometheus push gateway and then prometheus can scrape the metrics from the gateway however prometheus doesn't recommend using the push gateway.
To reduce the complexity and components that need to be deployed, we can use the caliper internal messaging system to send worker stats to the manager and have the manager expose an end point so that prometheus can scrape from the manager.
This provides a simplier setup to using the push gateway as well as prometheus attempting to scrape individual workers (in my case I have run 180-240 workers which is a lot of endpoints to locate and scrape from)
This will put more workload on the manager so it would be good to understand
the increased resources required for the manager
any resource changes at the worker level
Finally we need to be able to configure how the worker sends information, eg periodically or at some sort of threshold of number of stat objects collected so far (eg a value of 1 might mean to never collate and just transmit, what should a value of 0 mean then or should 0 be invalid)
More detail about this implementation can be provided
Caliper can send metrics to a prometheus push gateway and then prometheus can scrape the metrics from the gateway however prometheus doesn't recommend using the push gateway.
To reduce the complexity and components that need to be deployed, we can use the caliper internal messaging system to send worker stats to the manager and have the manager expose an end point so that prometheus can scrape from the manager.
This provides a simplier setup to using the push gateway as well as prometheus attempting to scrape individual workers (in my case I have run 180-240 workers which is a lot of endpoints to locate and scrape from)
This will put more workload on the manager so it would be good to understand
Finally we need to be able to configure how the worker sends information, eg periodically or at some sort of threshold of number of stat objects collected so far (eg a value of 1 might mean to never collate and just transmit, what should a value of 0 mean then or should 0 be invalid)
More detail about this implementation can be provided