Run model training and metrics exporting in two different containers

[4n4nd]

This would solve at least a couple of issues:

• 50

• We can just create one exporter for multiple metrics, so we don't have to change scraping targets in the prometheus host often.

The only tricky part is, how do you share the predicted metrics between the model-training container and the metrics exporter?

Possible solutions:

• Maybe we can just store these predicted metrics in s3 and at regular intervals delete them. (too expensive)
• Or maybe something like a pushgateway?

[4n4nd]

I think the pushgateway solution would work well when we convert everything to rx streams