Investigate multi-cluster setup

[DzMakatun]

investigate monitoring across multiple clusters (Spark History Server / Menas instances) Idea:

• One crawler process per cluster
• Each crawler writes to its own set of indexes in the same elasticsearch instance
• Kibana index patterns should be able to cover multiple indexes
• The collected data can be filtered by history_host attribute (raw, aggs, errors(?))

Open questions:

• How much flexible are Kibana's index patters?
• How would this interfere with rotating indexes? (For limiting the used storage space)
• how such setups will work for regression process?

[DzMakatun]

Solved within pr #44 It is possible to monitor multiple clusters (each with its own Spark History server) with Spot. For this scenario a separate Spot crawler process needs to be running for each Spark History server (and optionally Menas). Each process writes to its own set of indexes within the same elasticsearch instance. If the index names follow the defined pattern (spot_\<raw/agg/err>_\<clustername>\\<id>) the data can be visualized in Kibana using the setup provided in spot/kibana/. There the data can be filtered by history_host.keyword if required.