Epic: Improve deployment of pre-aggregations

[rpaik]

This is an Epic for improving the deployment of pre-aggregations.

Motivation

• It is difficult to build the process of deploying pre-aggregations and changes to pre-aggregations into production.
• There are some discrepancies in configuration options related to pre-aggregations management. It leads to incorrect configuration and bad practices.

Goals/plans

• [x] Add support for pre-aggregations warm-up.
• [x] Streamline and unify configuration options? (orchestratorOptions, preAggregationsOptions, and queueOptions).
• [x] Provide tools and UI in Cube Cloud for pre-aggregations management during the development, testing and deployment.

[benswinburne]

This is something I've been struggling with and having to deploy changes with multiple steps at night to not affect users (but we're about to expand to users in various other timezones so this will be less viable shortly).

Add support for pre-aggregations warm-up.

@rpaik How do you envisage this working on a production environment?

For example, this bug https://github.com/cube-js/cube.js/issues/1825 is going to cause me to have to rename a pre-aggreagion which means there will be a period where the new name aggregations don't exist.

The current route to get around this is

• to create a second pre-aggregation on the older version without the bug. ensure 2nd preaggregation is lower in the cube so the existing one is found first when queried.
• deploy and allow the new preaggregation to build† with the new name (no way of knowning when it's done except for reading logs)
• remove the original pre-aggregation, upgrade to version without name clash error
• deploy and allow cube to clean up tables used by the original pre-aggregation.

† this is super wasteful on a database like Athena where I have all the aggregations, i just need to rename it, perhaps i could indicate that with a rename property instead?

[rpaik]

@paveltiunov can you help with @benswinburne's question 👆 ?

[paveltiunov]

@benswinburne So the current vision is to have 2 simultaneously running cube.js servers with different schema versions in a single cluster: one with the older schema to serve traffic and another one with the new schema to build/warmup pre-aggregations. Then blue-green deployment approach is applied to switch instances after warmup is complete. It works in Cube Cloud so you can test it there to get a sense of how it works.