Design Improvement: Introduce Intermediary Service API for Modifying Node Metadata

[xanderdunn]

The current implementation assumes that node metadata could be changed by anyone, anywhere, at any time. As a result, it must watch for events triggered by anyone.

This has drawbacks:

• We do some client-side filtering across all transactions, which is not optimal
• If someone else makes rapid changes to node labels right after the node is added to the cluster, these changes will be overwritten by our Processor service, which sets all labels on the node with what we have stored.

Instead, it would be a considerable improvement to the design if we could make all node metadata changes through an intermediary API. This service could be responsible for committing versioned transactions to a DB. Then, a separate process could pick up these transactions from the DB and retry until success applying each idempotent metadata update to the node. This guarantees that all node metadata is stored and recoverable even before it has been applied.

[xanderdunn]

The downside to this proposed approach is that we could only make direct changes to node labels through this service. Node metadata modifications from anywhere else would not be stored. But, this should be easy to solve. The API for modifying node metadata exposed by this new service would be called in almost exactly the same way that the Kubernetes API is called to modify node metadata.

[xanderdunn]