Create a well-defined way to pass infra data to components

[johananl]

General

In many cases, a Lokomotive component may need information from the infrastructure layer (Terraform). Examples:

• A component should only be deployed on AWS.
• A component needs to know the IP address of a machine (as in https://github.com/kinvolk/lokomotive/pull/493 for example).

We should think about adding a generic mechanism for making infrastructure data available to components.

Implementation

Terraform knows everything about the infrastructure. Therefore, it probably makes sense to rely on Terraform outputs as the source of the information.

There are at least two main approaches on how to make infrastructure information available to component: via lokoctl and via the k8s datastore.

Passing information via lokoctl

In this approach, lokoctl contains logic which reads Terraform outputs into memory. The components code can then use the in-memory information to interpolate values into component manifests, enable/disable components altogether etc.

If this approach is taken, it is critical to re-apply components following an infrastructure change.

Passing information via the k8s datastore

In this approach, Terraform outputs are written to the k8s datastore as k8s objects (e.g. using ConfigMap objects or custom k8s resources). Then, a component manifest can refer to a value from the relevant k8s object rather than using a hardcoded value.

IMO this approach is problematic for the following reasons:

• You can't know exactly how a component is going to behave by looking at its rendered manifest.
• You can't decide not to deploy a component altogether based on a value derived from the infrastructure.

[johananl]

@iaguis @surajssd @invidian - FYI.

[invidian]

Some thoughts:

• This issue should also go together with reducing the amount of things the Terraform is responsible for (#181), as e.g. right now it deals with generating Helm chart values and current way of propagating the information is very complex because of that. Terraform should ideally be only responsible for talking to Cloud APIs to create the infrastructure. This means, for some things, it might make more sense to move them outside of Terraform, instead of creating them in Terraform and then passing them to the components configuration.
• I am also in favor of doing that in the lokoctl over passing the information via k8s. This again, reduces the amount of responsibility for Terraform. As we discussed in #364, at least on the code level, the components should be the "property" of the cluster, and so should be the infrastructure part of the cluster. This means, the plumbing we implement should be stored in cluster-related code. The relation diagram would look like following:
• For this kind of component configuration enhancements, we should decide if we want to support opting-out from these. This means, if component A is being installed because cluster is on platform B, should user have an option to disable this integration?

[surajssd]

Using kubernetes as data store makes more sense from the perspective of controllers and operators.

Going forward when Lokomotive relies on it’s own operators and controllers, getting that information from terraform output would pose same problem we have today; How to extract data from terraform!

[invidian]

@surajssd I think considering controllers is slightly beyond our scope for now, as I think we will still need Terraform for infrastructure parts, at least initially, so it doesn't really matter where we run it.