search

kinvolk / lokomotive

🪦 DISCONTINUED Further Lokomotive development has been discontinued. Lokomotive is a 100% open-source, easy to use and secure Kubernetes distribution from the volks at Kinvolk
https://kinvolk.io/lokomotive-kubernetes/
Apache License 2.0
321 stars 49 forks source link

Impossible to create a cluster with prometheus and a monitored contour right from the start #750

Open margamanterola opened 4 years ago

margamanterola commented 4 years ago

I'm creating a cluster from scratch that includes a contour component with monitoring enabled and a prometheus-operator component (among other components, as seen below).

When creating the cluster and applying the components, this happened:

INFO[0645] Applying component configuration              args="[]" command="lokoctl cluster apply"
Applying component 'dex'...
Successfully applied component 'dex' configuration!
Applying component 'gangway'...
Successfully applied component 'gangway' configuration!
Applying component 'openebs-operator'...
Successfully applied component 'openebs-operator' configuration!
Applying component 'openebs-storage-class'...
Successfully applied component 'openebs-storage-class' configuration!
Applying component 'cert-manager'...
Successfully applied component 'cert-manager' configuration!
Applying component 'contour'...
FATA[0682] Applying component configuration failed: installing release failed: unable to build kubernetes objects from release manifest: [unable to recognize "": no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"]  args="[]" command="lokoctl cluster apply"

This is because contour was applied before prometheus-operator and so it couldn't enable monitoring as requested. I manually applied prometheus-operator and then all components, and it worked:

$ lokoctl component apply prometheus-operator --lokocfg-vars ../lokocfg.vars 
Applying component 'prometheus-operator'...
2020/07/31 11:19:12 info: skipping unknown hook: "crd-install"
2020/07/31 11:19:12 info: skipping unknown hook: "crd-install"
2020/07/31 11:19:12 info: skipping unknown hook: "crd-install"
2020/07/31 11:19:12 info: skipping unknown hook: "crd-install"
2020/07/31 11:19:12 info: skipping unknown hook: "crd-install"
Successfully applied component 'prometheus-operator' configuration!
$ lokoctl component apply --lokocfg-vars ../lokocfg.vars 
Applying component 'dex'...
Successfully applied component 'dex' configuration!
Applying component 'gangway'...
Successfully applied component 'gangway' configuration!
Applying component 'openebs-operator'...
Successfully applied component 'openebs-operator' configuration!
Applying component 'openebs-storage-class'...
Successfully applied component 'openebs-storage-class' configuration!
Applying component 'cert-manager'...
Successfully applied component 'cert-manager' configuration!
Applying component 'contour'...
Successfully applied component 'contour' configuration!
Applying component 'metallb'...
Successfully applied component 'metallb' configuration!
Applying component 'external-dns'...
Successfully applied component 'external-dns' configuration!
Applying component 'prometheus-operator'...
2020/07/31 11:21:02 info: skipping unknown hook: "crd-install"
2020/07/31 11:21:02 info: skipping unknown hook: "crd-install"
2020/07/31 11:21:02 info: skipping unknown hook: "crd-install"
2020/07/31 11:21:02 info: skipping unknown hook: "crd-install"
2020/07/31 11:21:02 info: skipping unknown hook: "crd-install"
Successfully applied component 'prometheus-operator' configuration!

I'm not sure how the order of components is decided when doing cluster apply, but it should be done in a way that creating the cluster works when these two components are installed at the same time.

invidian commented 4 years ago

Thanks for reporting @marga-kinvolk. This is a known issue and already reported in #557. There is even PR opened for it https://github.com/kinvolk/lokomotive/pull/567 (I already forgot :smile:).

I'm not sure how the order of components is decided when doing cluster apply, but it should be done in a way that creating the cluster works when these two components are installed at the same time.

The order of installing the components is defined by the order in the configuration file. I don't know though, how they are ordered, if you spread the configuration across multiple files (e.g. one file per component). Can you tell how did you configure the components?

margamanterola commented 4 years ago

Sure, I have:

I'm guessing it goes alphabetically through the files, so I need to rename either ingress or monitoring to have them in a different order...

invidian commented 4 years ago

So, the files are read also alphabetically, as they are returned from the glob *.lokocfg.

invidian commented 4 years ago

I'm keeping this one open for now, as we need to address in which order the components are being installed, regardless of the user configuration form.