search

tigera / operator

Kubernetes operator for installing Calico and Calico Enterprise
Apache License 2.0
185 stars 140 forks source link

Calico-node daemonset should get custom environment variables through tigera-operator #2397

Open jacksamol24 opened 1 year ago

jacksamol24 commented 1 year ago

Hi

I tried to upgrade existing calico installation ( https://github.com/aws/amazon-vpc-cni-k8s/blob/v1.7.5/config/v1.7/calico.yaml )

If calico-node is not having any custom environment variable then tigera-operator does the migration well

but in few environments we have used squid proxy and provided three more variables to calico-node pod as proxy_http, proxy_https, proxy_not_needed

as current calico-node configuration is custom in such environments, Tigera-operator fails to do the migration

These variables are important from the feature and security implementation perspective so they can not be removed

So is it possible to get the support for

  1. adding environment variables through tigera-operator to calico-node while migration through Installation Object or some other way, as no downtime can be tolerated considering production grade scenario?
  2. and, changing the image repository to pull from private repository

These features are important for one of the production grade migration

Currently, latest tigera-operator v3.24.5 supports customizing name and resource attributes of calico-node through Installation object Reference: https://projectcalico.docs.tigera.io/reference/installation/api#operator.tigera.io/v1.CalicoNetworkSpec https://projectcalico.docs.tigera.io/reference/installation/api#operator.tigera.io/v1.CalicoNodeDaemonSetContainer

Attaching Logs of Tigera-operator: Kubectl logs -n tigera-operator

{"level":"error","ts":1673416430.9388688,"logger":"controller_installation","msg":"Existing Calico installation can not be managed by Tigera Operator as it is configured in a way that Operator does not currently support. Please update your existing Calico install config","Request.Namespace":"tigera-operator","Request.Name":"tigera-operator-token-726g9","error":"unexpected env vars: [calico-node/proxy_not_needed calico-node/proxy_http calico-node/proxy_https]. To fix it, remove these environment variables from the calico-node daemonest on daemonset/calico-node","stacktrace":"github.com/go-logr/zapr.(zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132\ngithub.com/tigera/operator/pkg/controller/installation.(ReconcileInstallation).SetDegraded\n\t/go/src/github.com/tigera/operator/pkg/controller/installation/core_controller.go:1324\ngithub.com/tigera/operator/pkg/controller/installation.(ReconcileInstallation).Reconcile\n\t/go/src/github.com/tigera/operator/pkg/controller/installation/core_controller.go:726\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:235\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:198\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:99"}

tmjd commented 1 year ago
  1. and, changing the image repository to pull from private repository

This is already available with the operator through the Installation resource, and the how-to for a private registry is here.

  1. adding environment variables through tigera-operator to calico-node while migration through Installation Object or some other way, as no downtime can be tolerated considering production grade scenario?

The environment variables you suggest you need are proxy config env variables. Are you sure you need those env variables because all calico-node traffic should be only to calico-typha and to the K8s apiserver (all traffic that I would expect to be internal traffic), so I would imagine there wouldn't be a need to direct the traffic to squid since it wouldn't be leaving your network. What about for calico-typha and calico-kube-controllers?

Did you also need to set those environment variables on the tigera-operator pod also?