search

argoflow / argoflow-aws

Argoflow-AWS has been superseded by deployKF
GNU Affero General Public License v3.0
44 stars 29 forks source link

Istio-operator, pod healthy but crds not being deployed #176

Closed EKami closed 3 years ago

EKami commented 3 years ago

When I deploy in my cluster I get this for istio-operator: 1

So everything seems fine until I get the pod logs:

$ k logs pod/istio-operator-769c594554-bctjh -n istio-operator
2021-07-18T09:52:58.674326Z     info    ControlZ available at 127.0.0.1:9876
2021-07-18T09:52:58.674485Z     info    leader election cm: istio-operator-lock
2021-07-18T09:52:59.725624Z     info    klog    Throttling request took 1.040253723s, request: GET:https://172.20.0.1:443/apis/scheduling.k8s.io/v1?timeout=32s
2021-07-18T09:53:00.566689Z     info    Creating operator metrics exporter
2021-07-18T09:53:00.566994Z     info    Registering Components.
2021-07-18T09:53:00.567636Z     info    installer       Adding controller for IstioOperator.
2021-07-18T09:53:00.568105Z     info    installer       Controller added
2021-07-18T09:53:00.568131Z     info    Starting the Cmd.
2021-07-18T09:53:00.568734Z     info    klog    attempting to acquire leader lease istio-operator/istio-operator-lock...
2021-07-18T09:53:23.919016Z     info    klog    successfully acquired lease istio-operator/istio-operator-lock
2021-07-18T09:53:23.924778Z     info    proto: tag has too few fields: "-"
2021-07-18T09:53:23.925446Z     error   klog    k8s.io/client-go@v0.20.5/tools/cache/reflector.go:167: Failed to watch *v1alpha1.IstioOperator: failed to list *v1alpha1.IstioOperator: v1alpha1.IstioOperatorList.Items: []v1alpha1.IstioOperator: v1alpha1.IstioOperator.Status: Spec: unmarshalerDecoder: json: cannot unmarshal object into Go value of type string, error found in #10 byte of ...|h"]}]}]}}},"status":|..., bigger context ...|e","kube-prometheus-stack-admission-patch"]}]}]}}},"status":{"componentStatus":{"Base":{"status":"HE|...
2021-07-18T09:53:25.410070Z     error   klog    k8s.io/client-go@v0.20.5/tools/cache/reflector.go:167: Failed to watch *v1alpha1.IstioOperator: failed to list *v1alpha1.IstioOperator: v1alpha1.IstioOperatorList.Items: []v1alpha1.IstioOperator: v1alpha1.IstioOperator.Status: Spec: unmarshalerDecoder: json: cannot unmarshal object into Go value of type string, error found in #10 byte of ...|h"]}]}]}}},"status":|..., bigger context ...|e","kube-prometheus-stack-admission-patch"]}]}]}}},"status":{"componentStatus":{"Base":{"status":"HE|...

and when I get try to get the crds:

$ k get crds
Enter passphrase to unlock /home/ekami/.awsvault/keys/:
NAME                                         CREATED AT
alertmanagerconfigs.monitoring.coreos.com    2021-07-17T21:03:12Z
alertmanagers.monitoring.coreos.com          2021-07-17T21:03:13Z
applications.argoproj.io                     2021-07-17T19:21:44Z
appprojects.argoproj.io                      2021-07-17T19:21:45Z
certificaterequests.cert-manager.io          2021-07-17T19:26:39Z
certificates.cert-manager.io                 2021-07-17T19:26:39Z
challenges.acme.cert-manager.io              2021-07-17T19:26:40Z
clusterissuers.cert-manager.io               2021-07-17T19:26:40Z
eniconfigs.crd.k8s.amazonaws.com             2021-07-17T19:15:09Z
externalsecrets.kubernetes-client.io         2021-07-17T19:21:30Z
ingressclassparams.elbv2.k8s.aws             2021-07-17T19:18:00Z
issuers.cert-manager.io                      2021-07-17T19:26:40Z
istiooperators.install.istio.io              2021-07-17T19:27:05Z
kialis.kiali.io                              2021-07-17T21:01:00Z
mpijobs.kubeflow.org                         2021-07-17T19:26:06Z
mxjobs.kubeflow.org                          2021-07-17T19:25:58Z
notebooks.kubeflow.org                       2021-07-17T19:23:14Z
orders.acme.cert-manager.io                  2021-07-17T19:26:40Z
poddefaults.kubeflow.org                     2021-07-17T19:28:09Z
podmonitors.monitoring.coreos.com            2021-07-17T21:03:12Z
probes.monitoring.coreos.com                 2021-07-17T21:03:12Z
prometheuses.monitoring.coreos.com           2021-07-17T21:03:13Z
prometheusrules.monitoring.coreos.com        2021-07-17T21:03:12Z
pytorchjobs.kubeflow.org                     2021-07-17T19:23:18Z
sealedsecrets.bitnami.com                    2021-07-17T19:22:40Z
securitygrouppolicies.vpcresources.k8s.aws   2021-07-17T19:15:12Z
servicemonitors.monitoring.coreos.com        2021-07-17T21:03:12Z
targetgroupbindings.elbv2.k8s.aws            2021-07-17T19:25:39Z
tensorboards.tensorboard.kubeflow.org        2021-07-17T19:23:09Z
tfjobs.kubeflow.org                          2021-07-17T19:23:14Z
thanosrulers.monitoring.coreos.com           2021-07-17T21:03:13Z
xgboostjobs.xgboostjob.kubeflow.org          2021-07-17T19:23:18Z

No trace of istio crds resulting in this: 2

It seems the distribution/istio/istio-spec.yaml is referencing a config map that is non-existing:

          service.beta.kubernetes.io/aws-load-balancer-scheme:
            valueFrom:
              configMapKeyRef:
                name: cluster-parameters
                key: load_balancer.scheme

I can't find any cluster-parameters configmap in the repo.

If I comment out this part of distribution/istio/istio-spec.yaml:

#  components:
#    ingressGateways:
#    - name: istio-ingressgateway
#      k8s:
#        service:
#          type: LoadBalancer
#        serviceAnnotations:
#          service.beta.kubernetes.io/aws-load-balancer-name:
#            valueFrom:
#              configMapKeyRef:
#                name: cluster-parameters
#                key: load_balancer.name
#          # service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold: "3"
#          # service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold: "3"
#          # service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval: "10"
#          #service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: "HTTP"
#          #service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "15021"
#          #service.beta.kubernetes.io/aws-load-balancer-healthcheck-path
#          service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
#          service.beta.kubernetes.io/aws-load-balancer-scheme:
#            valueFrom:
#              configMapKeyRef:
#                name: cluster-parameters
#                key: load_balancer.scheme
#          service.beta.kubernetes.io/aws-load-balancer-type: external
#          service.beta.kubernetes.io/aws-load-balancer-nlb-target-type:
#            valueFrom:
#              configMapKeyRef:
#                name: cluster-parameters
#                key: load_balancer.nlb_target_type
#          proxy.istio.io/config: '{"gatewayTopology" : { "numTrustedProxies": 2 } }'
#          external-dns.alpha.kubernetes.io/hostname:
#            valueFrom:
#              configMapKeyRef:
#                name: cluster-parameters
#                key: route53.hosted_zone_hostnames
#  values:
#    sidecarInjectorWebhook:
#      neverInjectSelector:
#      # kube-prometheus-stack
#      ## Admission Webhook jobs do not terminate as expected with istio-proxy
#      - matchExpressions:
#        - {key: app, operator: In, values: [kube-prometheus-stack-admission-create, kube-prometheus-stack-admission-patch]}

Then everything works flawlessly, the crds are deployed, #169 gets solved but ofc, that creates other issues down the line. Any idea how to solve this? Thanks!

EKami commented 3 years ago

Closing, it was probably something that I changed myself but the definition of istio-operator.yaml doesn't match what is currently on this repo, the is no configmap on this repo. Sorry for that.