search

voyagermesh / voyager

🚀 Secure L7/L4 (HAProxy) Ingress Controller for Kubernetes
https://voyagermesh.com
Apache License 2.0
1.35k stars 134 forks source link

creating voyager ingress on k8s version 1.20 is failing #1589

Open sankarpn opened 3 years ago

sankarpn commented 3 years ago

Trying to create voyager ingress on k8s version 1.20 in Oracle Linux Cloud Native Environment 1.3 https://docs.oracle.com/en/operating-systems/olcne/index.html

The voyager service is not getting created. Here is the complete reproduction steps.

This k8s cluster is running CRIO runtime

bash $ kubectl get nodes -o wide
NAME      STATUS   ROLES                  AGE   VERSION         INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                  KERNEL-VERSION                     CONTAINER-RUNTIME
olcne-2   Ready    control-plane,master   9d    v1.20.6+1.el7   ***********   <none>        Oracle Linux Server 7.9   5.4.17-2102.200.13.el7uek.x86_64   cri-o://1.20.2
olcne-3   Ready    <none>                 9d    v1.20.6+1.el7   ***********   <none>        Oracle Linux Server 7.9   5.4.17-2102.200.13.el7uek.x86_64   cri-o://1.20.2
base $ helm repo add appscode https://charts.appscode.com/stable/
"appscode" has been added to your repositories

bash $ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "ingress-nginx" chart repository
...Successfully got an update from the "weblogic-operator" chart repository
...Successfully got an update from the "traefik" chart repository
...Successfully got an update from the "appscode" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈ Happy Helming!⎈

bash $ helm install voyager-release-dpam appscode/voyager --namespace ns-dpam  --version 12.0.0 --set "apiserver.enableValidatingWebhook=false" --set "cloudProvider=baremetal"
NAME: voyager-release-dpam
LAST DEPLOYED: Fri Jul  9 17:36:09 2021
NAMESPACE: ns-dpam
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Set cloudProvider for installing Voyager

To verify that Voyager has started, run:

  kubectl get deployment --namespace ns-dpam -l "app.kubernetes.io/name=voyager,app.kubernetes.io/instance=voyager-release-dpam"

bash $ cat voyager.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.appscode.com/affinity: cookie
    ingress.appscode.com/type: NodePort
    kubernetes.io/ingress.class: voyager
  name: coherence-managed-domain1-ingress-host-routing
  namespace: ns-ejvn
spec:
  rules:
  - host: coherence-managed-domain1.ns-ejvn.cluster-1.test
    http:
      paths:
      - backend:
          serviceName: coherence-managed-domain1-cluster-cluster-1
          servicePort: 8001
  - host: coherence-managed-domain1.ns-ejvn.cluster-2.test
    http:
      paths:
      - backend:
          serviceName: coherence-managed-domain1-cluster-cluster-2
          servicePort: 8001

bash $ kubectl create -f voyager.yaml
Warning: networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
ingress.networking.k8s.io/coherence-managed-domain1-ingress-host-routing created

bash $ kubectl get services -n ns-ejvn -w
NAME                                                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)              AGE
coherence-managed-domain1-admin-server                ClusterIP   None            <none>        30012/TCP,7001/TCP   108m
coherence-managed-domain1-admin-server-ext            NodePort    10.109.3.30     <none>        7001:30229/TCP       108m
coherence-managed-domain1-cluster-1-managed-server1   ClusterIP   None            <none>        8001/TCP             107m
coherence-managed-domain1-cluster-1-managed-server2   ClusterIP   None            <none>        8001/TCP             107m
coherence-managed-domain1-cluster-2-managed-server1   ClusterIP   None            <none>        8001/TCP             107m
coherence-managed-domain1-cluster-2-managed-server2   ClusterIP   None            <none>        8001/TCP             107m
coherence-managed-domain1-cluster-cluster-1           ClusterIP   10.103.130.66   <none>        8001/TCP             107m
coherence-managed-domain1-cluster-cluster-2           ClusterIP   10.102.211.96   <none>        8001/TCP             107m
voyager-coherence-managed-domain1-ingress-host-routing   NodePort    10.98.65.190    <none>        80:32420/TCP         0s
voyager-coherence-managed-domain1-ingress-host-routing   NodePort    10.98.65.190    <none>        80:32420/TCP         0s
voyager-coherence-managed-domain1-ingress-host-routing   NodePort    10.96.118.25    <none>        80:30313/TCP         0s
voyager-coherence-managed-domain1-ingress-host-routing   NodePort    10.96.118.25    <none>        80:30313/TCP         0s
voyager-coherence-managed-domain1-ingress-host-routing   NodePort    10.101.125.31   <none>        80:30898/TCP         0s
voyager-coherence-managed-domain1-ingress-host-routing   NodePort    10.101.125.31   <none>        80:30898/TCP         0s

voyager operator logs shows this messages

bash $ kubectl logs -n ns-dpam pod/voyager-release-dpam-6f5c48dc55-kkrmg

I0709 17:38:06.065975 1 services.go:74] Add/Delete/Update of offshoot service ns-ejvn/voyager-coherence-managed-domain1-ingress-host-routing, Ingress ns-ejvn/coherence-managed-domain1-ingress-host-routing re-queued for update E0709 17:38:11.057137 1 worker.go:92] Failed to process key ns-ejvn/coherence-managed-domain1-ingress-host-routing. Reason: services "voyager-coherence-managed-domain1-ingress-host-routing" not found I0709 17:38:11.057487 1 worker.go:96] Error syncing key ns-ejvn/coherence-managed-domain1-ingress-host-routing: services "voyager-coherence-managed-domain1-ingress-host-routing" not found I0709 17:38:11.057735 1 nodeport.go:91] [2bba0482-3244-4e54-a4b8-d0857fd48f73] Initializing cloud manager for provider baremetal I0709 17:38:11.057844 1 nodeport.go:119] [2bba0482-3244-4e54-a4b8-d0857fd48f73] No cloud manager found for provider baremetal I0709 17:38:11.057868 1 ingresses.go:128] Sync/Add/Update for ingress ns-ejvn/coherence-managed-domain1-ingress-host-routing I0709 17:38:11.061296 1 service.go:35] Creating Service ns-ejvn/voyager-coherence-managed-domain1-ingress-host-routing. I0709 17:38:11.078513 1 services.go:55] Sync/Add/Update for Service voyager-coherence-managed-domain1-ingress-host-routing I0709 17:38:11.091023 1 endpoints.go:52] Sync/Add/Update for Endpoints ns-ejvn/voyager-coherence-managed-domain1-ingress-host-routing W0709 17:38:11.095246 1 services.go:43] Service ns-ejvn/voyager-coherence-managed-domain1-ingress-host-routing does not exist anymore

kube-apiserver shows these messages

bash $ kubectl logs -n kube-system pod/kube-apiserver-olcne-2

W0709 17:14:25.196826 1 controller.go:142] slow openapi aggregation of "certificates.voyager.appscode.com": 1.017908073s E0709 17:14:26.070779 1 customresource_handler.go:669] error building openapi models for ingresses.voyager.appscode.com: ERROR $root.definitions.com.appscode.voyager.v1beta1.Ingress.properties.spec.properties.backend.properties.servicePort has invalid property: anyOf ERROR $root.definitions.com.appscode.voyager.v1beta1.Ingress.properties.spec.properties.frontendRules.items..properties.port has invalid property: anyOf ERROR $root.definitions.com.appscode.voyager.v1beta1.Ingress.properties.spec.properties.livenessProbe.properties.httpGet.properties.port has invalid property: anyOf ERROR $root.definitions.com.appscode.voyager.v1beta1.Ingress.properties.spec.properties.livenessProbe.properties.tcpSocket.properties.port has invalid property: anyOf ERROR $root.definitions.com.appscode.voyager.v1beta1.Ingress.properties.spec.properties.readinessProbe.properties.httpGet.properties.port has invalid property: anyOf ERROR $root.definitions.com.appscode.voyager.v1beta1.Ingress.properties.spec.properties.readinessProbe.properties.tcpSocket.properties.port has invalid property: anyOf ERROR $root.definitions.com.appscode.voyager.v1beta1.Ingress.properties.spec.properties.rules.items..properties.http.properties.nodePort has invalid property: anyOf ERROR $root.definitions.com.appscode.voyager.v1beta1.Ingress.properties.spec.properties.rules.items..properties.http.properties.paths.items..properties.backend.properties.servicePort has invalid property: anyOf ERROR $root.definitions.com.appscode.voyager.v1beta1.Ingress.properties.spec.properties.rules.items..properties.http.properties.port has invalid property: anyOf ERROR $root.definitions.com.appscode.voyager.v1beta1.Ingress.properties.spec.properties.rules.items..properties.tcp.properties.backend.properties.servicePort has invalid property: anyOf ERROR $root.definitions.com.appscode.voyager.v1beta1.Ingress.properties.spec.properties.rules.items..properties.tcp.properties.nodePort has invalid property: anyOf ERROR $root.definitions.com.appscode.voyager.v1beta1.Ingress.properties.spec.properties.rules.items..properties.tcp.properties.port has invalid property: anyOf

tamalsaha commented 3 years ago

@sankarpn , please try the https://github.com/voyagermesh/voyager/releases/tag/v13.0.0-rc.0 release. That should work.

sankarpn commented 3 years ago

Its still not working, I tried both https://artifacthub.io/packages/helm/appscode/voyager/13.0.0-beta.1 and https://artifacthub.io/packages/helm/appscode/voyager/2021.4.24-rc.0

tamalsaha commented 3 years ago

@sankarpn, if you are interested in acquiring license for the Voyager Enterprise, please fill up the form here: https://appscode.com/contact/ . We have stopped doing free work for other businesses.

sankarpn commented 3 years ago

@tamalsaha We don't use voyager in production. We test and certify that voyager works with our Oracle WebLogic k8s operator in different environments. There are a lot of customers will be using voyager when we certify.

tamalsaha commented 3 years ago

@sankarpn , that's exciting. Could we set up a call to get a better understanding?

tamalsaha commented 3 years ago

My can reach me by email tamal AT appscode.com .

sankarpn commented 3 years ago

I tried to install version 2021.4.24-rc.0 using the configuration values from https://artifacthub.io/packages/helm/appscode/voyager/2021.4.24-rc.0 , looks like the "operator.tag" doesn't set the release version.

helm install voyager-release50-huha appscode/voyager --namespace ns-huha --set "operator.tag=v13.0.0-rc.0" --set "apiserver.enableValidatingWebhook=true" --set "cloudProvider=baremetal"

As you can see from the following describe it still uses the v12.0.0 version

[opc@olcne-1 ~]$ kubectl describe -n ns-huha pod/voyager-release50-huha-68c5c66d-l5w8m Name: voyager-release50-huha-68c5c66d-l5w8m Namespace: ns-huha Priority: 0 Node: olcne-3/10.0.3.76 Start Time: Fri, 03 Sep 2021 18:07:48 +0000 Labels: app.kubernetes.io/instance=voyager-release50-huha app.kubernetes.io/name=voyager pod-template-hash=68c5c66d Annotations: checksum/apiregistration.yaml: 351fa9985c13336fe4fd47127ed5f6f165a1c79e1c60d6a1296709b9331a4234 Status: Running IP: 10.244.1.99 IPs: IP: 10.244.1.99 Controlled By: ReplicaSet/voyager-release50-huha-68c5c66d Containers: voyager: Container ID: cri-o://59d8bc2d720fc76c096536acc022d52c2669c5e1d2f7889e5fe6e7f2f99627d1 Image: appscode/voyager:v12.0.0 Image ID: docker.io/appscode/voyager@sha256:1569c90b098c301fde05efeab508c5adc44c63690e091e9db55e6d362de04adb Ports: 8443/TCP, 56791/TCP Host Ports: 0/TCP, 0/TCP Args: run --cloud-provider=baremetal --cloud-config= --v=3 --ingress-class= --operator-service=voyager-release50-huha --docker-registry=appscode --haproxy-image-tag=1.9.15-v12.0.0-alpine --haproxy-image-repository=haproxy --exporter-image-tag=v12.0.0 --exporter-image-repository=voyager --secure-port=8443 --audit-log-path=- --tls-cert-file=/var/serving-cert/tls.crt --tls-private-key-file=/var/serving-cert/tls.key --enable-validating-webhook=true --bypass-validating-webhook-xray=false --use-kubeapiserver-fqdn-for-aks=true --enable-analytics=true State: Running Started: Fri, 03 Sep 2021 18:07:49 +0000 Ready: True Restart Count: 0 Environment: MY_POD_NAME: voyager-release50-huha-68c5c66d-l5w8m (v1:metadata.name) MY_POD_NAMESPACE: ns-huha (v1:metadata.namespace) Mounts: /var/run/secrets/kubernetes.io/serviceaccount from voyager-release50-huha-token-47xvl (ro) /var/serving-cert from serving-cert (rw)

I had to use thevoyager.tag to correctly install the version desired.

helm install voyager-release50-huha appscode/voyager --namespace ns-huha --set "voyager.tag=v13.0.0-rc.0" --set "apiserver.enableValidatingWebhook=true" --set "cloudProvider=baremetal"

[opc@olcne-1 ~]$ kubectl describe -n ns-huha pod/voyager-release50-huha-7f46b9fb6f-9p6lx Name: voyager-release50-huha-7f46b9fb6f-9p6lx Namespace: ns-huha Priority: 0 Node: olcne-3/10.0.3.76 Start Time: Fri, 03 Sep 2021 18:10:39 +0000 Labels: app.kubernetes.io/instance=voyager-release50-huha app.kubernetes.io/name=voyager pod-template-hash=7f46b9fb6f Annotations: checksum/apiregistration.yaml: de5e98866b6881061ef385371c49e7e22da221ac39955760e8da95c6182f9394 Status: Running IP: 10.244.1.101 IPs: IP: 10.244.1.101 Controlled By: ReplicaSet/voyager-release50-huha-7f46b9fb6f Containers: voyager: Container ID: cri-o://71c6fcb89bbfbfef3bb735e198a6029eceb4ec36c3f8dfcd9a36c66e9707f7f4 Image: appscode/voyager:v13.0.0-rc.0 Image ID: docker.io/appscode/voyager@sha256:6d35a828ef929c04c5cda70c8013389a2de78035d04bc10c339b585588b5abb0 Ports: 8443/TCP, 56791/TCP Host Ports: 0/TCP, 0/TCP Args: run --cloud-provider=baremetal --cloud-config= --v=3 --ingress-class= --operator-service=voyager-release50-huha --docker-registry=appscode --haproxy-image-tag=1.9.15-v12.0.0-alpine --haproxy-image-repository=haproxy --exporter-image-tag=v13.0.0-rc.0 --exporter-image-repository=voyager --secure-port=8443 --audit-log-path=- --tls-cert-file=/var/serving-cert/tls.crt --tls-private-key-file=/var/serving-cert/tls.key --enable-validating-webhook=true --bypass-validating-webhook-xray=false --use-kubeapiserver-fqdn-for-aks=true --enable-analytics=true

But again look at the --haproxy-image-tag=1.9.15-v12.0.0-alpine which is not matching with the version 1.9.15-v13.0.0-rc.0-alpine.

Also the operator pod kept crashing.

2m11s Normal Created pod/voyager-release50-huha-7f46b9fb6f-9p6lx Created container voyager 2m11s Normal Pulled pod/voyager-release50-huha-7f46b9fb6f-9p6lx Container image "appscode/voyager:v13.0.0-rc.0" already present on machine 2m7s Warning AdmissionWebhookNotActivated deployment/voyager-release50-huha timed out waiting for the condition 99s Warning BackOff pod/voyager-release50-huha-7f46b9fb6f-9p6lx Back-off restarting failed container

tamalsaha commented 3 years ago

Hello, We have released https://blog.byte.builders/post/voyager-v2021.09.15/ . Can you please check if this works with cri-o?