kateyes-client:v0.0.1-beta-006 does not work on an on premises kubernetes cluster

[gbonazzoli]

Hi,

I've installed it on my on premises kubernetes 1.24 cluster, without the ingress controller.

Everything seems OK:

root@image-builder:~/k8s-kateyes/install# kubectl get all -n kateyes 
NAME                                             READY   STATUS    RESTARTS   AGE
pod/kateyes-client-deployment-66784c6cb4-69qmq   1/1     Running   0          7m53s
pod/kateyes-server-deployment-6bf458fff4-84f5z   1/1     Running   0          7m53s

NAME                         TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)        AGE
service/kateyes-client-svc   LoadBalancer   10.101.37.173    172.16.5.203   80:31138/TCP   7m53s
service/kateyes-server-svc   ClusterIP      10.108.148.206   <none>         80/TCP         7m53s

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kateyes-client-deployment   1/1     1            1           7m53s
deployment.apps/kateyes-server-deployment   1/1     1            1           7m53s

NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/kateyes-client-deployment-66784c6cb4   1         1         1       7m54s
replicaset.apps/kateyes-server-deployment-6bf458fff4   1         1         1       7m54s

I've just modified the service deployment.apps/kateyes-client-deployment in LoadBalancer in order to get into the app.

The app works, but i shows me only zeros.

[nagarkarv]

Hi gbonazzoli,

Thanks for downloading.

Few Observations:

• The refresh button on right top corner is "OFF" which suggest the client is unable to connect to server
• Due to this you may not get the list of namespaces as well.
• Explorer tab would also display the default chart.

The beta version is based on a client server model and will need to have an ingress-controller connected for a successful communication between both. I would suggest install the ingress controller (or connect to any existing one) and update the ingress as per the readme in the repository. Give it a try and let me know

You can also spin up GKE/AKS or EKS and try it out. This has mainly been tested on GKE for now.

Do let me know if you get any further issue and will be glad to help.

[mjbright]

I'm having the same problem with an on-prem cluster - I'd really encourage you to provide a working example for that.

I have access to the KatEyes UI but there is a problem to connect to the backend services.

For the moment I've CrashLoopBackoffs on the ingress controller, and yet when I describe the Ingress rules (I created a modified NodePort service for the kateyes-server-svc) both rules have a backend endpoint.

Note: I'm running Kubernetes 1.24, I did wonder if the changes to ServiceAccount tokens might be causing a problem here.

I'll post some details below.

The good:

Name:             kateyes-client-ingress
Labels:           <none>
Namespace:        kateyes
Address:
Ingress Class:    <none>
Default backend:  <default>
Rules:
  Host        Path  Backends
  ----        ----  --------
  *
              /   kateyes-client-svc:80 (192.168.190.89:3000)
Annotations:  kubernetes.io/ingress.class: kateyes-nginx
              nginx.ingress.kubernetes.io/rewrite-target: /
Events:       <none>

Name:             kateyes-server-ingress
Labels:           <none>
Namespace:        kateyes
Address:
Ingress Class:    <none>
Default backend:  <default>
Rules:
  Host        Path  Backends
  ----        ----  --------
  *
              /api(/|$)(.*)   kateyes-server-svc:80 (192.168.80.225:5000)
Annotations:  kubernetes.io/ingress.class: kateyes-nginx
              nginx.ingress.kubernetes.io/rewrite-target: /$2
Events:       <none>

The bad: in the Kateyes UI under Ingress "Explorer" it says clearly Error connecting to Kateyes Server

The ugly, logs from the crashing Ingress Controller:

kubectl -n kateyes logs ingress-nginx-controller-6757487569-g7hsf
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v0.44.0
  Build:         f802554ccfadf828f7eb6d3f9a9333686706d613
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.6

-------------------------------------------------------------------------------

I0621 15:25:00.262081       7 flags.go:208] "Watching for Ingress" class="kateyes-nginx"
W0621 15:25:00.262126       7 flags.go:211] Only Ingresses with class "kateyes-nginx" will be processed by this Ingress controller
W0621 15:25:00.262450       7 client_config.go:614] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0621 15:25:00.262616       7 main.go:241] "Creating API client" host="https://10.96.0.1:443"
I0621 15:25:00.275481       7 main.go:285] "Running in Kubernetes cluster" major="1" minor="24" git="v1.24.2" state="clean" commit="f66044f4361b9f1f96f0053dd46cb7dce5e990a8" platform="linux/amd64"
I0621 15:25:00.446379       7 main.go:105] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0621 15:25:00.448389       7 main.go:115] "Enabling new Ingress features available since Kubernetes v1.18"
W0621 15:25:00.449977       7 main.go:127] No IngressClass resource with name kateyes-nginx found. Only annotation will be used.
I0621 15:25:00.464498       7 ssl.go:532] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0621 15:25:00.503128       7 nginx.go:254] "Starting NGINX Ingress controller"
I0621 15:25:00.515820       7 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kateyes", Name:"ingress-nginx-controller", UID:"f371142b-f770-4c9e-a262-630151ee018a", APIVersion:"v1", ResourceVersion:"259430", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap kateyes/ingress-nginx-controller
E0621 15:25:01.607147       7 reflector.go:138] k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
E0621 15:25:02.508671       7 reflector.go:138] k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
E0621 15:25:04.173003       7 reflector.go:138] k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
E0621 15:25:08.942655       7 reflector.go:138] k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
E0621 15:25:18.237617       7 reflector.go:138] k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
E0621 15:25:39.799889       7 reflector.go:138] k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
I0621 15:25:58.910453       7 main.go:187] "Received SIGTERM, shutting down"
I0621 15:25:58.910537       7 nginx.go:372] "Shutting down controller queues"
E0621 15:25:58.910854       7 store.go:178] timed out waiting for caches to sync
I0621 15:25:58.910898       7 nginx.go:296] "Starting NGINX process"
I0621 15:25:58.911307       7 leaderelection.go:243] attempting to acquire leader lease kateyes/ingress-controller-leader-kateyes-nginx...
I0621 15:25:58.911891       7 queue.go:78] "queue has been shutdown, failed to enqueue" key="&ObjectMeta{Name:initial-sync,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[]OwnerReference{},Finalizers:[],ClusterName:,ManagedFields:[]ManagedFieldsEntry{},}"
I0621 15:25:58.911921       7 nginx.go:316] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I0621 15:25:58.927134       7 status.go:84] "New leader elected" identity="ingress-nginx-controller-6757487569-rq6vj"
I0621 15:25:58.934881       7 status.go:131] "removing value from ingress status" address=[]
I0621 15:25:58.934944       7 nginx.go:380] "Stopping admission controller"
I0621 15:25:58.934981       7 nginx.go:388] "Stopping NGINX process"
E0621 15:25:58.935012       7 nginx.go:319] "Error listening for TLS connections" err="http: Server closed"
2022/06/21 15:25:58 [notice] 39#39: signal process started
I0621 15:25:59.939522       7 nginx.go:401] "NGINX process has stopped"
I0621 15:25:59.939541       7 main.go:195] "Handled quit, awaiting Pod deletion"
I0621 15:26:09.940207       7 main.go:198] "Exiting" code=0

and kubectl -n kateyes get all output:

NAME                                             READY   STATUS             RESTARTS         AGE
pod/ingress-nginx-admission-create-j2fp9         0/1     Completed          0                156m
pod/ingress-nginx-controller-6757487569-g7hsf    0/1     CrashLoopBackOff   25 (4m30s ago)   84m
pod/kateyes-client-deployment-6d45f8dfc5-n24ps   1/1     Running            0                156m
pod/kateyes-server-deployment-b949fb949-r5767    1/1     Running            0                156m

NAME                                         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/ingress-nginx-controller             LoadBalancer   10.96.92.73     <pending>     80:30723/TCP,443:30170/TCP   156m
service/ingress-nginx-controller-admission   ClusterIP      10.100.223.70   <none>        443/TCP                      156m
service/kateyes-client-deployment            NodePort       10.97.223.161   <none>        3000:30940/TCP               79m
service/kateyes-client-svc                   NodePort       10.99.168.118   <none>        3000:31037/TCP               57m
service/kateyes-server-svc                   ClusterIP      10.102.16.201   <none>        80/TCP                       156m

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ingress-nginx-controller    0/1     1            0           156m
deployment.apps/kateyes-client-deployment   1/1     1            1           156m
deployment.apps/kateyes-server-deployment   1/1     1            1           156m

NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/ingress-nginx-controller-6757487569    1         1         0       156m
replicaset.apps/kateyes-client-deployment-6d45f8dfc5   1         1         1       156m
replicaset.apps/kateyes-server-deployment-b949fb949    1         1         1       156m

NAME                                       COMPLETIONS   DURATION   AGE
job.batch/ingress-nginx-admission-create   1/1           9s         156m
job.batch/ingress-nginx-admission-patch    0/1           156m       156m

[nagarkarv]

Hi Mike,

Thanks for your interest and trying out kateyes.

Considering the issue with ingresses, I have now released a new standalone version 0.0.7 which does not need ingress controllers.

https://vikram-nagarkar.medium.com/kateyes-beta-with-minikube-support-870482f343dd

Please follow the steps

• Delete the old installations- kubectl delete -f ./install/.
• Pull the latest code from this repo
• Install the standalone version - kubectl apply -f ./install/standalone/.
• Access kateyes using the load balancer - no ingresses required. It only requires port 3000 to be open for communications

Do let me know if successful or otherwise

You can also try installing on minikube to start with as version 0.0.7 now support minikube. Instructions in readme of the latest codebase.

Thanks.

[gbonazzoli]

Well done, @nagarkarv 0.0.7 standalone now works like a charm.

root@image-builder:~/k8s-kateyes#` k get all -n kateyes-sa
NAME                                        READY   STATUS    RESTARTS   AGE
pod/kateyes-sa-deployment-bb56fc677-9lqq5   1/1     Running   0          8m27s

NAME                     TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                       AGE
service/kateyes-sa-svc   LoadBalancer   10.110.209.153   172.16.5.203   80:30507/TCP,8000:32241/TCP   8m27s

NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kateyes-sa-deployment   1/1     1            1           8m27s

NAME                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/kateyes-sa-deployment-bb56fc677   1         1         1       8m27s

[nagarkarv]

Thanks @gbonazzoli . Good to know that it worked .

Do explore it further and let me know any feedback.

[mjbright]

Thanks for pointing out the standalone subdir.

Hmm ... I'm still missing something I'm afraid.

I can curl to ports 3000, 5000 on the Pod I can curl to ports 80, 8000 on the ClusterIP Service I can curl to private or publicIP on the NodePorts (32385 and 31247 below) Note: I've since disabled those ports on my public address. I suppose that it's not actually necessary.

but the dashboard (ing/ Explorer view) still shows me "Error connecting Kateyes server".

Is this trying to connect to some external Kateyes server ? It looks like my on-prem Proxmox server has some external DNS issues ... could this be the problem? ( I manually added nameserver 8.8.8.8 to /etc/resolv.conf in the Pod to be able to nslookup google.com)

kubectl -n kateyes-sa get all -o wide
NAME                                        READY   STATUS    RESTARTS   AGE   IP               NODE   NOMINATED NODE   READINESS GATES
pod/kateyes-sa-deployment-bb56fc677-2kxxc   1/1     Running   0          46m   192.168.80.230   w2     <none>           <none>

NAME                     TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                       AGE   SELECTOR
service/kateyes-sa-svc   LoadBalancer   10.106.193.97   82.66.73.147   80:32385/TCP,8000:31427/TCP   46m   app=kateyes-sa-deployment

NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                               SELECTOR
deployment.apps/kateyes-sa-deployment   1/1     1            1           46m   kateyes-sa   kateyes/kateyes-sa:v0.0.1-beta-007   app=kateyes-sa-deployment

NAME                                              DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES                               SELECTOR
replicaset.apps/kateyes-sa-deployment-bb56fc677   1         1         1       46m   kateyes-sa   kateyes/kateyes-sa:v0.0.1-beta-007   app=kateyes-sa-deployment,pod-template-hash=bb56fc677

[nagarkarv]

Hi @mjbright ,

Port 8000 is actually needed to be available and open for the LB since the UI needs to bypass the ingress and connect to the server process on that port. The port cannot be changed for now but I will put that into the backlog for future release.

service/kateyes-sa-svc LoadBalancer 10.106.193.97 82.66.73.147 80:32385/TCP,8000:31427/TCP

The server is not an external server but a separate process in the application itself.

Please open the port and given it a try.

[mjbright]

OK, I finally got it ... no need for my public-ip, so no need to open any ports externally.

I just needed to add my private (on-prem) node ips to the externalIPs of the loadbalancer to make port 8000 accessible on them.

Thanks for the pointer !

[nagarkarv]

That's great.

Do let me know how you find it and any feedback that will be helpful to enhance the product.

Your support is greatly appreciated.