Deployment instructions on Kubernetes cluster

[legalgig]

I'm trying to figure out how to deploy this plugin to Kubernetes cluster (K3s). I've already had deployed OKD-Console and it's working fine by itself. I tried to deploy kubevirt plugin using the Container image from quay but all it returns is 404, after a little troubleshooting I figured out that the Dockerfile copies the html files to wrong direction, nginx expects them in /opt/app-root/src but Dockerfile copies it to /usr/share/nginx/html. After copying the files from /usr/share/nginx/html to /opt/app-root/src the Virtualization dropdown shows up on the OKD-Console and it semi-works.

After testing a little I found out:

• Listing anything (VMs, InstanceTypes) doesn't work, it just loads indefinitely
• Creating VMs from YAML works, it redirects me to newly created VM and I can start it and see the output in VNC console
• Templating is dependent on openshift CRDs so it shouldn't be listed on normal kubernetes clusters

Here are my deployment files for both okd-console and kubevirt-plugin kubevirt-ui-plugin

---
apiVersion: v1
kind: Service
metadata:
  name: kubevirt-ui-plugin
  namespace: kubevirt
  labels:    
    app.kubernetes.io/name: kubevirt-ui-plugin
    app.kubernetes.io/instance: kubevirt-ui-plugin
  annotations:
spec:
  type: ClusterIP
  ports:
    - name: http
      port: 8080
      targetPort: http
      protocol: TCP
  selector:    
    app.kubernetes.io/name: kubevirt-ui-plugin
    app.kubernetes.io/instance: kubevirt-ui-plugin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubevirt-ui-plugin
  namespace: kubevirt
  labels:    
    app.kubernetes.io/name: kubevirt-ui-plugin
    app.kubernetes.io/instance: kubevirt-ui-plugin
  annotations:
    reloader.stakater.com/auto: "true"
spec:
  revisionHistoryLimit: 3
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  selector:
    matchLabels:      
      app.kubernetes.io/name: kubevirt-ui-plugin
      app.kubernetes.io/instance: kubevirt-ui-plugin
  template:
    metadata:      
      annotations:
      labels:  
        app.kubernetes.io/name: kubevirt-ui-plugin
        app.kubernetes.io/instance: kubevirt-ui-plugin
    spec:
      restartPolicy: "Always"      
      automountServiceAccountToken: false
      terminationGracePeriodSeconds: 30
      enableServiceLinks: false
      affinity:  
        podAntiAffinity: 
          preferredDuringSchedulingIgnoredDuringExecution: 
            - weight: 100  
              podAffinityTerm:
                topologyKey: kubernetes.io/hostname
                labelSelector:
                  matchLabels:            
                    app.kubernetes.io/name: kubevirt-ui-plugin
                    app.kubernetes.io/instance: kubevirt-ui-plugin
      initContainers:  
      containers:  
        - name: "kubevirt-ui-plugin"
          image: "quay.io/kubevirt-ui/kubevirt-plugin:v4.16.0"
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          volumeMounts:
            - name: tmp
              mountPath: /tmp
      volumes:  
        - name: tmp
          emptyDir: {}
---
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: kubevirt-ui-plugin
  namespace: kubevirt
  labels:    
    app.kubernetes.io/name: kubevirt-ui-plugin
    app.kubernetes.io/instance: kubevirt-ui-plugin
spec:
  endpointSelector:
    matchLabels:      
      app.kubernetes.io/name: kubevirt-ui-plugin
      app.kubernetes.io/instance: kubevirt-ui-plugin

  ingress:

  egress:
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
          rules:
            dns:
              - matchPattern: "*"

okd-console

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: okd-console
  namespace: kube-system
  labels:    
    app.kubernetes.io/name: okd-console
    app.kubernetes.io/instance: okd-console
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: okd-console
  labels:    
    app.kubernetes.io/name: okd-console
    app.kubernetes.io/instance: okd-console
rules:
  - apiGroups:
    - '*'
    resources:
    - '*'
    verbs:
    - '*'
  - nonResourceURLs:
    - '*'
    verbs:
    - '*'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: okd-console
  namespace: kube-system
  labels:    
    app.kubernetes.io/name: okd-console
    app.kubernetes.io/instance: okd-console
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: okd-console
subjects:
  - kind: ServiceAccount
    name: okd-console
    namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
  name: okd-console
  namespace: kube-system
  labels:    
    app.kubernetes.io/name: okd-console
    app.kubernetes.io/instance: okd-console
  annotations:
spec:
  type: ClusterIP
  ports:
    - name: http
      port: 9000
      targetPort: http
      protocol: TCP
  selector:    
    app.kubernetes.io/name: okd-console
    app.kubernetes.io/instance: okd-console
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: okd-console
  namespace: kube-system
  labels:    
    app.kubernetes.io/name: okd-console
    app.kubernetes.io/instance: okd-console
  annotations:
    reloader.stakater.com/auto: "true"
spec:
  revisionHistoryLimit: 3
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  selector:
    matchLabels:      
      app.kubernetes.io/name: okd-console
      app.kubernetes.io/instance: okd-console
  template:
    metadata:      
      annotations:
      labels:  
        app.kubernetes.io/name: okd-console
        app.kubernetes.io/instance: okd-console
    spec:
      restartPolicy: "Always"      
      serviceAccountName: okd-console
      automountServiceAccountToken: true
      terminationGracePeriodSeconds: 30
      enableServiceLinks: false
      affinity:  
        podAntiAffinity: 
          preferredDuringSchedulingIgnoredDuringExecution: 
            - weight: 100  
              podAffinityTerm:
                topologyKey: kubernetes.io/hostname
                labelSelector:
                  matchLabels:            
                    app.kubernetes.io/name: okd-console
                    app.kubernetes.io/instance: okd-console
      securityContext:
        fsGroup: 999
        runAsGroup: 999
        runAsUser: 999
        seccompProfile:
          type: RuntimeDefault
      initContainers:  
      containers:  
        - name: "okd-console"
          image: "quay.io/openshift/origin-console:4.16.0"
          imagePullPolicy: IfNotPresent
          env:
            - name: BRIDGE_USER_AUTH
              value: disabled
            - name: BRIDGE_K8S_MODE
              value: in-cluster
            - name: BRIDGE_PLUGINS
              value: kubevirt-plugin=http://kubevirt-ui-plugin.kubevirt:8080
          ports:
            - name: http
              containerPort: 9000
              protocol: TCP
          volumeMounts:
            - name: tmp
              mountPath: /tmp
            - name: run
              mountPath: /run  
          startupProbe:
            httpGet:
              path: /
              port: http
              scheme: HTTP
            failureThreshold: 30
            timeoutSeconds: 1
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /
              port: http
              scheme: HTTP
            failureThreshold: 3
            timeoutSeconds: 1
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /
              port: http
              scheme: HTTP
            failureThreshold: 3
            timeoutSeconds: 1
            periodSeconds: 10
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            readOnlyRootFilesystem: true
            runAsNonRoot: true
      volumes:  
        - name: tmp
          emptyDir: {}
        - name: run
          emptyDir: {}
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: okd-console
  namespace: kube-system
  labels:    
    app.kubernetes.io/name: okd-console
    app.kubernetes.io/instance: okd-console
  annotations:
    traefik.ingress.kubernetes.io/router.tls: "true"
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.middlewares: "kube-system-okd-console-auth@kubernetescrd"
spec:
  ingressClassName: traefik 
  rules:
    - host: console.domain
      http:
        paths:
          - pathType: ImplementationSpecific
            path: /
            backend:
              service:
                name: okd-console
                port:
                  name: http
---
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: okd-console
  namespace: kube-system
  labels:    
    app.kubernetes.io/name: okd-console
    app.kubernetes.io/instance: okd-console
spec:
  endpointSelector:
    matchLabels:      
      app.kubernetes.io/name: okd-console
      app.kubernetes.io/instance: okd-console

  ingress:
    - fromEndpoints:
        - matchLabels:
            app.kubernetes.io/instance: traefik-traefik
            app.kubernetes.io/name: traefik
            io.kubernetes.pod.namespace: traefik
      toPorts:
        - ports:
            - port: http

  egress:
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
          rules:
            dns:
              - matchPattern: "*"
    - toEntities:
        - kube-apiserver
    - toEndpoints:
      - matchLabels:
          app.kubernetes.io/instance: kubevirt-ui-plugin
          app.kubernetes.io/name: kubevirt-ui-plugin
          io.kubernetes.pod.namespace: kubevirt

btw. This plugin isn't listed anywhere inside kubevirt docs so it was really hard to find it (separate github organization for kubevirt and kubevirt-ui didn't help either).

[qkfrksvl]

any update ?

[haiwu]

seeing the same issue.

[metalice]

@orenc1 can you shade some light here?

[orenc1]

I think @upalatucci has some experience with deploying an openshift console and a plugin on a kubernetes (kind?) cluster.

[qkfrksvl]

I have one question, it's not supported with native kubernetes? We're not using okd and just using native k8s and kubevirt.

[qkfrksvl]

@upalatucci Could you share an idea for native k8s ?

[upalatucci]

Hei @qkfrksvl we use the ConfigMap to override the nginx configuration. You can find the oc-manifest.yaml file in the root directory where we define what we need to install the plugin (with the openshift template format ).

One thing that can be a problem for the deployment is the cert that we use plugin-serving-cert. those certs in openshift are automatically managed but not in k8s

[upalatucci]

@qkfrksvl @legalgig One issue can be that we are using Projects to list everything and not Namespaces. Native k8s do not have Projects. Can you share the errors that you see in the browser console?

[qkfrksvl]

@upalatucci sorry for late.

when I'm trying to open VirtualMachine menu, it was stuck with blank page. console logs shows below two messages.

"Could not create user settings ConfigMap r: Failed to get user data to handle user setting request: Post "https://kubernetes.default.svc/apis/authentication.k8s.io/v1/selfsubjectreviews": tls: failed to verify certificate: x509: certificate signed by unknown authority"

unhandled promise rejection: r: namespaces "openshift-cnv" not found r: namespaces "openshift-cnv" not found

[upalatucci]

hey guys i created a pr: https://github.com/kubevirt-ui/kubevirt-plugin/pull/2227

[qkfrksvl]

wow it seems work well, big thanks @upalatucci

[haiwu]

Yes this is great! The plugin now works for "VirtualMachines". Although if clicking into any VM, its 'Events' and 'Console' tabs are not working for me.

@qkfrksvl : Are they working for you?