loft-sh / vcluster

vCluster - Create fully functional virtual Kubernetes clusters - Each vcluster runs inside a namespace of the underlying k8s cluster. It's cheaper than creating separate full-blown clusters and it offers better multi-tenancy and isolation than regular namespaces.
https://www.vcluster.com
Apache License 2.0
6.93k stars 428 forks source link

Services lose "endpoints" #1979

Open CiraciNicolo opened 4 months ago

CiraciNicolo commented 4 months ago

What happened?

With different Ingress Controller, such as Traefik and Nginx, the service used to route traffic "lose" endpoints. This results in Ingresses not working:

Service "harbor-system/harbor-portal" does not have any active Endpoint

What did you expect to happen?

Services should not lose endpoints

How can we reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

No response

Host cluster Kubernetes version

Server Version: version.Info{Major:"1", Minor:"29", GitVersion:"v1.29.0", GitCommit:"3f7a50f38688eb332e2a1b013678c6435d539ae6", GitTreeState:"clean", BuildDate:"2023-12-13T08:45:03Z", GoVersion:"go1.21.5", Compiler:"gc", Platform:"linux/amd64"}

vcluster version

vcluster version 0.19.6

VCluster Config

---
sync:
  persistentvolumes:
    enabled: true
  pods:
    ephemeralContainers: true
    status: true
  secrets:
    all: true
  configmaps:
    all: true
  nodes:
    nodeSelector: sync=true
    enabled: true
    enableScheduler: true

syncer:
  replicas: 3

etcd:
  replicas: 3

coredns:
  replicas: 3
  config: |
    .:1053 {
      errors
      health
      ready
      rewrite name regex .*\.nodes\.vcluster\.com kubernetes.default.svc.cluster.local
      kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
      }
      hosts /etc/NodeHosts {
          ttl 60
          reload 15s
          fallthrough
      }
      prometheus :9153
      forward . 1.1.1.1
      cache 30
      loop
      reload
      loadbalance
      whoami
    }

    import /etc/coredns/custom/*.server

isolation:
  enabled: true

  podSecurityStandard: baseline

  resourceQuota:
    enabled: true
    quota:
      requests.cpu: 10
      requests.memory: 20Gi
      requests.storage: 100Gi
      requests.ephemeral-storage: 100Gi
      limits.cpu: 40
      limits.memory: 40Gi
      limits.ephemeral-storage: 300Gi
      services.nodeports: 0
      services.loadbalancers: 0
      count/endpoints: 40
      count/pods: 40
      count/services: 40
      count/secrets: 100
      count/configmaps: 100
      count/persistentvolumeclaims: 20
    scopeSelector:
      matchExpressions:
    scopes:

  networkPolicy:
    enabled: false

telemetry:
  disabled: true
deniseschannon commented 3 months ago

What command do you use to start your vcluster?

CiraciNicolo commented 3 months ago

Hi! The one from the documentation:

vcluster create foo -f foo.yaml --distro k8s --connect=false -n foo