search

nephio-project / nephio

Nephio is a Kubernetes-based automation platform for deploying and managing highly distributed, interconnected workloads such as 5G Network Functions, and the underlying infrastructure on which those workloads depend.
Apache License 2.0
106 stars 53 forks source link

kubectl get clusters does not work consistently #327

Open johnbelamaric opened 1 year ago

johnbelamaric commented 1 year ago

The problem is this:

ubuntu@nephio-r1-e2e:~$ kubectl api-resources | grep clusters
clusters                          cl           cluster.x-k8s.io/v1beta1                  true         Cluster
clusters                                       clusterregistry.k8s.io/v1alpha1           true         Cluster
clusterselectors                               configmanagement.gke.io/v1                false        ClusterSelector
workloadclusters                               infra.nephio.org/v1alpha1                 true         WorkloadCluster
dockerclusters                                 infrastructure.cluster.x-k8s.io/v1beta1   true         DockerCluster
ubuntu@nephio-r1-e2e:~$

At some point, the clusterregistry.k8s.io/v1alpha1 started interfering with the cluster.x-k8s.io/v1beta1. Where did the cluster registry one come from? I never saw this until we update the mgmt cluster to a newer K8s version. I don't know if that's because the algorithm in the older k8s version always picked the right "clusters", or because something else changed and we now have the clusterregistry one and we did not before. It needs to be investigated - do we even need the clusterregistry one? How is it getting added to our system?

Most of the e2e scripts have been fixed to account for this, but it will still cause confusion for end users if we do not fix it.

/kind bug

nephio-prow[bot] commented 1 year ago

@johnbelamaric: The label(s) kind/bug cannot be applied, because the repository doesn't have them.

In response to [this](https://github.com/nephio-project/nephio/issues/327): >The problem is this: > >``` >ubuntu@nephio-r1-e2e:~$ kubectl api-resources | grep clusters >clusters cl cluster.x-k8s.io/v1beta1 true Cluster >clusters clusterregistry.k8s.io/v1alpha1 true Cluster >clusterselectors configmanagement.gke.io/v1 false ClusterSelector >workloadclusters infra.nephio.org/v1alpha1 true WorkloadCluster >dockerclusters infrastructure.cluster.x-k8s.io/v1beta1 true DockerCluster >ubuntu@nephio-r1-e2e:~$ >``` > >At some point, the `clusterregistry.k8s.io/v1alpha1` started interfering with the `cluster.x-k8s.io/v1beta1`. Where did the cluster registry one come from? I never saw this until we update the mgmt cluster to a newer K8s version. I don't know if that's because the algorithm in the older k8s version always picked the right "clusters", or because something else changed and we now *have* the clusterregistry one and we *did not* before. It needs to be investigated - do we even *need* the clusterregistry one? How is it getting added to our system? > >Most of the e2e scripts have been fixed to account for this, but it will still cause confusion for end users if we do not fix it. > >/kind bug > Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
electrocucaracha commented 1 year ago

I could be wrong, but it doesn't come from ConfigSync?

johnbelamaric commented 1 year ago

I don't think so. That's more an informational thing in CS for when it is running on clusters that have that installed. Let me grep around.

johnbelamaric commented 1 year ago

Oof, I think you're right:

https://github.com/GoogleContainerTools/kpt-config-sync/blob/v1.13/manifests/cluster-registry-crd.yaml

johnbelamaric commented 1 year ago

Let me look into why that's there. It's probably for ClusterSelectors which we do not use.

johnbelamaric commented 1 year ago

Ok, I filed an issue with CS and also reached out to some folks internally.