search

kubernetes-retired / cluster-api-provider-nested

Cluster API Provider for Nested Clusters
Apache License 2.0
301 stars 67 forks source link

VirtualCluster fail to create with "cannot find sts/apiserver in ns" on minikube running k8s 1.20.2 #198

Closed sriram-kannan-infoblox closed 2 years ago

sriram-kannan-infoblox commented 3 years ago

The VirtualCluster fail to create the apiserver and the apiserver-0 is in the container creating state kubectl-vc create -f virtualcluster_1_nodeport.yaml -o vc.kubeconfig 2021/07/28 08:20:03 etcd is ready cannot find sts/apiserver in ns default-e4d075-vc-sample-1: default-e4d075-vc-sample-1/apiserver is not ready in 120 seconds

kubectl get po -n default-e4d075-vc-sample-1 NAME READY STATUS RESTARTS AGE apiserver-0 0/1 ContainerCreating 0 46m etcd-0 1/1 Running 1 47m

What steps did you take and what happened: Followed the step as per the virtual cluster walkthrough demo and all the steps were successful till the Create VirtualCluster.
https://github.com/kubernetes-sigs/cluster-api-provider-nested/blob/main/virtualcluster/doc/demo.md

During the Create VirtualCluster the etcd came up fine but the apiserver-0 stayed in the ContainerCreating state.

What did you expect to happen: Expected to see the apiserver and controller-manager to be in running state.

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

jichenjc commented 3 years ago

apiserver-0 0/1 ContainerCreating 0 46m

what's the reason for this creating status? I ecountered this before due to docker pull limit but seems you are not, how about check the reason for why container creating in 46 min? e.g describe the container for additional info?

sriram-kannan-infoblox commented 3 years ago

Good point, i checked the pod and the failure is due to Warning FailedMount 5m43s (x50 over 91m) kubelet, minikube MountVolume.SetUp failed for volume "front-proxy-ca" : secret "front-proxy-ca" not found Warning FailedMount 33s (x12 over 86m) kubelet, minikube Unable to attach or mount volumes: unmounted volumes=[front-proxy-ca], unattached volumes=[apiserver-ca front-proxy-ca root-ca serviceaccount-rsa default-token-8xxsj]: timed out waiting for the condition

jichenjc commented 3 years ago

ok, looks like the ca has issue and to my limited knowledge those ca are created by CAPN directly @christopherhein any insight for further trouble shooting on this?

sriram-kannan-infoblox commented 3 years ago

Looks to me like Minikube doesn't create the certificates in the /etc/kubernetes/pki, unlike kubeadm. Do we need frontend-proxy for the virtualcluster to work?

gyliu513 commented 3 years ago

@sriram-kannan-infoblox this was introduced in https://github.com/kubernetes-sigs/cluster-api-provider-nested/pull/167 , can you make sure you are using latest code and build all of the images/binaries from the latest code?

sriram-kannan-infoblox commented 3 years ago

Hi @gyliu513, I am only following the steps as per the walkthrough demo below and haven't tried to build the image at all. https://github.com/kubernetes-sigs/cluster-api-provider-nested/blob/main/virtualcluster/doc/demo.md

My query was do we need the aggregate API for virtual cluster to work? I can go back few commits and try out virtual cluster without aggregate API in minikube provided aggregate API change is not a breaking change.

My plan is to try out virtual cluster in minikube first and then take it to actual cluster.

Thanks

gyliu513 commented 3 years ago

@sriram-kannan-infoblox as a workaround, please remove the aggregate api support, check https://github.com/kubernetes-sigs/cluster-api-provider-nested/pull/167/files for how to remove, you only need to update the statefulset for apiserver.

The https://github.com/kubernetes-sigs/cluster-api-provider-nested/blob/main/virtualcluster/doc/demo.md need some update, as it is not using the latest image.

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

jasonliu747 commented 2 years ago

The https://github.com/kubernetes-sigs/cluster-api-provider-nested/blob/main/virtualcluster/doc/demo.md need some update, as it is not using the latest image.

Hi @gyliu513 do you have any update on this demo.md? Thanks.

jasonliu747 commented 2 years ago

/remove-lifecycle stale

gyliu513 commented 2 years ago

The https://github.com/kubernetes-sigs/cluster-api-provider-nested/blob/main/virtualcluster/doc/demo.md need some update, as it is not using the latest image.

@vincent-pli I recalled you opened another issue to track this? What is the status?

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

jasonliu747 commented 2 years ago

/remove-lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot commented 2 years ago

@k8s-triage-robot: Closing this issue.

In response to [this](https://github.com/kubernetes-sigs/cluster-api-provider-nested/issues/198#issuecomment-1193077244): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues and PRs according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue or PR with `/reopen` >- Mark this issue or PR as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.