search

kubernetes-retired / cluster-api-provider-nested

Cluster API Provider for Nested Clusters
Apache License 2.0
301 stars 67 forks source link

📖 Update docs/README.md with beta clusterctl setup #105

Closed jichenjc closed 3 years ago

jichenjc commented 3 years ago

follow this https://github.com/kubernetes-sigs/cluster-api-provider-nested/tree/main/docs (latest code)

and seems the control pane not startup correctly

kubectl logs capn-nested-control-plane-controller-manager-7fcc8c7bd5-k2wtk -n capn-nested-control-plane-system -c manager

E0609 09:09:54.667549       1 nestedcontrollermanager_controller.go:108] controllers/controlplane/NestedControllerManager "msg"="fail to create NestedControllerManager StatefulSet" "error"="fail to generate the Statefulset object: fail to fetch the default template for the NestedControllerManager StatefulSet: open /component-templates/nested-controllermanager/nested-controllermanager-statefulset-template.yaml: permission denied" "nestedcontrollermanager"={"Namespace":"default","Name":"nestedcontrollermanager-sample"}
E0609 09:09:58.042221       1 nestedapiserver_controller.go:117] controllers/controlplane/NestedAPIServer "msg"="fail to create NestedAPIServer StatefulSet" "error"="fail to generate the Statefulset object: fail to fetch the default template for the NestedAPIServer StatefulSet: open /component-templates/nested-apiserver/nested-apiserver-statefulset-template.yaml: permission denied" "nestedapiserver"={"Namespace":"default","Name":"nestedapiserver-sample"}
E0609 09:10:02.748178       1 nestedetcd_controller.go:119] controllers/controlplane/NestedEtcd "msg"="fail to create NestedEtcd StatefulSet" "error"="fail to generate the Statefulset object: fail to fetch the default template for the NestedEtcd StatefulSet: open /component-templates/nested-etcd/nested-etcd-statefulset-template.yaml: permission denied" "nestedetcd"={"Namespace":"default","Name":"nestedetcd-sample"}

any guide in fixing this? thanks

christopherhein commented 3 years ago

Looks like somewhere the RBAC role aren't being generated properly anymore. I would check to see if the code still has the proper // +kubebuilder go comments on the controlplane/nested/controllers/ directory. This should declare that a statefulset permission is needed.

christopherhein commented 3 years ago

/kind bug

christopherhein commented 3 years ago

/retitle 🐛 Permission Errors trying to create StatefulSet from NCP

christopherhein commented 3 years ago

/assign

christopherhein commented 3 years ago

It appears this happens because we don't have the serviceAccountName in the Deployment specs as I'm currently doing #61 and experiencing the same issue.

jichenjc commented 3 years ago

/reopen

@christopherhein thanks for the PR, however, I updated to latest code , follow the same doc and still get same issue , I am trying to manually apply the rbac and got following "config/rbac/service_account.yaml": namespaces "system" not found which makes me curious seems there is no system ns defined at all?

k8s-ci-robot commented 3 years ago

@jichenjc: Reopened this issue.

In response to [this](https://github.com/kubernetes-sigs/cluster-api-provider-nested/issues/105#issuecomment-858328919): >/reopen > >@christopherhein thanks for the PR, however, I updated and still get same issue , I am trying to > >manually apply the rbac and got following >`"config/rbac/service_account.yaml": namespaces "system" not found` >which makes me curious seems there is no `system` ns defined at all? Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
christopherhein commented 3 years ago

@jichenjc thanks for reopening. Use case wise if I changed this doc to be the published prod images instead of you needing to generate those images from scratch would that suffice for your use case or are you looking for developer docs?

I think we need both but I'm actively working on #54 and which will make this doc even lighter along with that we have published images and manifests as of today. So if you want to just get a working example you can download https://github.com/kubernetes-sigs/cluster-api-provider-nested/releases/download/v0.1.0/cluster-api-provider-nested-components.yaml and apply that to get the CAPN bits deployed and as of today you can download the v0.4 clusterctl binary so that could be updated.

jichenjc commented 3 years ago

@jichenjc thanks for reopening. Use case wise if I changed this doc to be the published prod images instead of you needing to generate those images from scratch would that suffice for your use case or are you looking for developer docs?

I think we need both but I'm actively working on #54 and which will make this doc even lighter along with that we have published images and manifests as of today. So if you want to just get a working example you can download https://github.com/kubernetes-sigs/cluster-api-provider-nested/releases/download/v0.1.0/cluster-api-provider-nested-components.yaml and apply that to get the CAPN bits deployed and as of today you can download the v0.4 clusterctl binary so that could be updated.

thanks for the detailed info , appreciate~ I will re-try based on your comments and get back here, thanks

christopherhein commented 3 years ago

Hopefully that all worked for you, we just merged https://github.com/kubernetes-sigs/cluster-api/pull/4792 so if you want we can repivot this issue to be about updating the docs/README.md to use the published deployment route, wdyt?

jichenjc commented 3 years ago

Thanks for the info, will validate with more tries later on ,currently having following error and will retry after clean up env

# clusterctl init --core cluster-api:v0.4.0-beta.0  --control-plane nested:v0.1.0  --infrastructure nested:v0.1.0 -v 66
No default config file available
Fetching providers
Fetching File="core-components.yaml" Provider="cluster-api" Type="CoreProvider" Version="v0.4.0-beta.0"
Fetching File="bootstrap-components.yaml" Provider="kubeadm" Type="BootstrapProvider" Version="v0.3.19"
Fetching File="control-plane-components.yaml" Provider="nested" Type="ControlPlaneProvider" Version="v0.1.0"
Fetching File="infrastructure-components.yaml" Provider="nested" Type="InfrastructureProvider" Version="v0.1.0"
Fetching File="metadata.yaml" Provider="cluster-api" Type="CoreProvider" Version="v0.4.0-beta.0"
Fetching File="metadata.yaml" Provider="kubeadm" Type="BootstrapProvider" Version="v0.3.19"
Error: current version of clusterctl could install only v1alpha4 providers, detected v1alpha3 for provider bootstrap-kubeadm
sigs.k8s.io/cluster-api/cmd/clusterctl/client/cluster.(*providerInstaller).getProviderContract
        /root/cluster-api-provider-nested/cluster-api/cmd/clusterctl/client/cluster/installer.go:177
sigs.k8s.io/cluster-api/cmd/clusterctl/client/cluster.(*providerInstaller).Validate
        /root/cluster-api-provider-nested/cluster-api/cmd/clusterctl/client/cluster/installer.go:129
sigs.k8s.io/cluster-api/cmd/clusterctl/client.(*clusterctlClient).Init
        /root/cluster-api-provider-nested/cluster-api/cmd/clusterctl/client/init.go:102
sigs.k8s.io/cluster-api/cmd/clusterctl/cmd.runInit
        /root/cluster-api-provider-nested/cluster-api/cmd/clusterctl/cmd/init.go:139
sigs.k8s.io/cluster-api/cmd/clusterctl/cmd.glob..func11
        /root/cluster-api-provider-nested/cluster-api/cmd/clusterctl/cmd/init.go:84
github.com/spf13/cobra.(*Command).execute
        /root/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:852
github.com/spf13/cobra.(*Command).ExecuteC
        /root/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:960
github.com/spf13/cobra.(*Command).Execute
        /root/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:897
sigs.k8s.io/cluster-api/cmd/clusterctl/cmd.Execute
        /root/cluster-api-provider-nested/cluster-api/cmd/clusterctl/cmd/root.go:93
main.main
        /root/cluster-api-provider-nested/cluster-api/cmd/clusterctl/main.go:25
runtime.main
        /usr/local/go/src/runtime/proc.go:225
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1371
christopherhein commented 3 years ago

Since 0.4.0 isn't fully released try adding this file:

→ cat ~/.cluster-api/dev-repository/config.yaml
---
providers:
- name: "cluster-api"
  url: "https://github.com/kubernetes-sigs/cluster-api/releases/v0.4.0-beta.0/core-components.yaml"
  type: "CoreProvider"
- name: "kubeadm"
  url: "https://github.com/kubernetes-sigs/cluster-api/releases/v0.4.0-beta.0/bootstrap-components.yaml"
  type: "BootstrapProvider"

And then try using:

→ clusterctl init --core cluster-api:v0.4.0-beta.0  --control-plane nested:v0.1.0  --infrastructure nested:v0.1.0 --config ~/.cluster-api/dev-repository/config.yaml

Also, this route still needs to be build off main for CAPI.

If that doesn't work check out https://cluster-api.sigs.k8s.io/clusterctl/developers.html to see if there are more steps missing. I was able to as long as I added the v0.4.0 binaries to the dev-repository/config.

@vincepri potentially you have a better way of using v0.4.0 binaries for CAPI from clusterctl.

christopherhein commented 3 years ago

/retitle 📖 Update docs/README.md with beta clusterctl setup

christopherhein commented 3 years ago

/unassign

christopherhein commented 3 years ago

This should be much easier now:

# Create Cluster
kind create cluster --name=capn

# Get unreleased clusterctl
git clone git@github.com:kubernetes-sigs/cluster-api.git
cd cluster-api
make clusterctl

# Create Dev Repo
mkdir -p ~/.cluster-api/dev-repository/

# Update config for CAPI Beta
cat <<EOF >>~/.cluster-api/dev-repository/config.yaml
---
providers:
- name: "cluster-api"
  url: "https://github.com/kubernetes-sigs/cluster-api/releases/v0.4.0-beta.0/core-components.yaml"
  type: "CoreProvider"
- name: "kubeadm"
  url: "https://github.com/kubernetes-sigs/cluster-api/releases/v0.4.0-beta.0/bootstrap-components.yaml"
  type: "BootstrapProvider"
EOF

# Init Cluster API
./bin/clusterctl init --core cluster-api:v0.4.0-beta.0  --control-plane nested:v0.1.0  --infrastructure nested:v0.1.0 --config ~/.cluster-api/dev-repository/config.yaml

# Set Name
export CLUSTER_NAME=cluster-sample

./bin/clusterctl generate cluster ${CLUSTER_NAME} --infrastructure=nested:v0.1.0 | kubectl apply -f -

# Get kubeconfig
./bin/clusterctl get kubeconfig ${CLUSTER_NAME} > kubeconfig

# Port Forward Cluster
kubectl port-forward svc/${CLUSTER_NAME}-apiserver 6443:6443

# Update /etc/hosts
127.0.0.1 ${CLUSTER_NAME}-apiserver

# Get resources; Profit 🎉 
kubectl --kubeconfig kubeconfig get all -A
christopherhein commented 3 years ago

/help

k8s-ci-robot commented 3 years ago

@christopherhein: This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed by commenting with the /remove-help command.

In response to [this](https://github.com/kubernetes-sigs/cluster-api-provider-nested/issues/105): >/help Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
christopherhein commented 3 years ago

We're looking for someone to use the information from https://github.com/kubernetes-sigs/cluster-api-provider-nested/issues/105#issuecomment-860072112 to update docs/README.md so that the new quick start guide uses the current unreleased clusterctl and is more reliable.

jichenjc commented 3 years ago

/assign

I am working on this ,will provide an update soon

jichenjc commented 3 years ago

I can move to here already, so should be good now 

root@jitest19:~/cluster-api-provider-nested# kubectl --kubeconfig kubeconfig get all -A
NAMESPACE   NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
default     service/kubernetes   ClusterIP   10.32.0.1    <none>        443/TCP   2d1h