search

vmware-archive / kops

Kubernetes Operations (kops) - Production Grade K8s Installation, Upgrades, and Management
Apache License 2.0
3 stars 3 forks source link

Basic end to end test of creating k8s cluster on vsphere #28

Closed msterin closed 7 years ago

msterin commented 7 years ago

We need to have a nightly run of deplpying K8S cluster with vmware/kops code. The steps would be (per https://github.com/vmware/kops/blob/vsphere-develop-2/docs/development/vsphere-dev.md)

  1. git clone
  2. build
  3. run kops command

Pre-req: CoreDNS VM.

Please sync on Telegram or in person with @prashima or @msterin or @abrarshivani or @luomiao

divyenpatel commented 7 years ago

Job is available at http://cna-storage-jenkins.eng.vmware.com:8080/job/vsphere-kops/

This job deploys Kubernetes cluster using locally buit kops from https://github.com/vmware/kops.git - Branch vsphere-develop-2

I can see VMs created on the VC

$ govc ls /vcqaDC/vm
/vcqaDC/vm/nodes.kubernetes.skydns.local2
/vcqaDC/vm/nodes.kubernetes.skydns.local1
/vcqaDC/vm/ubuntu_16_04
/vcqaDC/vm/master-us-west-2a.masters.kubernetes.skydns.local1
/vcqaDC/vm/coredns.ova
/vcqaDC/vm/Discovered virtual machine

currently job is failing with Error exec: "genisoimage": executable file not found in $PATH

Log: http://cna-storage-jenkins.eng.vmware.com:8080/job/vsphere-kops/2/consoleText

I think, we need genisoimage command in the container image we are executing command from?

luomiao commented 7 years ago

Yes genisoimage is needed on the machine where "kops create cluster" commands are executed.

divyenpatel commented 7 years ago

Finally cluster is deployed after all dependencies added to the CI container image. http://cna-storage-jenkins.eng.vmware.com:8080/job/vsphere-kops/4/consoleText

I0405 01:54:49.279724     279 executor.go:91] Tasks: 32 done / 32 total; 0 can run
I0405 01:54:49.279764     279 dns.go:141] Pre-creating DNS records
I0405 01:54:49.279847     279 coredns.go:66] Using CoreDNS DNS provider
I0405 01:54:50.040988     279 update_cluster.go:209] Exporting kubecfg for cluster
Kops has set your kubectl context to kubernetes.skydns.local

Cluster is starting.  It should be ready in a few minutes.

Suggestions:
 * validate cluster: kops validate cluster
 * list nodes: kubectl get nodes --show-labels
 * ssh to the master: ssh -i ~/.ssh/id_rsa admin@api.kubernetes.skydns.local
 * read about installing addons: https://github.com/kubernetes/kops/blob/master/docs/addons.md

Stopping Docker container after build completion
Stopping all containers
Finished: SUCCESS

Questions:

  1. Approximately how much time we need to wait after we see 'Cluster is starting. It should be ready in a few minutes.'
  2. I see we have 1 CPU and 1024 MB Memory set for all nodes. Is there a config parameter, we can set to increase cpu and memory for VMs? or we can just set desired config on the template file?
luomiao commented 7 years ago

Great!

For your questions:

  1. To be safe, 10 mins.
  2. This should be adjusted directly to the template file.
prashima commented 7 years ago

This issue will be marked as closed when:

prashima commented 7 years ago

@divyenpatel can you please update, if this issue can be closed now? As we discussed offline, kops is generating the config file that can be used by kubectl. Hopefully that will help in resolving two issues discussed in previous comment.

Please note that your jenkins script will have to talk to api.internal.clustername.skydns.local and not api.clustername.skydns.local, because of this issue https://github.com/vmware/kops/issues/39

divyenpatel commented 7 years ago

http://cna-storage-jenkins.eng.vmware.com:8080/job/vsphere-kops/ is updated to perform following additional tasks after cluster is deployed.

divyenpatel commented 7 years ago

Job is ready at - http://cna-storage-jenkins.eng.vmware.com:8080/job/vsphere-kops/ Verified Job is building kops, deploying kubernetes cluster, wait until all nodes are up, and then execute e2e test case.

closing this task.