search

kubernetes-sigs / sig-windows-dev-tools

This is a batteries included local development environment for Kubernetes on Windows.
Apache License 2.0
79 stars 46 forks source link

Switch to custom Vagrant boxes and Magefiles instead of Makefile #264

Closed mloskot closed 1 year ago

mloskot commented 1 year ago

Features

This PR includes two features:

  1. These are experimental Vagrant boxes which attempt to address issues like WinRM not starting, and boot timeouts, OS and VirtualBox guest additions update, and enable/disable VirtualBox features explicitly (for documentation, control, easier troubleshooting). These boxes have proved to be a working solution, and significantly speed up cluster provisioning for testing. Check the boxes repository for details: https://github.com/mloskot/sig-windows-dev-tools-boxes

  2. Add Magefiles - https://magefile.org - as functional equivalent to the existing Makefile, but as a portable solution for all platforms supported by Go. Magefiles remove requirement of use of WSL as a proxy providing GNU Make on Windows host, aks true windows support.

Tester's Quickstart (with Mage)

TODO: This guide could be copied into the README.md before merge

Warning: Currently tested on Windows host, but this procedure should (be made to) work on Linux, Mac OS too

  1. Install latest VirtualBox 7.0, latest Vagrant, Go 1.20 and latest Mage. See updated README.md
  2. Switch your local repo to this PR: gh pr checkout 264
  3. Optionally, copy variables.yaml to variables.local.yaml and modify variables.local.yaml as you desire, for example:
    • tweak RAM and CPUs for machines
    • bump Kubernetes to 1.27
    • bump Calico to 3.25.1
    • bump containerd to 1.7.0
  4. Launch PowerShell or Command Prompt and run mage -l to list all available Mage targets (as check mage works for you)
  5. Run mage all | tee m.log or just mage | tee m.log to download required Kubernetes software, download Vagrant boxes (1 and 2), run Vagrant to provision the two-node cluster, run smoke tests (it is known the smoke tests may fail).

  6. Run mage status in separate console. As soon as Vagrant completes provisioning the two VM-s, open new console and start running mage status every couple of minutes to watch the cluster status. The mage status is a convenient proxy to execute sequence of

    vagrant status
vagrant ssh controlplane -c 'kubectl get nodes'
vagrant ssh controlplane -c 'kubectl get -A pods'

    Alternatively, you can run vagrant ssh controlplane and inspect the cluster with kubectl.

    Alternatively, you can download kubeconfig form the controlplane node to run any Kubernetes client directly from the Windows host:

    vagrant plugin install vagrant-scp
vagrant scp controlplane:~/.kube/config ./swdt-kubeconfig
kubectl --kubeconfig=./swdt-kubeconfig get nodes
kubectl --kubeconfig=./swdt-kubeconfig get -A pods
...
  7. Report back your results

My initial run logs https://gist.github.com/mloskot/2670e7e7331f90e390eb5a750de7bf78

pellared commented 1 year ago

I encourage also take a look at https://github.com/goyek/goyek

Here is a few reasons why it was created https://github.com/goyek/goyek#mage. The API is inspired by testing, cobra, flag, http packages so it should be easier to adopt for Go devs (and more idiomatic).

mloskot commented 1 year ago

@pellared

I encourage also take a look at https://github.com/goyek/goyek

The goal of this PR is to not to discuss and decide what builder to pick. It will/may be discussed in future (/cc @jayunit100) The goal here is to test VirtualBox-based setup on Windows host without WSL, and mage is just convenient for the goal at hand. That's it. The thread of this PR is not the right place for exchanges on why scripting tasks with A is superior over B and vice versa.

pellared commented 1 year ago

The goal of this PR is to not to discuss and decide what builder to pick.

I find it normal in code reviews is to discuss how something are implemented. I just wanted to give you feedback and share with you an alternative way to accomplish the same goal. I am not requesting any changes 🤷

jayunit100 commented 1 year ago

can you re-target the PR to the to the main-windows-native branch ? that way we have

mloskot commented 1 year ago

@jayunit100

can you re-target the PR to the to the main-windows-native branch ?

Sure. Can you create main-windows-native branch from current master? Then I will be able to switch the PR-s base from master to main-windows-native.

that way we have

  • main-qemu for qemu m1 folks and
  • main-windows-native for windows laptops

Good idea!

mloskot commented 1 year ago

To Ubuntu 22.04 or not, and Ubuntu 20.04 only

In https://github.com/kubernetes-sigs/sig-windows-dev-tools/pull/253#issuecomment-1525271825 @claudiubelu mentioned problems with Kubernetes on Ubuntu 22.04. So, in order to enable ourselves here with variety of test boxes, I've created Ubuntu 20.04 box based on roboxes/ubuntu2004 - https://app.vagrantup.com/mloskot/boxes/sig-windows-dev-tools-ubuntu-2004

So, SWDT Vagrantfile can be modified for local testing or perhaps box could be declared in variables.yaml. Then, testers can switch between boxes using user-specific non-versioned variables.local.yaml which will be picked automatically by Vagrantfile (thanks to mage).

By the way of answering @jayunit100 's comment in https://github.com/kubernetes-sigs/sig-windows-dev-tools/pull/264#discussion_r1179569876, I've realised that I should have named all my boxes with -test-box suffix to make it clear that here we are not proposing to switch SWDT over to mloskot/sig-windows-dev-tools-* boxes for good, but temporarily for period of this testing phase.

jayunit100 commented 1 year ago

pushed a branch you can PR against https://github.com/kubernetes-sigs/sig-windows-dev-tools/tree/master-windows-native

jayunit100 commented 1 year ago

/lgtm /approve lets merge, i retargetted the branch for u

k8s-ci-robot commented 1 year ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jayunit100, mloskot

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/kubernetes-sigs/sig-windows-dev-tools/blob/master-windows-native/OWNERS)~~ [jayunit100] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment