Long Term: support libvirt

dougsland commented 2 years ago

Would be nice to have support for libvirt/kvm/virt-manager/virsh for building the environment.

jayunit100 commented 2 years ago

/good-first-issue

k8s-ci-robot commented 2 years ago

@jayunit100: This request has been marked as suitable for new contributors.

Guidelines

Please ensure that the issue body includes answers to the following questions:

Why are we solving this issue?
To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
Does this issue have zero to low barrier of entry?
How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed by commenting with the /remove-good-first-issue command.

In response to [this](https://github.com/kubernetes-sigs/sig-windows-dev-tools/issues/157): >/good-first-issue Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

jayunit100 commented 2 years ago

/reopen

k8s-ci-robot commented 2 years ago

@jayunit100: Reopened this issue.

In response to [this](https://github.com/kubernetes-sigs/sig-windows-dev-tools/issues/157#issuecomment-1019507082): >/reopen > Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue or PR with /reopen
Mark this issue or PR as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot commented 2 years ago

@k8s-triage-robot: Closing this issue.

In response to [this](https://github.com/kubernetes-sigs/sig-windows-dev-tools/issues/157#issuecomment-1163351680): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues and PRs according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue or PR with `/reopen` >- Mark this issue or PR as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

knabben commented 2 years ago

/remove-lifecycle rotten /reopen

k8s-ci-robot commented 2 years ago

@knabben: Reopened this issue.

In response to [this](https://github.com/kubernetes-sigs/sig-windows-dev-tools/issues/157#issuecomment-1329931211): >/remove-lifecycle rotten >/reopen Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

knabben commented 2 years ago

A few advances on this front using vagrant-libvirt

Vagrant.configure("2") do |config|
  config.vm.provider :libvirt do |libvirt|
    libvirt.driver = "qemu"
    libvirt.cpus = 4
    libvirt.memory = 4096
    libvirt.disk_device = "sda"
    libvirt.disk_bus = "sata"
    libvirt.machine_arch = "x86_64"
    libvirt.machine_type= "pc-q35-5.2"
    libvirt.nic_model_type = "e1000e"
    libvirt.hyperv_feature :name => 'relaxed', :state => 'on'
    libvirt.hyperv_feature :name => 'vapic', :state => 'on'
    libvirt.hyperv_feature :name => 'spinlocks', :state => 'on', :retries => '8191'
  end

  config.vm.define :win2k19 do |win2k19|
    win2k19.vm.network :private_network, :ip => "10.20.30.10"
    win2k19.vm.box = "sig-windows-dev-tools/windows-2019"
    win2k19.vm.communicator = "winrm"
    win2k19.winrm.username = "vagrant"
    win2k19.winrm.password = "vagrant"
  end
end

The box was converted to qcow2 and Vagrant has the libvirt provider image to download https://app.vagrantup.com/sig-windows-dev-tools/boxes/windows-2019

aravindhp commented 1 year ago

I was able to bring up the controlplane VM successfully with the following patch:

diff --git a/Vagrantfile b/Vagrantfile
index df8a626..b96ffab 100644
--- a/Vagrantfile
+++ b/Vagrantfile
@@ -24,21 +24,37 @@ cni = settings['cni']
 Vagrant.configure(2) do |config|
   puts "cni:"
   puts cni
+  config.vm.provider :libvirt do |libvirt|
+    libvirt.driver = "kvm"
+    libvirt.disk_device = "sda"
+    libvirt.disk_bus = "sata"
+    libvirt.machine_arch = "x86_64"
+    libvirt.machine_type= "pc-q35-5.2"
+    libvirt.nic_model_type = "e1000e"
+    libvirt.hyperv_feature :name => 'relaxed', :state => 'on'
+    libvirt.hyperv_feature :name => 'vapic', :state => 'on'
+    libvirt.hyperv_feature :name => 'spinlocks', :state => 'on', :retries => '8191'
+  end

 #   LINUX Control Plane
   config.vm.define :controlplane do |controlplane|
     controlplane.vm.host_name = "controlplane"
-    controlplane.vm.box = "roboxes/ubuntu2004"
+    controlplane.vm.box = "roboxes/ubuntu2204"

     controlplane.vm.network :private_network, ip:"#{k8s_linux_kubelet_nodeip}"
+    controlplane.vm.synced_folder "./sync/shared", "/var/sync/shared", type: "rsync"
+    controlplane.vm.synced_folder "./forked", "/var/sync/forked", type: "rsync"
+    controlplane.vm.synced_folder "./sync/linux", "/var/sync/linux", type: "rsync"  
+
     controlplane.vm.provider :virtualbox do |vb|
-    controlplane.vm.synced_folder "./sync/shared", "/var/sync/shared"
-    controlplane.vm.synced_folder "./forked", "/var/sync/forked"
-    controlplane.vm.synced_folder "./sync/linux", "/var/sync/linux"
       vb.memory = linux_ram
       vb.cpus = linux_cpus
     end

+    controlplane.vm.provider :libvirt do |libvirt|
+      libvirt.memory = linux_ram
+      libvirt.cpus = linux_cpus
+    end
     ### This allows the node to default to the right IP i think....
     # 1) this seems to break the ability to get to the internet

@@ -65,12 +81,18 @@ Vagrant.configure(2) do |config|
       vb.gui = false
     end

+    winw1.vm.provider :libvirt do |libvirt|
+      libvirt.memory = windows_ram
+      libvirt.cpus = windows_cpus
+    end
+
     winw1.vm.network :private_network, ip:"#{windows_node_ip}"
     winw1.vm.synced_folder ".", "/vagrant", disabled:true
     winw1.vm.synced_folder "./sync/shared", "C:/sync/shared"
     winw1.vm.synced_folder "./sync/windows/", "C:/sync/windows/"
     winw1.vm.synced_folder "./forked", "C:/forked/"

+    winw1.vm.communicator = "winrm"
     winw1.winrm.username = "vagrant"
     winw1.winrm.password = "vagrant"

Vagrant hangs waiting for the Windows VM to get an IP and the console is blanks for the VM:

Bringing machine 'winw1' up with 'libvirt' provider...
==> winw1: Checking if box 'sig-windows-dev-tools/windows-2019' version '1.0' is up to date...
==> winw1: Creating image (snapshot of base box volume).
==> winw1: Creating domain with the following settings...
==> winw1:  -- Name:              sig-windows-dev-tools-libvirt_winw1
==> winw1:  -- Description:       Source: /home/aravindh/devel/github/kubernetes-sigs/sig-windows-dev-tools-libvirt/Vagrantfile
==> winw1:  -- Domain type:       kvm
==> winw1:  -- Cpus:              4
==> winw1:  -- Feature:           acpi
==> winw1:  -- Feature:           apic
==> winw1:  -- Feature:           pae
==> winw1:  -- Feature (HyperV):  name=relaxed, state=on
==> winw1:  -- Feature (HyperV):  name=vapic, state=on
==> winw1:  -- Feature (HyperV):  name=spinlocks, state=on, retries=8191
==> winw1:  -- Clock offset:      utc
==> winw1:  -- Memory:            6048M
==> winw1:  -- Base box:          sig-windows-dev-tools/windows-2019
==> winw1:  -- Storage pool:      default
==> winw1:  -- Image(sda):        /var/lib/libvirt/images/sig-windows-dev-tools-libvirt_winw1.img, sata, 80G
==> winw1:  -- Disk driver opts:  cache='default'
==> winw1:  -- Graphics Type:     vnc
==> winw1:  -- Video Type:        cirrus
==> winw1:  -- Video VRAM:        16384
==> winw1:  -- Video 3D accel:    false
==> winw1:  -- Keymap:            en-us
==> winw1:  -- TPM Backend:       passthrough
==> winw1:  -- INPUT:             type=mouse, bus=ps2
==> winw1: Creating shared folders metadata...
==> winw1: Starting domain.
==> winw1: Domain launching with graphics connection settings...
==> winw1:  -- Graphics Port:      5901
==> winw1:  -- Graphics IP:        127.0.0.1
==> winw1:  -- Graphics Password:  Not defined
==> winw1:  -- Graphics Websocket: 5701
==> winw1: Waiting for domain to get an IP address...

aravindhp commented 1 year ago

Note that you have to use Vagrant v2.3.3 + vagrant-libvirt v0.11.2, otherwise you will run into vagrant-libvirt/vagrant-libvirt/issues/1688

jayunit100 commented 1 year ago

oh interesting, so, wait, can we spuport libvirt via QEMU , or , would this be like a special thing for only linux users ?

jayunit100 commented 1 year ago

@aravindhp should we have like

sig-windows-dev-tools/
  linux/
  mac/
  windows/
  common/
    install/
    antrea/
    calico/

as a new dir structure or something ?

aravindhp commented 1 year ago

oh interesting, so, wait, can we spuport libvirt via QEMU , or ,

It is actually the other way around. We can support QEMU using libvirt by specifying libvirt.driver = "qemu".

would this be like a special thing for only linux users ?

I did some digging and apparently you can run libvirt on MacOS and Windows:

libvirt MacOS support libvirt Windows support

aravindhp commented 1 year ago

@aravindhp should we have like
sig-windows-dev-tools/
  linux/
  mac/
  windows/
  common/
    install/
    antrea/
    calico/
as a new dir structure or something ?

Hopefully we wouldn't need anything OS specific if we can bring up qemu x86_64 instances on all platforms.

kubernetes-sigs / sig-windows-dev-tools

Long Term: support libvirt #157

Guidelines