search

SovereignCloudStack / k8s-cluster-api-provider

Automation to use the OpenStack Kubernetes API Provider on SCS
https://scs.community/
Other
20 stars 11 forks source link

Test Ubuntu 22.04 images #409

Closed jschoone closed 1 year ago

jschoone commented 1 year ago

Since the subiquity bug should be fixed now, let's test the Ubuntu 22.04 CAPI images again.

As discussed @matofeder @chess-knight, I would be grateful if you could test them. Some groundwork has been done in branch feat/use-ubu2204

chess-knight commented 1 year ago

Since the subiquity bug should be fixed now

I did tests with osism and also self-build images but without success.

So I decided to investigate this issue more and I found an upstream issue with the same problems. Then I found a solution and created an upstream PR for that. With that changes, I self-built capi image and uploaded it to OpenStack and I was able to create the k8s cluster successfully.

It seems that we will be able to run Ubuntu 22.04 capi images but still, there is one "big" issue. Images are too big - 10-15GB each.

chess-knight commented 1 year ago

It seems that we will be able to run Ubuntu 22.04 capi images but still, there is one "big" issue. Images are too big - 10-15GB each.

I created an issue https://github.com/kubernetes-sigs/image-builder/issues/1155 for the big qcow2 images.

dscaravaggi commented 1 year ago

Hi @chess-knight, just for comparing your info with mine, which is the assigned nic device name assigned inside QEMU guest ens192, eth0 en0ps0 ? you can see my problem in #1153

may you check the output of: dmesg | grep -i eth

thanks in adavnce Diego

chess-knight commented 1 year ago

Hi @dscaravaggi, I did these steps, and installed k8s workload cluster using this repository on the OpenStack.

just for comparing your info with mine, which is the assigned nic device name assigned inside QEMU guest ens192, eth0 en0ps0 ?

may you check the output of: dmesg | grep -i eth

When I run that command on the control-plane I get:

# dmesg | grep -i eth
[    2.281547] virtio_net virtio1 ens3: renamed from eth0
[  186.108816] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[  190.620644] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[  196.748558] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

For the worker I get:

# dmesg | grep -i eth
[    3.035473] virtio_net virtio1 ens3: renamed from eth0
dscaravaggi commented 1 year ago

Thanks @chess-knight 

# dmesg | grep -i eth
[    2.281547] virtio_net virtio1 ens3: renamed from eth0
[  186.108816] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

eth0 is renamed only once and ens192 is expected ! I will compare the builder kit for QEMU vs VSPHERE

because useing a machine created as:

make  build-node-ova-vsphere-ubuntu-2204

I got 

[    5.206207] vmxnet3 0000:0b:00.0 eth0: NIC Link is Up 10000 Mbps
[    5.521328] vmxnet3 0000:0b:00.0 ens192: renamed from eth0
[   13.945937] vmxnet3 0000:0b:00.0 eth0: renamed from ens192

the second rename is not correct from my point of view, but I did not see the exact point in cloud-init script

dscaravaggi commented 1 year ago

Hi @chess-knight , I checked kubevirt-netplan, whichi is used in both vsphere and QEMU

ethernets:
    id0:
      match:
        name: enp*s*
      dhcp4: true

ethernet rule is enps which does not match your ens3 how did you sicceeded Kubevirt initialization ?

are kubevirt pods up & running in your test box ?

thanks a lot for your patience I'm jsut tryning to undestand difference with mine setup procedure.

Diego

chess-knight commented 1 year ago

Hi @dscaravaggi, The kubevirt-netplan as far as I know is only used when kubevirt == "true". But I am not building images for the kubevirt(I run build-qemu-ubuntu-2204 instead of build-kubevirt-qemu-ubuntu-2204)

chess-knight commented 1 year ago

I did tests with osism and also self-build images but without success.

I did tests again after the upstream PR was merged.

I tried ubuntu-capi-image-v1.26.5 image from osism, which has image_build_date 2023-05-19 - day after the upstream merge, and it works as expected!

I created an issue https://github.com/kubernetes-sigs/image-builder/issues/1155 for the big qcow2 images.

But the problem with big image disk size still exists. There is no progress on that issue. Only one more issue with the same experiences is created in the upstream repository.

berendt commented 1 year ago

The disc size is not pretty. But is it really a blocker? I would prefer if we could go to Ubuntu 22.04.

chess-knight commented 1 year ago

The disc size is not pretty. But is it really a blocker? I would prefer if we could go to Ubuntu 22.04.

Hi @berendt, also in my opinion it is not a blocker. qcow2 1.26.5 of 22.04 has 13.6GB vs 20.04 4GB

The only problem with this repository that I found so far is kube_image_raw variable. It is set by default to true but the script with qcow2 conversion fails:

$ qemu-img convert ubuntu-2204-kube-v1.26.5.qcow2 -O raw -S 4k ubuntu-2204-kube-v1.26.5.raw
qemu-img: error while writing sector 29654528: No space left on device

But I think it is not something what cannot be overcome. Probably only bigger disk size for capi mgmt node will do that.

chess-knight commented 1 year ago

I found a way to reduce image size so I created an upstream pull request, let's wait.

chess-knight commented 1 year ago

I found a way to reduce image size so I created an upstream pull request, let's wait.

The upstream PR was successfully merged yesterday(2023-06-20). @berendt can you please rebuild some capi images, so we can retest them and finally switch to Ubuntu 22.04?

berendt commented 1 year ago

Must customize the CI so that we can do daily rebuilds. Currently we only have static builds that can't be rebuilt.

chess-knight commented 1 year ago

Must customize the CI so that we can do daily rebuilds. Currently we only have static builds that can't be rebuilt.

Ok so now, ubuntu 22.04 capi images work from the 2023-05-19(05-18 upstream merge) but have a big image disk size. According to https://minio.services.osism.tech/openstack-k8s-capi-images, versions >= v1.24.14, v1.25.10, v1.26.5, v1.27.1

Small disk size I suppose will be available from the 2023-06-21(06-20 upstream merge).

berendt commented 1 year ago

I only rebuilt the following new images. Please check them.

chess-knight commented 1 year ago

I only rebuilt the following new images. Please check them.

I tested all the mentioned images(2023-06-21 build date) with branch ubuntu_2204 and can confirm that they work(and have a small qcow2 disk size ~4,35GB).

berendt commented 1 year ago

I disabled/archived the Ubuntu 20.04 builds and will add the three listed CAPI images above to the openstack-image-manager repository.