Deprecate raw images - Githubissues

AbdelrahmanElawady commented 4 months ago

Description

Our full-vm images are currently done as a raw images (single image.raw file). It's been deprecated for a while and now it's removed from zos documentation. We should move to normal flist with kernel as part of it as mentioned here.

[x] #237

xmonader commented 4 months ago

@sabrinasadik core images like ubuntu need to get updated please give it priority

AbdelrahmanElawady commented 4 months ago

Also, I should mention that we need virtiofs module to be part of the kernel in new images and especially loaded with initramfs.

muhamadazmy commented 4 months ago

The procedure in documentation can be easily followed to build flists for all ubuntu flavours. Similar steps can be followed to also build arch images and others.

PeterNashaat commented 4 months ago

Getting error with new jammy image
Step i followed to create the image

mkdir ubuntu-jammy
sudo debootstrap jammy ubuntu-jammy  http://archive.ubuntu.com/ubuntu
sudo mount -o bind /proc ubuntu-jammy/proc
sudo mount -o bind /sys ubuntu-jammy/sys
sudo mount -o bind /dev ubuntu-jammy/dev
sudo mount -o bind /run ubuntu-jammy/run
chroot ubuntu-jammy/
export PATH=/usr/local/sbin/:/usr/local/bin/:/usr/sbin/:/usr/bin/:/sbin:/bin
rm /etc/resolv.conf
echo 'nameserver 1.1.1.1' > /etc/resolv.conf
apt-get update
apt-get install cloud-init openssh-server curl
cloud-init clean
apt-get install linux-modules-extra-5.15.0-25-generic
echo 'fs-virtiofs' >> /etc/initramfs-tools/modules
update-initramfs -c -k all
apt-get clean
sudo umount ubuntu-jammy/run
sudo umount -l ubuntu-jammy/dev
sudo umount ubuntu-jammy/sys
sudo umount ubuntu-jammy/proc
rm -rf ubuntu-jammy/dev/*
tar -czf ubuntu-jammy.tar.gz -C  ubuntu-jammy .

/boot directory

root@imagecreator:~/tf-images/tfgrid3# ls -la ubuntu-jammy/boot/
total 57612
drwxr-xr-x  3 root root     4096 Apr 24 12:46 .
drwxr-xr-x 17 root root     4096 Apr 24 12:42 ..
-rw-------  1 root root  6246119 Mar 30  2022 System.map-5.15.0-25-generic
-rw-r--r--  1 root root   260489 Mar 30  2022 config-5.15.0-25-generic
drwxr-xr-x  2 root root     4096 Apr 24 12:45 grub
lrwxrwxrwx  1 root root       28 Apr 24 12:45 initrd.img -> initrd.img-5.15.0-25-generic
-rw-r--r--  1 root root 41394934 Apr 24 12:46 initrd.img-5.15.0-25-generic
lrwxrwxrwx  1 root root       28 Apr 24 12:45 initrd.img.old -> initrd.img-5.15.0-25-generic
lrwxrwxrwx  1 root root       25 Apr 24 12:45 vmlinuz -> vmlinuz-5.15.0-25-generic
-rw-------  1 root root 11073600 Mar 30  2022 vmlinuz-5.15.0-25-generic
lrwxrwxrwx  1 root root       25 Apr 24 12:45 vmlinuz.old -> vmlinuz-5.15.0-25-generic

Link to flist

PeterNashaat commented 4 months ago

Used micro VM and tried installing new vm with the flist again
- Getting this error, tried it on multiple nodes man times and got the same error

WorkloadDeployError: Failed to deploy zmachine with name test12 on node 13 due to: cloud-hypervisor: 44.92169ms: <payload_loader> WARN:vmm/src/vm.rs:1011 -- Loading of legacy (non-PVH) firmware is deprecated and will be removed in a future version. Error booting VM: VmBoot(FirmwareTooLarge) : failed to spawn vm machine process '604-407362-test12'.

AbdelrahmanElawady commented 4 months ago

This works with me on devnet:

Can you share vm configuration?

PeterNashaat commented 4 months ago

Getting the same error on testnet

AbdelrahmanElawady commented 4 months ago

I think it's because the kernel decompression feature didn't reach other networks yet, if you tried readelf -n <vmlinuz> it should return an error.

you can extract the kernel by doing something similar to this:

curl -O https://raw.githubusercontent.com/torvalds/linux/master/scripts/extract-vmlinux
chmod +x extract-vmlinux

sudo ./extract-vmlinux ubuntu-jammy/boot/vmlinuz | sudo tee ubuntu-jammy/boot/vmlinuz-5.15.0-25-generic.elf > /dev/null
# then replace original kernel
sudo mv ubuntu-jammy/boot/vmlinuz-5.15.0-25-generic.elf ubuntu-jammy/boot/vmlinuz-5.15.0-25-generic

PeterNashaat commented 3 months ago

Ubuntu 22 flist created and tested
Kernel 5.15.0-25-generic is not available in the package repository for Ubuntu versions 18 and 20
- in arch-chroot env : ubuntu 18

root@imagecreator:/# apt-get install linux-modules-extra-5.15.0-25-generic
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package linux-modules-extra-5.15.0-25-generic
E: Couldn't find any package by glob 'linux-modules-extra-5.15.0-25-generic'
E: Couldn't find any package by regex 'linux-modules-extra-5.15.0-25-generic'
root@imagecreator:/# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04 LTS
Release:        18.04
Codename:       bionic

ubuntu 20

root@imagecreator:/# apt-get install linux-modules-extra-5.15.0-25-generic
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package linux-modules-extra-5.15.0-25-generic
E: Couldn't find any package by glob 'linux-modules-extra-5.15.0-25-generic'
E: Couldn't find any package by regex 'linux-modules-extra-5.15.0-25-generic'
root@imagecreator:/# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04 LTS
Release:        20.04
Codename:       focal

AbdelrahmanElawady commented 3 months ago

I believe kernel 5.15 was specified as an example, so other versions can be used too.

PeterNashaat commented 3 months ago

Ubuntu 24 flist
Tested

PeterNashaat commented 3 months ago

Ubuntu 18 flist created but can's ssh to vms, investigating

ssh root@185.69.167.182 -A
ssh: connect to host 185.69.167.182 port 22: Connection refused

PeterNashaat commented 3 months ago

Focal flist Link but had issue with extracting kernel from vmlinuz file

Issue extracting with extract-vmlinux script

root@imagecreator:~# ./extract-vmlinux ubuntu-focal/boot/vmlinuz-5.4.0-26-generic > ubuntu-focal/boot/vmlinuz-5.4.0-26-generic.elf
extract-vmlinux: Cannot find vmlinux.

root@imagecreator:~# binwalk --extract --signature ubuntu-focal/boot/vmlinuz-5.4.0-26-generic --run-as=root

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             Microsoft executable, portable (PE)
19377         0x4BB1          LZ4 compressed data, legacy
5375834       0x52075A        MySQL ISAM index file Version 1
6973538       0x6A6862        ELF, 64-bit LSB processor-specific, ("")
8810431       0x866FBF        xz compressed data
9322046       0x8E3E3E        AES Inverse S-Box
9322302       0x8E3F3E        AES S-Box
10001323      0x989BAB        LZ4 compressed data, legacy
10230704      0x9C1BB0        Certificate in DER format (x509 v3), header length: 4, sequence length: 1353
11410887      0xAE1DC7        LZ4 compressed data, legacy
11410923      0xAE1DEB        LZ4 compressed data, legacy
11656072      0xB1DB88        Object signature in DER format (PKCS header length: 4, sequence length: 1899
11656241      0xB1DC31        Certificate in DER format (x509 v3), header length: 4, sequence length: 1056

Then tried to extract output from binwalk file 866FBF.xz


root@imagecreator:~/ubuntu-focal/boot/_vmlinuz-5.4.0-26-generic.extracted# file 866FBF.xz
866FBF.xz: XZ compressed data, checksum

root@imagecreator:~/ubuntu-focal/boot/_vmlinuz-5.4.0-26-generic.extracted# xzcat 866FBF.xz > vmlinux xzcat: 866FBF.xz: Compressed data is corrupt

root@imagecreator:~/ubuntu-focal/boot/_vmlinuz-5.4.0-26-generic.extracted# unxz 866FBF.xz unxz: 866FBF.xz: Compressed data is corrupt


- Uploaded the flist and tried testing without extracting the flist

Failed to send request to twinId 4927 with command: zos.deployment.get, payload: {"contract_id":442897} RMBError: 106 e2e error: invalid cipher data



<img width="1154" alt="image" src="https://github.com/threefoldtech/tf-images/assets/13523434/6bdcc0b5-36ab-47ec-8565-25491c3975fa">

- Will try another kernel version instead of the default Linux kernel versions

PeterNashaat commented 3 months ago

Ubuntu 18 flist created but can's ssh to vms, investigating
ssh root@185.69.167.182 -A
ssh: connect to host 185.69.167.182 port 22: Connection refused

Not reachable over wireguard as well

root@grid-monitoring:/etc/wireguard# ping 10.20.2.2 PING 10.20.2.2 (10.20.2.2) 56(84) bytes of data. ^C --- 10.20.2.2 ping statistics --- 55 packets transmitted, 0 received, 100% packet loss, time 55292ms

PeterNashaat commented 3 months ago

Deployed the ubuntu 20 flist on devnet as it has the latest zos version and no need to extract the kernel

not reachable over wireguard

root@grid-monitoring:/etc/wireguard# ping 10.20.4.2
PING 10.20.4.2 (10.20.4.2) 56(84) bytes of data.
^C
--- 10.20.4.2 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4093ms

not reachable over mycelium

root@imagecreator:~# ping6 478:466d:a78f:64c4:ff0f:0:400:2
PING 478:466d:a78f:64c4:ff0f:0:400:2(478:466d:a78f:64c4:ff0f:0:400:2) 56 data bytes
From 509:5bf7:d072:f113:f37f:768e:9055:220b icmp_seq=1 Destination unreachable: No route
From 509:5bf7:d072:f113:f37f:768e:9055:220b icmp_seq=11 Destination unreachable: No route
^C
--- 478:466d:a78f:64c4:ff0f:0:400:2 ping statistics ---
11 packets transmitted, 0 received, +11 errors, 100% packet loss, time 10250ms

Node id 166 , Logs : | | 2024-05-09 08:30:39 | storagedstdout | [+] storaged: 2024-05-09T06:30:39Z warn Could not find filesystem 1438-109627-ubu20te9


2024-05-09 08:49:30 | provisiondstdout | [+] provisiond: 2024-05-09T06:49:30Z info contract pause state contract=109625 paused=false twin=1438
-- | -- | --
  |   | 2024-05-09 08:49:30 | provisiondstdout | [+] provisiond: 2024-05-09T06:49:30Z info contract pause state contract=109301 paused=true twin=89
  |   | 2024-05-09 08:49:30 | provisiondstdout | [+] provisiond: 2024-05-09T06:49:30Z info contract pause state contract=109300 paused=false twin=89
  |   | 2024-05-09 08:48:11 | redisstdout | [+] redis: 1314:M 09 May 2024 06:48:11.166 * Background saving terminated with success
  |   | 2024-05-09 08:48:11 | redisstdout | [+] redis: 24546:C 09 May 2024 06:48:11.066 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
  |   | 2024-05-09 08:48:11 | redisstdout | [+] redis: 24546:C 09 May 2024 06:48:11.066 * DB saved on disk
  |   | 2024-05-09 08:48:11 | redisstdout | [+] redis: 1314:M 09 May 2024 06:48:11.065 * Background saving started by pid 24546
  |   | 2024-05-09 08:48:11 | redisstdout | [+] redis: 1314:M 09 May 2024 06:48:11.065 * 100 changes in 300 seconds. Saving...
  |   | 2024-05-09 08:43:10 | redisstdout | [+] redis: 1314:M 09 May 2024 06:43:10.139 * Background saving terminated with success
  |   | 2024-05-09 08:43:10 | redisstdout | [+] redis: 15916:C 09 May 2024 06:43:10.043 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
  |   | 2024-05-09 08:43:10 | redisstdout | [+] redis: 15916:C 09 May 2024 06:43:10.042 * DB saved on disk
  |   | 2024-05-09 08:43:10 | redisstdout | [+] redis: 1314:M 09 May 2024 06:43:10.038 * Background saving started by pid 15916
  |   | 2024-05-09 08:43:10 | redisstdout | [+] redis: 1314:M 09 May 2024 06:43:10.037 * 100 changes in 300 seconds. Saving...
  |   | 2024-05-09 08:38:09 | redisstdout | [+] redis: 1314:M 09 May 2024 06:38:09.107 * Background saving terminated with success
  |   | 2024-05-09 08:38:09 | redisstdout | [+] redis: 9361:C 09 May 2024 06:38:09.011 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
  |   | 2024-05-09 08:38:09 | redisstdout | [+] redis: 9361:C 09 May 2024 06:38:09.010 * DB saved on disk
  |   | 2024-05-09 08:38:09 | redisstdout | [+] redis: 1314:M 09 May 2024 06:38:09.006 * Background saving started by pid 9361
  |   | 2024-05-09 08:38:09 | redisstdout | [+] redis: 1314:M 09 May 2024 06:38:09.006 * 100 changes in 300 seconds. Saving...
  |   | 2024-05-09 08:33:08 | redisstdout | [+] redis: 1314:M 09 May 2024 06:33:08.188 * Background saving terminated with success
  |   | 2024-05-09 08:33:08 | redisstdout | [+] redis: 2981:C 09 May 2024 06:33:08.091 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
  |   | 2024-05-09 08:33:08 | redisstdout | [+] redis: 2981:C 09 May 2024 06:33:08.090 * DB saved on disk
  |   | 2024-05-09 08:33:08 | redisstdout | [+] redis: 1314:M 09 May 2024 06:33:08.087 * Background saving started by pid 2981
  |   | 2024-05-09 08:33:08 | redisstdout | [+] redis: 1314:M 09 May 2024 06:33:08.086 * 100 changes in 300 seconds. Saving...
  |   | 2024-05-09 08:30:39 | flistdstdout | [+] flistd: 2024-05-09T06:30:39Z info request to mount flist storage= url=https://hub.grid.tf/petep.3bot/ubuntu-focal.flist
  |   | 2024-05-09 08:30:39 | flistdstdout | [+] flistd: 2024-05-09T06:30:39Z info request to mount flist: {ReadOnly:false Limit:0 Storage: PersistedVolume:/mnt/dde7a534-22d5-4924-ac56-65d9a9aa2fab/rootfs:1438-109627-ubu20te9} name=1438-109627-ubu20te9 storage= url=https://hub.grid.tf/petep.3bot/ubuntu-focal.flist
  |   | 2024-05-09 08:30:39 | storagedstdout | [+] storaged: 2024-05-09T06:30:39Z info Creating new volume with size 26843545600
  |   | 2024-05-09 08:30:39 | storagedstdout | [+] storaged: 2024-05-09T06:30:39Z warn Could not find filesystem 1438-109627-ubu20te9
  |   | 2024-05-09 08:30:39 | storagedstdout | [+] storaged: 2024-05-09T06:30:39Z info Deleting volume 1438-109627-ubu20te9
  |   | 2024-05-09 08:30:39 | flistdstdout | [+] flistd: 2024-05-09T06:30:39Z info request to mount flist: {ReadOnly:true Limit:0 Storage: PersistedVolume:} name=cloud-container:c1f77d34c40c7879a220ba3d20b3535a storage= url=https://hub.grid.tf/tf-autobuilder/cloud-container-9dba60e.flist
  |   | 2024-05-09 08:30:36 | flistdstdout | [+] flistd: 2024-05-09T06:30:36Z info request to mount flist storage= url=https://hub.grid.tf/petep.3bot/ubuntu-focal.flist
  |   | 2024-05-09 08:30:36 | flistdstdout | [+] flistd: 2024-05-09T06:30:36Z info request to mount flist: {ReadOnly:true Limit:0 Storage: PersistedVolume:} name=1438-109627-ubu20te9 storage= url=https://hub.grid.tf/petep.3bot/ubuntu-focal.flist
  |   | 2024-05-09 08:30:36 | networkdstdout | [+] networkd: 2024-05-09T06:30:36Z info Setting up mycelium tap interface tap-name=HVCbLtrUxUF5w
  |   | 2024-05-09 08:30:36 | networkdstdout | [+] networkd: 2024-05-09T06:30:36Z info Setting up tap interface network-id=Gn8mNieVU6Dkq

Tried linux-generic-hwe-20.04 kernel version got the same error when extracting the kernel

PeterNashaat commented 3 months ago

Blocked on ubuntu 18 and 20.

PeterNashaat commented 3 months ago

Ubuntu 22 and 24 images are done and tested https://github.com/threefoldtech/tf-images/issues/237

PeterNashaat commented 3 months ago

Created a PR with updated ubuntu 22 and 24.04 full vm flists.
- Updated flists directories with 2 sub directories Fullvm and microvm
- Added automation script which automates the setup, configuration, archiving, and uploading of an Ubuntu system ready for use as an FLIST.
- Testing Full vms for now is done through Terrafrom
- Added readme.md file for Automated, Manual and Testing process

Flists links https://hub.grid.tf/petep.3bot/ubuntu-jammy.flist.md https://hub.grid.tf/petep.3bot/ubuntu-noble.flist.md

Tested both

threefoldtech / tf-images

Deprecate raw images #227

Description