Open AbdelrahmanElawady opened 4 months ago
@sabrinasadik core images like ubuntu need to get updated please give it priority
Also, I should mention that we need virtiofs module to be part of the kernel in new images and especially loaded with initramfs.
The procedure in documentation can be easily followed to build flists for all ubuntu flavours. Similar steps can be followed to also build arch images and others.
Getting error with new jammy image
Step i followed to create the image
mkdir ubuntu-jammy
sudo debootstrap jammy ubuntu-jammy http://archive.ubuntu.com/ubuntu
sudo mount -o bind /proc ubuntu-jammy/proc
sudo mount -o bind /sys ubuntu-jammy/sys
sudo mount -o bind /dev ubuntu-jammy/dev
sudo mount -o bind /run ubuntu-jammy/run
chroot ubuntu-jammy/
export PATH=/usr/local/sbin/:/usr/local/bin/:/usr/sbin/:/usr/bin/:/sbin:/bin
rm /etc/resolv.conf
echo 'nameserver 1.1.1.1' > /etc/resolv.conf
apt-get update
apt-get install cloud-init openssh-server curl
cloud-init clean
apt-get install linux-modules-extra-5.15.0-25-generic
echo 'fs-virtiofs' >> /etc/initramfs-tools/modules
update-initramfs -c -k all
apt-get clean
sudo umount ubuntu-jammy/run
sudo umount -l ubuntu-jammy/dev
sudo umount ubuntu-jammy/sys
sudo umount ubuntu-jammy/proc
rm -rf ubuntu-jammy/dev/*
tar -czf ubuntu-jammy.tar.gz -C ubuntu-jammy .
/boot
directory
root@imagecreator:~/tf-images/tfgrid3# ls -la ubuntu-jammy/boot/
total 57612
drwxr-xr-x 3 root root 4096 Apr 24 12:46 .
drwxr-xr-x 17 root root 4096 Apr 24 12:42 ..
-rw------- 1 root root 6246119 Mar 30 2022 System.map-5.15.0-25-generic
-rw-r--r-- 1 root root 260489 Mar 30 2022 config-5.15.0-25-generic
drwxr-xr-x 2 root root 4096 Apr 24 12:45 grub
lrwxrwxrwx 1 root root 28 Apr 24 12:45 initrd.img -> initrd.img-5.15.0-25-generic
-rw-r--r-- 1 root root 41394934 Apr 24 12:46 initrd.img-5.15.0-25-generic
lrwxrwxrwx 1 root root 28 Apr 24 12:45 initrd.img.old -> initrd.img-5.15.0-25-generic
lrwxrwxrwx 1 root root 25 Apr 24 12:45 vmlinuz -> vmlinuz-5.15.0-25-generic
-rw------- 1 root root 11073600 Mar 30 2022 vmlinuz-5.15.0-25-generic
lrwxrwxrwx 1 root root 25 Apr 24 12:45 vmlinuz.old -> vmlinuz-5.15.0-25-generic
WorkloadDeployError: Failed to deploy zmachine with name test12 on node 13 due to: cloud-hypervisor: 44.92169ms: <payload_loader> WARN:vmm/src/vm.rs:1011 -- Loading of legacy (non-PVH) firmware is deprecated and will be removed in a future version. Error booting VM: VmBoot(FirmwareTooLarge) : failed to spawn vm machine process '604-407362-test12'.
This works with me on devnet:
Can you share vm configuration?
I think it's because the kernel decompression feature didn't reach other networks yet, if you tried readelf -n <vmlinuz>
it should return an error.
you can extract the kernel by doing something similar to this:
curl -O https://raw.githubusercontent.com/torvalds/linux/master/scripts/extract-vmlinux
chmod +x extract-vmlinux
sudo ./extract-vmlinux ubuntu-jammy/boot/vmlinuz | sudo tee ubuntu-jammy/boot/vmlinuz-5.15.0-25-generic.elf > /dev/null
# then replace original kernel
sudo mv ubuntu-jammy/boot/vmlinuz-5.15.0-25-generic.elf ubuntu-jammy/boot/vmlinuz-5.15.0-25-generic
Ubuntu 22 flist created and tested
Kernel 5.15.0-25-generic is not available in the package repository for Ubuntu versions 18 and 20
arch-chroot
env :
ubuntu 18root@imagecreator:/# apt-get install linux-modules-extra-5.15.0-25-generic
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package linux-modules-extra-5.15.0-25-generic
E: Couldn't find any package by glob 'linux-modules-extra-5.15.0-25-generic'
E: Couldn't find any package by regex 'linux-modules-extra-5.15.0-25-generic'
root@imagecreator:/# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04 LTS
Release: 18.04
Codename: bionic
ubuntu 20
root@imagecreator:/# apt-get install linux-modules-extra-5.15.0-25-generic
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package linux-modules-extra-5.15.0-25-generic
E: Couldn't find any package by glob 'linux-modules-extra-5.15.0-25-generic'
E: Couldn't find any package by regex 'linux-modules-extra-5.15.0-25-generic'
root@imagecreator:/# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04 LTS
Release: 20.04
Codename: focal
I believe kernel 5.15 was specified as an example, so other versions can be used too.
ssh root@185.69.167.182 -A
ssh: connect to host 185.69.167.182 port 22: Connection refused
extract-vmlinux
script
root@imagecreator:~# ./extract-vmlinux ubuntu-focal/boot/vmlinuz-5.4.0-26-generic > ubuntu-focal/boot/vmlinuz-5.4.0-26-generic.elf
extract-vmlinux: Cannot find vmlinux.
root@imagecreator:~# binwalk --extract --signature ubuntu-focal/boot/vmlinuz-5.4.0-26-generic --run-as=root
DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
0 0x0 Microsoft executable, portable (PE)
19377 0x4BB1 LZ4 compressed data, legacy
5375834 0x52075A MySQL ISAM index file Version 1
6973538 0x6A6862 ELF, 64-bit LSB processor-specific, ("")
8810431 0x866FBF xz compressed data
9322046 0x8E3E3E AES Inverse S-Box
9322302 0x8E3F3E AES S-Box
10001323 0x989BAB LZ4 compressed data, legacy
10230704 0x9C1BB0 Certificate in DER format (x509 v3), header length: 4, sequence length: 1353
11410887 0xAE1DC7 LZ4 compressed data, legacy
11410923 0xAE1DEB LZ4 compressed data, legacy
11656072 0xB1DB88 Object signature in DER format (PKCS header length: 4, sequence length: 1899
11656241 0xB1DC31 Certificate in DER format (x509 v3), header length: 4, sequence length: 1056
866FBF.xz
root@imagecreator:~/ubuntu-focal/boot/_vmlinuz-5.4.0-26-generic.extracted# file 866FBF.xz
866FBF.xz: XZ compressed data, checksum
root@imagecreator:~/ubuntu-focal/boot/_vmlinuz-5.4.0-26-generic.extracted# xzcat 866FBF.xz > vmlinux xzcat: 866FBF.xz: Compressed data is corrupt
root@imagecreator:~/ubuntu-focal/boot/_vmlinuz-5.4.0-26-generic.extracted# unxz 866FBF.xz unxz: 866FBF.xz: Compressed data is corrupt
- Uploaded the flist and tried testing without extracting the flist
Failed to send request to twinId 4927 with command: zos.deployment.get, payload: {"contract_id":442897} RMBError: 106 e2e error: invalid cipher data
<img width="1154" alt="image" src="https://github.com/threefoldtech/tf-images/assets/13523434/6bdcc0b5-36ab-47ec-8565-25491c3975fa">
- Will try another kernel version instead of the default Linux kernel versions
- Ubuntu 18 flist created but can's ssh to vms, investigating
ssh root@185.69.167.182 -A ssh: connect to host 185.69.167.182 port 22: Connection refused
![]()
Not reachable over wireguard as well
root@grid-monitoring:/etc/wireguard# ping 10.20.2.2 PING 10.20.2.2 (10.20.2.2) 56(84) bytes of data. ^C --- 10.20.2.2 ping statistics --- 55 packets transmitted, 0 received, 100% packet loss, time 55292ms
Deployed the ubuntu 20 flist on devnet as it has the latest zos version and no need to extract the kernel
not reachable over wireguard
root@grid-monitoring:/etc/wireguard# ping 10.20.4.2
PING 10.20.4.2 (10.20.4.2) 56(84) bytes of data.
^C
--- 10.20.4.2 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4093ms
not reachable over mycelium
root@imagecreator:~# ping6 478:466d:a78f:64c4:ff0f:0:400:2
PING 478:466d:a78f:64c4:ff0f:0:400:2(478:466d:a78f:64c4:ff0f:0:400:2) 56 data bytes
From 509:5bf7:d072:f113:f37f:768e:9055:220b icmp_seq=1 Destination unreachable: No route
From 509:5bf7:d072:f113:f37f:768e:9055:220b icmp_seq=11 Destination unreachable: No route
^C
--- 478:466d:a78f:64c4:ff0f:0:400:2 ping statistics ---
11 packets transmitted, 0 received, +11 errors, 100% packet loss, time 10250ms
Node id 166 , Logs :
| | 2024-05-09 08:30:39 | storagedstdout | [+] storaged: 2024-05-09T06:30:39Z warn Could not find filesystem 1438-109627-ubu20te9
2024-05-09 08:49:30 | provisiondstdout | [+] provisiond: 2024-05-09T06:49:30Z info contract pause state contract=109625 paused=false twin=1438
-- | -- | --
| | 2024-05-09 08:49:30 | provisiondstdout | [+] provisiond: 2024-05-09T06:49:30Z info contract pause state contract=109301 paused=true twin=89
| | 2024-05-09 08:49:30 | provisiondstdout | [+] provisiond: 2024-05-09T06:49:30Z info contract pause state contract=109300 paused=false twin=89
| | 2024-05-09 08:48:11 | redisstdout | [+] redis: 1314:M 09 May 2024 06:48:11.166 * Background saving terminated with success
| | 2024-05-09 08:48:11 | redisstdout | [+] redis: 24546:C 09 May 2024 06:48:11.066 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
| | 2024-05-09 08:48:11 | redisstdout | [+] redis: 24546:C 09 May 2024 06:48:11.066 * DB saved on disk
| | 2024-05-09 08:48:11 | redisstdout | [+] redis: 1314:M 09 May 2024 06:48:11.065 * Background saving started by pid 24546
| | 2024-05-09 08:48:11 | redisstdout | [+] redis: 1314:M 09 May 2024 06:48:11.065 * 100 changes in 300 seconds. Saving...
| | 2024-05-09 08:43:10 | redisstdout | [+] redis: 1314:M 09 May 2024 06:43:10.139 * Background saving terminated with success
| | 2024-05-09 08:43:10 | redisstdout | [+] redis: 15916:C 09 May 2024 06:43:10.043 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
| | 2024-05-09 08:43:10 | redisstdout | [+] redis: 15916:C 09 May 2024 06:43:10.042 * DB saved on disk
| | 2024-05-09 08:43:10 | redisstdout | [+] redis: 1314:M 09 May 2024 06:43:10.038 * Background saving started by pid 15916
| | 2024-05-09 08:43:10 | redisstdout | [+] redis: 1314:M 09 May 2024 06:43:10.037 * 100 changes in 300 seconds. Saving...
| | 2024-05-09 08:38:09 | redisstdout | [+] redis: 1314:M 09 May 2024 06:38:09.107 * Background saving terminated with success
| | 2024-05-09 08:38:09 | redisstdout | [+] redis: 9361:C 09 May 2024 06:38:09.011 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
| | 2024-05-09 08:38:09 | redisstdout | [+] redis: 9361:C 09 May 2024 06:38:09.010 * DB saved on disk
| | 2024-05-09 08:38:09 | redisstdout | [+] redis: 1314:M 09 May 2024 06:38:09.006 * Background saving started by pid 9361
| | 2024-05-09 08:38:09 | redisstdout | [+] redis: 1314:M 09 May 2024 06:38:09.006 * 100 changes in 300 seconds. Saving...
| | 2024-05-09 08:33:08 | redisstdout | [+] redis: 1314:M 09 May 2024 06:33:08.188 * Background saving terminated with success
| | 2024-05-09 08:33:08 | redisstdout | [+] redis: 2981:C 09 May 2024 06:33:08.091 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
| | 2024-05-09 08:33:08 | redisstdout | [+] redis: 2981:C 09 May 2024 06:33:08.090 * DB saved on disk
| | 2024-05-09 08:33:08 | redisstdout | [+] redis: 1314:M 09 May 2024 06:33:08.087 * Background saving started by pid 2981
| | 2024-05-09 08:33:08 | redisstdout | [+] redis: 1314:M 09 May 2024 06:33:08.086 * 100 changes in 300 seconds. Saving...
| | 2024-05-09 08:30:39 | flistdstdout | [+] flistd: 2024-05-09T06:30:39Z info request to mount flist storage= url=https://hub.grid.tf/petep.3bot/ubuntu-focal.flist
| | 2024-05-09 08:30:39 | flistdstdout | [+] flistd: 2024-05-09T06:30:39Z info request to mount flist: {ReadOnly:false Limit:0 Storage: PersistedVolume:/mnt/dde7a534-22d5-4924-ac56-65d9a9aa2fab/rootfs:1438-109627-ubu20te9} name=1438-109627-ubu20te9 storage= url=https://hub.grid.tf/petep.3bot/ubuntu-focal.flist
| | 2024-05-09 08:30:39 | storagedstdout | [+] storaged: 2024-05-09T06:30:39Z info Creating new volume with size 26843545600
| | 2024-05-09 08:30:39 | storagedstdout | [+] storaged: 2024-05-09T06:30:39Z warn Could not find filesystem 1438-109627-ubu20te9
| | 2024-05-09 08:30:39 | storagedstdout | [+] storaged: 2024-05-09T06:30:39Z info Deleting volume 1438-109627-ubu20te9
| | 2024-05-09 08:30:39 | flistdstdout | [+] flistd: 2024-05-09T06:30:39Z info request to mount flist: {ReadOnly:true Limit:0 Storage: PersistedVolume:} name=cloud-container:c1f77d34c40c7879a220ba3d20b3535a storage= url=https://hub.grid.tf/tf-autobuilder/cloud-container-9dba60e.flist
| | 2024-05-09 08:30:36 | flistdstdout | [+] flistd: 2024-05-09T06:30:36Z info request to mount flist storage= url=https://hub.grid.tf/petep.3bot/ubuntu-focal.flist
| | 2024-05-09 08:30:36 | flistdstdout | [+] flistd: 2024-05-09T06:30:36Z info request to mount flist: {ReadOnly:true Limit:0 Storage: PersistedVolume:} name=1438-109627-ubu20te9 storage= url=https://hub.grid.tf/petep.3bot/ubuntu-focal.flist
| | 2024-05-09 08:30:36 | networkdstdout | [+] networkd: 2024-05-09T06:30:36Z info Setting up mycelium tap interface tap-name=HVCbLtrUxUF5w
| | 2024-05-09 08:30:36 | networkdstdout | [+] networkd: 2024-05-09T06:30:36Z info Setting up tap interface network-id=Gn8mNieVU6Dkq
linux-generic-hwe-20.04
kernel version got the same error when extracting the kernel
Flists links https://hub.grid.tf/petep.3bot/ubuntu-jammy.flist.md https://hub.grid.tf/petep.3bot/ubuntu-noble.flist.md
Tested both
Description
Our full-vm images are currently done as a raw images (single image.raw file). It's been deprecated for a while and now it's removed from zos documentation. We should move to normal flist with kernel as part of it as mentioned here.