Builders often fail at startup because of missing swap

praiskup commented 10 months ago



TASK [Make sure that we have 140G memory or SWAP for Mock caches] **************
Saturday 09 December 2023  21:11:15 +0000 (0:00:00.952)       0:01:11.132 ***** 
fatal: [50.17.25.239]: FAILED! => {
    "assertion": "ansible_memory_mb.swap.free + ansible_memory_mb.real.free >= 1024 * 140",
    "changed": false,
    "evaluated_to": false,
    "msg": "Not enough memory+swap, the builder is unusable: 0 MB SWAP and 6768 MB RAM"
}

PLAY RECAP *********************************************************************
50.17.25.239               : ok=16   changed=8    unreachable=0    failed=1    skipped=19   rescued=0    ignored=0

FrostyX commented 10 months ago

This seems to be more complicated than we expected.

I found this in journalctl -u enable-swap:

systemd[1]: Started enable-swap.service - Enable swap on /dev/nvme0n1.
enable-swap.sh[719]: + set -e
enable-swap.sh[719]: + part_suffix=p
enable-swap.sh[719]: + swap_device=
enable-swap.sh[719]: + test -f /config/resalloc-vars.sh
enable-swap.sh[719]: + grep POWER9 /proc/cpuinfo
enable-swap.sh[719]: + test -e /dev/xvda1
enable-swap.sh[719]: + test -e /dev/nvme1n1
enable-swap.sh[719]: + swap_device=/dev/nvme1n1
enable-swap.sh[719]: + test -n /dev/nvme1n1
enable-swap.sh[719]: + systemctl unmask tmp.mount
enable-swap.sh[719]: + systemctl start tmp.mount
enable-swap.sh[827]: + echo 'n
enable-swap.sh[827]: p
enable-swap.sh[827]: +16G
enable-swap.sh[827]: n
enable-swap.sh[827]: p
enable-swap.sh[827]: 2
enable-swap.sh[827]: w
enable-swap.sh[827]: '
enable-swap.sh[828]: + fdisk /dev/nvme1n1
enable-swap.sh[828]: Welcome to fdisk (util-linux 2.38.1).
enable-swap.sh[828]: Changes will remain in memory only, until you decide to write them.
enable-swap.sh[828]: Be careful before using the write command.
enable-swap.sh[828]: Device does not contain a recognized partition table.
enable-swap.sh[828]: Created a new DOS disklabel with disk identifier 0x9afa9268.
enable-swap.sh[828]: Command (m for help): Partition type
enable-swap.sh[828]:    p   primary (0 primary, 0 extended, 4 free)
enable-swap.sh[828]:    e   extended (container for logical partitions)
enable-swap.sh[828]: Select (default p): Partition number (1-4, default 1): First sector (2048-335544319, default 2048): Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-335544319, default 3>
enable-swap.sh[828]: Created a new partition 1 of type 'Linux' and of size 16 GiB.
enable-swap.sh[828]: Command (m for help): Partition type
enable-swap.sh[828]:    p   primary (1 primary, 0 extended, 3 free)
enable-swap.sh[828]:    e   extended (container for logical partitions)
enable-swap.sh[828]: Select (default p): Partition number (2-4, default 2): First sector (33556480-335544319, default 33556480): Last sector, +/-sectors or +/-size{K,M,G,T,P} (33556480-33554431>
enable-swap.sh[828]: Created a new partition 2 of type 'Linux' and of size 144 GiB.
enable-swap.sh[828]: Command (m for help): The partition table has been altered.
enable-swap.sh[828]: Calling ioctl() to re-read partition table.
enable-swap.sh[828]: Syncing disks.
enable-swap.sh[719]: + mkfs.ext4 /dev/nvme1n1p1
enable-swap.sh[832]: mke2fs 1.46.5 (30-Dec-2021)
enable-swap.sh[832]: The file /dev/nvme1n1p1 does not exist and no size was specified.
systemd[1]: enable-swap.service: Main process exited, code=exited, status=1/FAILURE

Our guess with @praiskup is that we may need to run partprobe or while test -n /dev/nvme1n1p1; do sleep 1; done after fdisk in enable-swap.sh.

The problem is, that we copy the enable-swap.sh script, and enable-swap.service and bake them to the builder image. We also systemctl enable the service, so that it is run when booting the builder. As a consequence, there seems to be no way of fixing the enable-swap.sh without rebuilding the builder image.

This issue might need to be done as a part of #3036

FrostyX commented 10 months ago

This happened when I executed the script manually

[root@aws-aarch64-spot-dev-00063572-20231214-195213 ~]# enable-swap.sh
+ set -e
+ part_suffix=p
+ swap_device=
+ test -f /config/resalloc-vars.sh
+ grep POWER9 /proc/cpuinfo
+ test -e /dev/xvda1
+ test -e /dev/nvme1n1
+ swap_device=/dev/nvme1n1
+ test -n /dev/nvme1n1
+ systemctl unmask tmp.mount
+ systemctl start tmp.mount
+ echo 'n
p

+16G
n
p
2

w
'
+ fdisk /dev/nvme1n1

Welcome to fdisk (util-linux 2.38.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Command (m for help): All space for primary partitions is in use.

Command (m for help): Disk /dev/nvme1n1: 160 GiB, 171798691840 bytes, 335544320 sectors
Disk model: Amazon Elastic Block Store
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x9afa9268

Device         Boot    Start       End   Sectors  Size Id Type
/dev/nvme1n1p1          2048  33556479  33554432   16G 83 Linux
/dev/nvme1n1p2      33556480 335544319 301987840  144G 83 Linux

Command (m for help):

Command (m for help):

Command (m for help): +: unknown command

Command (m for help):
All space for primary partitions is in use.

Command (m for help): Disk /dev/nvme1n1: 160 GiB, 171798691840 bytes, 335544320 sectors
Disk model: Amazon Elastic Block Store
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x9afa9268

Device         Boot    Start       End   Sectors  Size Id Type
/dev/nvme1n1p1          2048  33556479  33554432   16G 83 Linux
/dev/nvme1n1p2      33556480 335544319 301987840  144G 83 Linux

Command (m for help): 2: unknown command

Command (m for help):

Command (m for help):

Command (m for help):
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

+ mkfs.ext4 /dev/nvme1n1p1
mke2fs 1.46.5 (30-Dec-2021)
Creating filesystem with 4194304 4k blocks and 1048576 inodes
Filesystem UUID: 0df93fa3-49b1-4b4d-bf59-c4a1982f7828
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

+ mount /dev/nvme1n1p1 /var/lib/copr-rpmbuild
+ mkdir /var/lib/copr-rpmbuild/results
+ mkdir /var/lib/copr-rpmbuild/workspace
+ rpm --setperms copr-rpmbuild
+ rpm --setugids copr-rpmbuild
+ mkswap /dev/nvme1n1p2
Setting up swapspace version 1, size = 144 GiB (154617769984 bytes)
no label, UUID=d3991c50-de8e-42f0-a0bf-e5fb50c9b21a
+ swapon /dev/nvme1n1p2
[root@aws-aarch64-spot-dev-00063572-20231214-195213 ~]# echo $?
0