search

shiftstack / dev-install

13 stars 16 forks source link

fallocate failed: No space left on device #173

Closed pierreprinetti closed 2 years ago

pierreprinetti commented 2 years ago

make osp_full fails with:

TASK [Create a backing file for ceph] *******************************************************************************************************************************************************************************
fatal: [standalone]: FAILED! => {"changed": true, "cmd": ["/usr/bin/fallocate", "-l", "100G", "/var/lib/ceph-osd.img"], "delta": "0:00:00.003927", "end": "2021-11-19 22:49:11.211600", "msg": "non-zero return code", "rc": 1, "start": "2021-11-19 22:49:11.207673", "stderr": "fallocate: fallocate failed: No space left on device", "stderr_lines": ["fallocate: fallocate failed: No space left on device"], "stdout": "", "stdout_lines": []}

After that, with no modification, running make osp_full again makes it go a bit further then fails with:

TASK [Create volume group for ceph] *********************************************************************************************************************************************************************************
fatal: [standalone]: FAILED! => {"changed": false, "err": "  Cannot use /dev/loop1: device is too small (pv_min_size)\n", "msg": "Creating physical volume '/dev/loop1' failed", "rc": 5}

This is the current output of df -h:

# df -h
Filesystem                  Size  Used Avail Use% Mounted on
devtmpfs                     63G     0   63G   0% /dev
tmpfs                        63G     0   63G   0% /dev/shm
tmpfs                        63G  9.0M   63G   1% /run
tmpfs                        63G     0   63G   0% /sys/fs/cgroup
/dev/mapper/rhel_osp1-root   70G   67G  3.5G  96% /
/dev/mapper/rhel_osp1-home  856G  6.0G  850G   1% /home
/dev/nvme0n1p2             1014M  290M  725M  29% /boot
/dev/nvme0n1p1              599M  5.8M  594M   1% /boot/efi
tmpfs                        13G     0   13G   0% /run/user/1000

RHEL was installed with automatic partitioning on a 1TB disk, using up all the available space.

mdbooth commented 2 years ago

The default partitioning scheme allocates all space to /home which is... an interesting choice.

We might want to update https://github.com/shiftstack/dev-install/blob/be1293b25af4b2918f7898b1ec56b6f2de88ea14/README.md?plain=1#L94-L103 to take account of us now using fallocate, but I don't think we're going to fix this. In my previous project we had some ansible to automatically delete /home and reallocate its space to /, but it's gross and honestly the best approach is to not do that in the first place.

pierreprinetti commented 2 years ago

Thanks @mdbooth. I hadn’t read the « sizing » section of README as a requirement, which in hindsight it obviously is