Closed DWSR closed 1 year ago
cehoffman
commented
1 year ago I'd like to add the requirements for encrypted drives also overlaps with the requirements to run multiple OSDs per device which is needed to the expected performance from NVMe clusters.
smira
commented
1 year ago What exactly is missing?
cehoffman
commented
1 year ago The LVM2 tools in the base Talos image. The OSD prepare step will try to nsenter to the host and execute LVM commands like vgcreate.
https://rook.io/docs/rook/v1.11/Getting-Started/Prerequisites/prerequisites/#lvm-package
frezbo
commented
1 year ago The LVM2 tools in the base Talos image. The OSD prepare step will try to nsenter to the host and execute LVM commands like vgcreate.
https://rook.io/docs/rook/v1.11/Getting-Started/Prerequisites/prerequisites/#lvm-package
vgcreate should already be there I guess, an example here: https://github.com/siderolabs/talos/blob/main/internal/app/machined/pkg/runtime/v1alpha1/v1alpha1_sequencer_tasks.go#L2199
frezbo
commented
1 year ago The LVM2 tools in the base Talos image. The OSD prepare step will try to nsenter to the host and execute LVM commands like vgcreate. https://rook.io/docs/rook/v1.11/Getting-Started/Prerequisites/prerequisites/#lvm-package
vgcreateshould already be there I guess, an example here: https://github.com/siderolabs/talos/blob/main/internal/app/machined/pkg/runtime/v1alpha1/v1alpha1_sequencer_tasks.go#L2199
from the docs we already run vgchange
cehoffman
commented
1 year ago I didn't get too deep into the error than seeing it couldn't find some LVM commands in the logs. On searching it seemed to cross over and match this issue as well, https://github.com/rook/rook/issues/12012
Perhaps it is a bug within Rook.
smira
commented
1 year ago It would be nice to provide a short reproducer, e.g. with helm install to trigger the problem. That would help to get the issue resolved.
cehoffman
commented
1 year ago It took a bit to get the system back to state where I could retry multiple OSDs per device, but I got there and encountered the error again. This is the logs from the OSD prepare job where I'm trying to only set the nvme1n1 device to have 4 OSDs.
The core error appears to happen on these statements.
2023-06-26 14:25:48.385421 D | exec: Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts vgcreate --force --yes ceph-a6df2450-88a1-4fbb-be8f-9c91664c2900 /dev/nvme1n1
2023-06-26 14:25:48.385426 D | exec: stdout: Physical volume "/dev/nvme1n1" successfully created.
2023-06-26 14:25:48.385432 D | exec: stdout: Volume group "ceph-a6df2450-88a1-4fbb-be8f-9c91664c2900" successfully created
2023-06-26 14:25:48.385438 D | exec: Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts lvcreate --yes -l 61047 -n osd-block-3fdec961-415a-434b-ac5b-d344f9916fe9 ceph-a6df2450-88a1-4fbb-be8f-9c91664c2900
2023-06-26 14:25:48.385443 D | exec: stderr: Command failed with status code 5.I tried running the lvcreate command manually with extra logging (-vvvv) and this is what I got.
The important piece from these logs is where it tries to create a directory on the root filesystem and then unwinds.
14:50:39.377908 lvcreate[140414] device_mapper/libdm-file.c:46 Creating directory "/etc/lvm/archive"
14:50:39.377949 lvcreate[140414] device_mapper/libdm-file.c:102 <backtrace>This is where the command fails because the root filesystem is read-only. At this point I think we know what the problem is, but I don't know what the right solution for Talos to this issue is.
mkdir /rootfs/etc/lvm/archive
mkdir: cannot create directory '/rootfs/etc/lvm/archive': Read-only file systemEdit:
After a bit more reading on LVM, it seems the /etc/lvm/lvm.conf file should be modified in the base image to reflect one of these options.
Disable backup and archive.
backup {
# Configuration option backup/backup.
# Maintain a backup of the current metadata configuration.
# Think very hard before turning this off!
backup = 0
# Configuration option backup/archive.
# Maintain an archive of old metadata configurations.
# Think very hard before turning this off.
archive = 0
}or change the location of those features to a persistent location.
backup {
# Configuration option backup/backup_dir.
# Location of the metadata backup files.
# Remember to back up this directory regularly!
backup_dir = "/var/lib/lvm/backup"
# Configuration option backup/archive_dir.
# Location of the metdata archive files.
# Remember to back up this directory regularly!
archive_dir = "/var/lib/lvm/archive"
}For those coming across this, for now I have chosen to get past the LVM issue by adding a file overwrite to the machine config for /etc/lvm/lvm.conf. As I understand the configuration file, it is simply a documentation file with all the defaults of the tool set/commented, so you should be able to use this machineconfig patch setting. I have so far only tried using the full configuration file with the two settings changed though.
machine:
files:
- op: overwrite
path: /etc/lvm/lvm.conf
permissions: 0o644
content: |
backup {
backup = 0
archive = 0
}Thanks for your analysis, with Talos rootfs being read-only, and /var being ephemeral, I would say turning off backups sounds like a more sane solution to me
Feature Request
Description
Right now, Ceph/Rook run correctly on top of Talos clusters, but only if the OSDs aren't encrypted. Talos should provide support for encrypted OSDs.
https://docs.ceph.com/en/latest/ceph-volume/lvm/encryption/