search

siderolabs / talos

Talos Linux is a modern Linux distribution built for Kubernetes.
https://www.talos.dev
Mozilla Public License 2.0
6.46k stars 516 forks source link

LVM device mapper not appearing talosctl disks or in privileged pod #7677

Closed DreamingRaven closed 1 year ago

DreamingRaven commented 1 year ago

Bug Report

LVS do not show in privileged pod for rook data removal as per https://www.talos.dev/v1.5/kubernetes-guides/configuration/ceph-with-rook/#talos-linux-rook-metadata-removal. LVM devices also do not show in talosctl disks. However LVS do appear with the lvs command and are properly consumed by rook-ceph.

As an asside are there any system extensions planned for software raid? (https://www.talos.dev/v1.5/talos-guides/configuration/system-extensions/) If there are not is there any way I can contribute to this as this is the biggest problem I currently face with Talos? I note the extensions repository has no mention of raid in the issues https://github.com/siderolabs/extensions/issues?q=raid nor is it mentioned in https://github.com/siderolabs/extensions#storage

Description

As a sysadmin I would like to be able to view existing lvmraid volumes or /dev/mapper devices in talos and within pods with the correct permissions. This would allow me to better monitor and maintain software raid volumes without having to boot a seperate OS to monitor or maintain the volumes used to back rook-ceph storage.

I have a software lvm raid volume between 3 pv devices /dev/sda1 /dev/sdb1 /dev/sdc1 in the volume group data and in the lv data. This was created outside of talos and talos at some level picks it up because rook-ceph is capable of claiming this lvmraid device, and it appears in priveleged pods using lvs command. However lsblk or dd do not pick up this volume!

This lv is created using pvcreate /dev/sda1 /dev/sdb1 /dev/sdc1 && vgcreate data /dev/sda1 /dev/sdb1 /dev/sdc1 && lvcreate 100%FREE data -n data --type raid5 outside of talos

# worker node with lvm config
machine:
  kernel:
    modules:
    - name: dm_raid # <--- side note while it does not seem to matter one seems to be able to use both _ or - in dm_raid module
    - name: dm_mod
    - name: md_mod
    - name: raid0
    - name: raid1
    - name: raid10
    - name: raid456
    - name: rbd
    - name: nbd # <--- side note nbd seems absent
    - name: ceph
  time:
      disabled: false # Indicates if the time service is disabled for the machine.
      servers:
          - time.cloudflare.com
      bootTimeout: 2m0s # Specifies the timeout when the node time is considered to be in sync unlocking the boot sequence.
  kubelet:
    extraArgs:
      rotate-server-certificates: true
  network:
    hostname: *************
    interfaces:
    - interface: bond0
      dhcp: true
      bond:
        mode: 802.3ad
        lacpRate: fast
        xmitHashPolicy: layer3+4
        miimon: 100
        updelay: 200
        downdelay: 200
        interfaces:
        - eth0
  install:
    diskSelector:
      type: nvme

privileged pod spec:

apiVersion: v1
kind: Pod
metadata:
  name: disk-wipe
  namespace: rook-ceph
spec:
  restartPolicy: Never
  nodeName: ***********
  containers:
  - name: disk-wipe
    image: alpine
    securityContext:
      privileged: true
    command: ["/bin/sh", "-c", "apk add lvm2 lsblk && lvs && lsblk && dd if=/dev/zero bs=1M count=100 oflag=direct of=/dev/mapper/data-data && sleep 120"]

output:

> kubectl logs -n rook-ceph pod/disk-wipe
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz
(1/9) Installing libblkid (2.38.1-r8)
(2/9) Installing libmount (2.38.1-r8)
(3/9) Installing libsmartcols (2.38.1-r8)
(4/9) Installing lsblk (2.38.1-r8)
(5/9) Installing libaio (0.3.113-r1)
(6/9) Installing device-mapper-libs (2.03.21-r3)
(7/9) Installing device-mapper-event-libs (2.03.21-r3)
(8/9) Installing lvm2-libs (2.03.21-r3)
(9/9) Installing lvm2 (2.03.21-r3)
Executing busybox-1.36.1-r2.trigger
OK: 13 MiB in 24 packages
  LV   VG   Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data data rwi-a-r--- <5.46t                                    30.61           <----- present here
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0         7:0    0  51.3M  1 loop 
sda           8:0    1   2.7T  0 disk 
└─sda1        8:1    1   2.7T  0 part  <---- absent here
sdb           8:16   1   2.7T  0 disk 
└─sdb1        8:17   1   2.7T  0 part 
sdc           8:32   1   2.7T  0 disk 
└─sdc1        8:33   1   2.7T  0 part 
nvme0n1     259:0    0 931.5G  0 disk 
├─nvme0n1p1 259:1    0   100M  0 part 
├─nvme0n1p2 259:2    0     1M  0 part 
├─nvme0n1p3 259:3    0  1000M  0 part 
├─nvme0n1p4 259:4    0     1M  0 part 
├─nvme0n1p5 259:5    0   100M  0 part 
└─nvme0n1p6 259:6    0 930.3G  0 part /etc/resolv.conf
                                      /etc/hostname
                                      /dev/termination-log
                                      /etc/hosts
dd: can't open '/dev/data/data': No such file or directory
> talosctl disks --talosconfig ./talosconfig --nodes=*************
NODE             DEV            MODEL                    SERIAL                 TYPE   UUID   WWID   MODALIAS      NAME   SIZE     BUS_PATH                                                                                           SUBSYSTEM          SYSTEM_DISK
192.168.10.129   /dev/nvme0n1   Sabrent Rocket 4.0 1TB   ****************   NVME   -      -      -             -      1.0 TB   /pci0000:00/0000:00:01.1/0000:01:00.0/nvme/nvme0/nvme0n1                                           /sys/class/block   *
192.168.10.129   /dev/sda       TOSHIBA HDWD130          -                      HDD    -      -      scsi:t-0x00   -      3.0 TB   /pci0000:00/0000:00:01.2/0000:02:00.0/0000:03:09.0/0000:06:00.0/ata3/host2/target2:0:0/2:0:0:0/    /sys/class/block   
192.168.10.129   /dev/sdb       TOSHIBA HDWD130          -                      HDD    -      -      scsi:t-0x00   -      3.0 TB   /pci0000:00/0000:00:01.2/0000:02:00.0/0000:03:0a.0/0000:07:00.0/ata6/host5/target5:0:0/5:0:0:0/    /sys/class/block   
192.168.10.129   /dev/sdc       TOSHIBA HDWD130          -                      HDD    -      -      scsi:t-0x00   -      3.0 TB   /pci0000:00/0000:00:01.2/0000:02:00.0/0000:03:0a.0/0000:07:00.0/ata10/host9/target9:0:0/9:0:0:0/   /sys/class/block

Environment

smira commented 1 year ago

I don't think it has anything to do with Talos itself, it might be that you need to map /dev from the host into your container, or look into using host mount namespace, but tl;dr is that container spec doesn't allow you to do what you're looking for, but not Talos

DreamingRaven commented 1 year ago

@smira, thanks for the expedient response, ah ok, I though since other devices were accessible that this was a bug. I will look into proper mapping on the container side and report back if I have any success.

Also are there any plans for software raid extensions in talos?

frezbo commented 1 year ago

Also are there any plans for software raid extensions in talos?

No plans officially, but PR's are welcome if it just works as an extension without modifications on the Talos side

DreamingRaven commented 1 year ago

Cool thanks for letting me know, will close as not a bug. Cheers guys sorry for pestering a few times recently. (will still report back when I have a moment for the next person)