search

rook / rook

Storage Orchestration for Kubernetes
https://rook.io
Apache License 2.0
12.39k stars 2.69k forks source link

Replace OSD with metadatadevice configured #6654

Closed obi12341 closed 3 years ago

obi12341 commented 3 years ago

Is this a bug report or feature request?

Deviation from expected behavior: OSD is not created

Expected behavior: OSD should be recreated

How to reproduce it (minimal and precise): We had a failing device and want to replace it. We followed your instructions in the docs, but I think we have kind of a diffrent setup, because we use a metadatadevice. When the osd on the new device is beeing recreated ceph-volume produces an error because, /dev/sda (which is the metadatadevice) is locked. So far I understand this is the right behaviour, but can you describe a way to replace a disk with configured metadatadevice.

File(s) to submit:

Environment:

Prepare Job Log:

2020-11-17 22:33:57.569680 I | cephcmd: desired devices to configure osds: [{Name:sdb OSDsPerDevice:1 MetadataDevice:sda DatabaseSizeMB:0 DeviceClass: IsFilter:false IsDevicePathFilter:false} {Name:sdc OSDsPerDevice:1 MetadataDevice:sda DatabaseSizeMB:0 DeviceClass: IsFilter:false IsDevicePathFilter:false} {Name:sdd OSDsPerDevice:1 MetadataDevice:sda DatabaseSizeMB:0 DeviceClass: IsFilter:false IsDevicePathFilter:false} {Name:sde OSDsPerDevice:1 MetadataDevice:sda DatabaseSizeMB:0 DeviceClass: IsFilter:false IsDevicePathFilter:false} {Name:sdf OSDsPerDevice:1 MetadataDevice:sda DatabaseSizeMB:0 DeviceClass: IsFilter:false IsDevicePathFilter:false} {Name:sdg OSDsPerDevice:1 MetadataDevice:sda DatabaseSizeMB:0 DeviceClass: IsFilter:false IsDevicePathFilter:false}]
2020-11-17 22:33:57.570759 I | rookcmd: starting Rook v1.4.7 with arguments '/rook/rook ceph osd provision'
2020-11-17 22:33:57.570771 I | rookcmd: flag values: --cluster-id=34cc4f42-3c75-4f89-abf1-a47d49a070e4, --data-device-filter=, --data-device-path-filter=, --data-devices=[{"id":"sdb","storeConfig":{"osdsPerDevice":1,"metadataDevice":"sda"}},{"id":"sdc","storeConfig":{"osdsPerDevice":1,"metadataDevice":"sda"}},{"id":"sdd","storeConfig":{"osdsPerDevice":1,"metadataDevice":"sda"}},{"id":"sde","storeConfig":{"osdsPerDevice":1,"metadataDevice":"sda"}},{"id":"sdf","storeConfig":{"osdsPerDevice":1,"metadataDevice":"sda"}},{"id":"sdg","storeConfig":{"osdsPerDevice":1,"metadataDevice":"sda"}}], --drive-groups=, --encrypted-device=false, --force-format=false, --help=false, --location=, --log-flush-frequency=5s, --log-level=DEBUG, --metadata-device=, --node-name=de-her-s2-k8sm-2, --operator-image=, --osd-database-size=0, --osd-store=, --osd-wal-size=576, --osds-per-device=1, --pvc-backed-osd=false, --service-account=
2020-11-17 22:33:57.570774 I | op-mon: parsing mon endpoints: a=10.0.111.223:6789,b=10.0.98.127:6789,c=10.0.111.247:6789
2020-11-17 22:33:57.586946 I | op-osd: CRUSH location=root=default host=de-her-s2-k8sm-2
2020-11-17 22:33:57.586971 I | cephcmd: crush location of osd: root=default host=de-her-s2-k8sm-2
2020-11-17 22:33:57.586989 D | exec: Running command: nsenter --mount=/rootfs/proc/1/ns/mnt -- /usr/sbin/lvm --help
2020-11-17 22:33:57.589265 D | cephosd: failed to call nsenter. failed to execute nsenter. output: nsenter: failed to execute /usr/sbin/lvm: No such file or directory: exit status 127
2020-11-17 22:33:57.589311 D | cephosd: failed to lookup binary path "/rootfs/usr/sbin/lvm" on the host rootfs. stat /rootfs/usr/sbin/lvm: no such file or directory
2020-11-17 22:33:57.589335 D | exec: Running command: nsenter --mount=/rootfs/proc/1/ns/mnt -- /sbin/lvm --help
2020-11-17 22:33:57.605782 I | cephosd: successfully called nsenter
2020-11-17 22:33:57.605815 I | cephosd: binary "/sbin/lvm" found on the host, proceeding with osd preparation
2020-11-17 22:33:57.618989 D | cephclient: No ceph configuration override to merge as "rook-config-override" configmap is empty
2020-11-17 22:33:57.619019 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2020-11-17 22:33:57.619201 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2020-11-17 22:33:57.619435 D | cephosd: config file @ /etc/ceph/ceph.conf: [global]
fsid                = 32d81a5c-f910-4f5f-8a41-5e7aac975b23
mon initial members = a b c
mon host            = [v2:10.0.111.223:3300,v1:10.0.111.223:6789],[v2:10.0.98.127:3300,v1:10.0.98.127:6789],[v2:10.0.111.247:3300,v1:10.0.111.247:6789]
public addr         = 10.0.149.234
cluster addr        = 10.0.149.234

[client.admin]
keyring = /var/lib/rook/rook-ceph/client.admin.keyring

2020-11-17 22:33:57.619449 I | cephosd: discovering hardware
2020-11-17 22:33:57.619460 D | exec: Running command: lsblk --all --noheadings --list --output KNAME
2020-11-17 22:33:57.627698 D | exec: Running command: lsblk /dev/loop0 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:57.631601 W | inventory: skipping device "loop0". diskType is empty
2020-11-17 22:33:57.631629 D | exec: Running command: lsblk /dev/loop1 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:57.634266 W | inventory: skipping device "loop1". diskType is empty
2020-11-17 22:33:57.634289 D | exec: Running command: lsblk /dev/loop2 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:57.637686 W | inventory: skipping device "loop2". diskType is empty
2020-11-17 22:33:57.637722 D | exec: Running command: lsblk /dev/loop3 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:57.639965 W | inventory: skipping device "loop3". diskType is empty
2020-11-17 22:33:57.639987 D | exec: Running command: lsblk /dev/loop4 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:57.643456 W | inventory: skipping device "loop4". diskType is empty
2020-11-17 22:33:57.643493 D | exec: Running command: lsblk /dev/loop5 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:57.646763 W | inventory: skipping device "loop5". diskType is empty
2020-11-17 22:33:57.646808 D | exec: Running command: lsblk /dev/loop6 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:57.649977 W | inventory: skipping device "loop6". diskType is empty
2020-11-17 22:33:57.650016 D | exec: Running command: lsblk /dev/loop7 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:57.653079 W | inventory: skipping device "loop7". diskType is empty
2020-11-17 22:33:57.653112 D | exec: Running command: lsblk /dev/sda --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:57.656278 D | exec: Running command: sgdisk --print /dev/sda
2020-11-17 22:33:57.662113 D | exec: Running command: udevadm info --query=property /dev/sda
2020-11-17 22:33:57.671856 D | exec: Running command: lsblk --noheadings --pairs /dev/sda
2020-11-17 22:33:57.679162 I | inventory: skipping device "sda" because it has child, considering the child instead.
2020-11-17 22:33:57.679197 D | exec: Running command: lsblk /dev/sdb --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:57.682861 D | exec: Running command: sgdisk --print /dev/sdb
2020-11-17 22:33:57.834267 D | exec: Running command: udevadm info --query=property /dev/sdb
2020-11-17 22:33:57.845231 D | exec: Running command: lsblk --noheadings --pairs /dev/sdb
2020-11-17 22:33:57.852100 D | exec: Running command: lsblk /dev/sdc --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:57.856034 D | exec: Running command: sgdisk --print /dev/sdc
2020-11-17 22:33:57.859728 D | exec: Running command: udevadm info --query=property /dev/sdc
2020-11-17 22:33:57.869457 D | exec: Running command: lsblk --noheadings --pairs /dev/sdc
2020-11-17 22:33:57.876496 I | inventory: skipping device "sdc" because it has child, considering the child instead.
2020-11-17 22:33:57.876525 D | exec: Running command: lsblk /dev/sdd --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:57.880447 D | exec: Running command: sgdisk --print /dev/sdd
2020-11-17 22:33:57.904576 D | exec: Running command: udevadm info --query=property /dev/sdd
2020-11-17 22:33:57.915400 D | exec: Running command: lsblk --noheadings --pairs /dev/sdd
2020-11-17 22:33:57.923729 I | inventory: skipping device "sdd" because it has child, considering the child instead.
2020-11-17 22:33:57.923774 D | exec: Running command: lsblk /dev/sde --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:57.927478 D | exec: Running command: sgdisk --print /dev/sde
2020-11-17 22:33:57.931555 D | exec: Running command: udevadm info --query=property /dev/sde
2020-11-17 22:33:57.941491 D | exec: Running command: lsblk --noheadings --pairs /dev/sde
2020-11-17 22:33:57.948482 I | inventory: skipping device "sde" because it has child, considering the child instead.
2020-11-17 22:33:57.948518 D | exec: Running command: lsblk /dev/sdf --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:57.952187 D | exec: Running command: sgdisk --print /dev/sdf
2020-11-17 22:33:57.955653 D | exec: Running command: udevadm info --query=property /dev/sdf
2020-11-17 22:33:57.965006 D | exec: Running command: lsblk --noheadings --pairs /dev/sdf
2020-11-17 22:33:57.971721 I | inventory: skipping device "sdf" because it has child, considering the child instead.
2020-11-17 22:33:57.971757 D | exec: Running command: lsblk /dev/sdg --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:57.973878 D | exec: Running command: sgdisk --print /dev/sdg
2020-11-17 22:33:57.978589 D | exec: Running command: udevadm info --query=property /dev/sdg
2020-11-17 22:33:57.985781 D | exec: Running command: lsblk --noheadings --pairs /dev/sdg
2020-11-17 22:33:57.992884 I | inventory: skipping device "sdg" because it has child, considering the child instead.
2020-11-17 22:33:57.992919 D | exec: Running command: lsblk /dev/sdh --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:57.996531 D | exec: Running command: sgdisk --print /dev/sdh
2020-11-17 22:33:58.001401 D | exec: Running command: udevadm info --query=property /dev/sdh
2020-11-17 22:33:58.009829 D | exec: Running command: lsblk --noheadings --pairs /dev/sdh
2020-11-17 22:33:58.017873 I | inventory: skipping device "sdh" because it has child, considering the child instead.
2020-11-17 22:33:58.017937 D | exec: Running command: lsblk /dev/sdh1 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:58.021662 D | exec: Running command: udevadm info --query=property /dev/sdh1
2020-11-17 22:33:58.031465 D | exec: Running command: lsblk /dev/sdh2 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:58.035177 D | exec: Running command: udevadm info --query=property /dev/sdh2
2020-11-17 22:33:58.044035 D | exec: Running command: lsblk /dev/dm-0 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:58.047673 D | exec: Running command: sgdisk --print /dev/dm-0
2020-11-17 22:33:58.052566 D | exec: Running command: udevadm info --query=property /dev/dm-0
2020-11-17 22:33:58.061915 D | exec: Running command: lsblk /dev/dm-1 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:58.065518 D | exec: Running command: sgdisk --print /dev/dm-1
2020-11-17 22:33:58.069221 D | exec: Running command: udevadm info --query=property /dev/dm-1
2020-11-17 22:33:58.079291 D | exec: Running command: lsblk /dev/dm-2 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:58.082978 D | exec: Running command: sgdisk --print /dev/dm-2
2020-11-17 22:33:58.087824 D | exec: Running command: udevadm info --query=property /dev/dm-2
2020-11-17 22:33:58.097935 D | exec: Running command: lsblk /dev/dm-3 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:58.101454 D | exec: Running command: sgdisk --print /dev/dm-3
2020-11-17 22:33:58.106230 D | exec: Running command: udevadm info --query=property /dev/dm-3
2020-11-17 22:33:58.112052 D | exec: Running command: lsblk /dev/dm-4 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:58.115726 D | exec: Running command: sgdisk --print /dev/dm-4
2020-11-17 22:33:58.118883 D | exec: Running command: udevadm info --query=property /dev/dm-4
2020-11-17 22:33:58.128900 D | exec: Running command: lsblk /dev/dm-5 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:58.132505 D | exec: Running command: sgdisk --print /dev/dm-5
2020-11-17 22:33:58.137618 D | exec: Running command: udevadm info --query=property /dev/dm-5
2020-11-17 22:33:58.148492 D | exec: Running command: lsblk /dev/dm-6 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:58.152226 D | exec: Running command: sgdisk --print /dev/dm-6
2020-11-17 22:33:58.156881 D | exec: Running command: udevadm info --query=property /dev/dm-6
2020-11-17 22:33:58.166630 D | exec: Running command: lsblk /dev/dm-7 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:58.170430 D | exec: Running command: sgdisk --print /dev/dm-7
2020-11-17 22:33:58.175758 D | exec: Running command: udevadm info --query=property /dev/dm-7
2020-11-17 22:33:58.184756 D | exec: Running command: lsblk /dev/dm-8 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:58.188360 D | exec: Running command: sgdisk --print /dev/dm-8
2020-11-17 22:33:58.191231 D | exec: Running command: udevadm info --query=property /dev/dm-8
2020-11-17 22:33:58.199382 D | exec: Running command: lsblk /dev/dm-9 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:58.203083 D | exec: Running command: sgdisk --print /dev/dm-9
2020-11-17 22:33:58.207837 D | exec: Running command: udevadm info --query=property /dev/dm-9
2020-11-17 22:33:58.219105 D | exec: Running command: lsblk /dev/dm-10 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:58.221637 D | exec: Running command: sgdisk --print /dev/dm-10
2020-11-17 22:33:58.226238 D | exec: Running command: udevadm info --query=property /dev/dm-10
2020-11-17 22:33:58.232321 D | exec: Running command: lsblk /dev/dm-11 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:58.236013 D | exec: Running command: sgdisk --print /dev/dm-11
2020-11-17 22:33:58.239204 D | exec: Running command: udevadm info --query=property /dev/dm-11
2020-11-17 22:33:58.249581 D | exec: Running command: lsblk /dev/dm-12 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:58.253119 D | exec: Running command: sgdisk --print /dev/dm-12
2020-11-17 22:33:58.255559 D | exec: Running command: udevadm info --query=property /dev/dm-12
2020-11-17 22:33:58.265937 D | inventory: discovered disks are [0xc0001778c0 0xc0001947e0 0xc0001c26c0 0xc0001c2b40 0xc000690480 0xc000690b40 0xc000690ea0 0xc0004cd320 0xc000690fc0 0xc000691320 0xc000691680 0xc0006919e0 0xc0001f5b00 0xc0001c2240 0xc0001c25a0 0xc0004cc000]
2020-11-17 22:33:58.265962 I | cephosd: creating and starting the osds
2020-11-17 22:33:58.274941 D | cephosd: No Drive Groups configured.
2020-11-17 22:33:58.275007 D | cephosd: desiredDevices are [{Name:sdb OSDsPerDevice:1 MetadataDevice:sda DatabaseSizeMB:0 DeviceClass: IsFilter:false IsDevicePathFilter:false} {Name:sdc OSDsPerDevice:1 MetadataDevice:sda DatabaseSizeMB:0 DeviceClass: IsFilter:false IsDevicePathFilter:false} {Name:sdd OSDsPerDevice:1 MetadataDevice:sda DatabaseSizeMB:0 DeviceClass: IsFilter:false IsDevicePathFilter:false} {Name:sde OSDsPerDevice:1 MetadataDevice:sda DatabaseSizeMB:0 DeviceClass: IsFilter:false IsDevicePathFilter:false} {Name:sdf OSDsPerDevice:1 MetadataDevice:sda DatabaseSizeMB:0 DeviceClass: IsFilter:false IsDevicePathFilter:false} {Name:sdg OSDsPerDevice:1 MetadataDevice:sda DatabaseSizeMB:0 DeviceClass: IsFilter:false IsDevicePathFilter:false}]
2020-11-17 22:33:58.275024 D | cephosd: context.Devices are [0xc0001778c0 0xc0001947e0 0xc0001c26c0 0xc0001c2b40 0xc000690480 0xc000690b40 0xc000690ea0 0xc0004cd320 0xc000690fc0 0xc000691320 0xc000691680 0xc0006919e0 0xc0001f5b00 0xc0001c2240 0xc0001c25a0 0xc0004cc000]
2020-11-17 22:33:58.275040 D | exec: Running command: lsblk /dev/sdb --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-17 22:33:58.277274 D | exec: Running command: ceph-volume inventory --format json /dev/sdb
2020-11-17 22:33:59.000458 I | cephosd: device "sdb" is available.
2020-11-17 22:33:59.000490 I | cephosd: "sdb" found in the desired devices
2020-11-17 22:33:59.000499 I | cephosd: device "sdb" is selected by the device filter/name "sdb"
2020-11-17 22:33:59.000513 I | cephosd: skipping device "sdh1" because it contains a filesystem "vfat"
2020-11-17 22:33:59.000523 I | cephosd: skipping device "sdh2" because it contains a filesystem "LVM2_member"
2020-11-17 22:33:59.000530 I | cephosd: skipping 'dm' device "dm-0"
2020-11-17 22:33:59.000537 I | cephosd: skipping 'dm' device "dm-1"
2020-11-17 22:33:59.000543 I | cephosd: skipping 'dm' device "dm-2"
2020-11-17 22:33:59.000549 I | cephosd: skipping 'dm' device "dm-3"
2020-11-17 22:33:59.000555 I | cephosd: skipping 'dm' device "dm-4"
2020-11-17 22:33:59.000561 I | cephosd: skipping 'dm' device "dm-5"
2020-11-17 22:33:59.000567 I | cephosd: skipping 'dm' device "dm-6"
2020-11-17 22:33:59.000573 I | cephosd: skipping 'dm' device "dm-7"
2020-11-17 22:33:59.000580 I | cephosd: skipping 'dm' device "dm-8"
2020-11-17 22:33:59.000586 I | cephosd: skipping 'dm' device "dm-9"
2020-11-17 22:33:59.000601 I | cephosd: skipping 'dm' device "dm-10"
2020-11-17 22:33:59.000607 I | cephosd: skipping 'dm' device "dm-11"
2020-11-17 22:33:59.000617 I | cephosd: skipping 'dm' device "dm-12"
2020-11-17 22:33:59.000777 I | cephosd: configuring osd devices: {"Entries":{"sdb":{"Data":-1,"Metadata":null,"Config":{"Name":"sdb","OSDsPerDevice":1,"MetadataDevice":"sda","DatabaseSizeMB":0,"DeviceClass":"","IsFilter":false,"IsDevicePathFilter":false},"PersistentDevicePaths":["/dev/disk/by-id/ata-ST2000NX0423_W462KLVD","/dev/disk/by-path/pci-0000:17:00.0-scsi-0:0:3:0","/dev/disk/by-id/wwn-0x5000c500c4c9e6c8"]}}}
2020-11-17 22:33:59.000861 I | cephclient: getting or creating ceph auth key "client.bootstrap-osd"
2020-11-17 22:33:59.001163 D | exec: Running command: ceph auth get-or-create-key client.bootstrap-osd mon allow profile bootstrap-osd --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --name=client.admin --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/353626153
2020-11-17 22:33:59.418983 I | cephosd: configuring new device sdb
2020-11-17 22:33:59.419020 I | cephosd: using sda as metadataDevice for device /dev/sdb and let ceph-volume lvm batch decide how to create volumes
2020-11-17 22:33:59.419081 D | exec: Running command: stdbuf -oL ceph-volume --log-path /tmp/ceph-log lvm batch --prepare --bluestore --yes --osds-per-device 1 /dev/sdb --db-devices /dev/sda --report
2020-11-17 22:34:00.624165 D | exec: Traceback (most recent call last):
2020-11-17 22:34:00.624244 D | exec:   File "/usr/sbin/ceph-volume", line 11, in <module>
2020-11-17 22:34:00.624254 D | exec:     load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()
2020-11-17 22:34:00.624262 D | exec:   File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 40, in __init__
2020-11-17 22:34:00.624271 D | exec:     self.main(self.argv)
2020-11-17 22:34:00.624279 D | exec:   File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
2020-11-17 22:34:00.624286 D | exec:     return f(*a, **kw)
2020-11-17 22:34:00.624304 D | exec:   File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 151, in main
2020-11-17 22:34:00.624314 D | exec:     terminal.dispatch(self.mapper, subcommand_args)
2020-11-17 22:34:00.624323 D | exec:   File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
2020-11-17 22:34:00.624332 D | exec:     instance.main()
2020-11-17 22:34:00.624341 D | exec:   File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 42, in main
2020-11-17 22:34:00.624350 D | exec:     terminal.dispatch(self.mapper, self.argv)
2020-11-17 22:34:00.624360 D | exec:   File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
2020-11-17 22:34:00.624369 D | exec:     instance.main()
2020-11-17 22:34:00.624378 D | exec:   File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
2020-11-17 22:34:00.624387 D | exec:     return func(*a, **kw)
2020-11-17 22:34:00.624396 D | exec:   File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 322, in main
2020-11-17 22:34:00.624405 D | exec:     self._get_explicit_strategy()
2020-11-17 22:34:00.624415 D | exec:   File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 332, in _get_explicit_strategy
2020-11-17 22:34:00.624423 D | exec:     self._filter_devices()
2020-11-17 22:34:00.624431 D | exec:   File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 385, in _filter_devices
2020-11-17 22:34:00.624438 D | exec:     raise RuntimeError(err.format(len(devs) - len(usable)))
2020-11-17 22:34:00.624446 D | exec: RuntimeError: 1 devices were filtered in non-interactive mode, bailing out
failed to configure devices: failed to initialize devices: failed ceph-volume report: exit status 1

Cluster:

Namespace:    rook-ceph
Labels:       <none>
Annotations:  <none>
API Version:  ceph.rook.io/v1
Kind:         CephCluster
Metadata:
  Creation Timestamp:  2020-04-06T13:25:24Z
  Finalizers:
    cephcluster.ceph.rook.io
  Generation:        248290
  Resource Version:  118450828
  Self Link:         /apis/ceph.rook.io/v1/namespaces/rook-ceph/cephclusters/rook-ceph
  UID:               34cc4f42-3c75-4f89-abf1-a47d49a070e4
Spec:
  Ceph Version:
    Image:  ceph/ceph:v15.2.5-20200916
  Cleanup Policy:
  Crash Collector:
    Disable:  false
  Dashboard:
    Enabled:           true
    Ssl:               true
  Data Dir Host Path:  /var/lib/rook
  Disruption Management:
    Machine Disruption Budget Namespace:  openshift-machine-api
    Osd Maintenance Timeout:              30
  External:
    Enable:  false
  Mgr:
  Mon:
    Count:  3
  Monitoring:
    Rules Namespace:  rook-ceph
  Network:
    Host Network:  false
    Provider:
    Selectors:     <nil>
  Placement:
    All:
      Tolerations:
        Effect:  NoSchedule
        Key:     node-role.kubernetes.io/master
  Rbd Mirroring:
    Workers:                               0
  Remove OS Ds If Out And Safe To Remove:  false
  Storage:
    Nodes:
      Devices:
        Config:
          Metadata Device:  sda
        Name:               sdb
        Config:
          Metadata Device:  sda
        Name:               sdc
        Config:
          Metadata Device:  sda
        Name:               sdd
        Config:
          Metadata Device:  sda
        Name:               sde
        Config:
          Metadata Device:  sda
        Name:               sdf
        Config:
          Metadata Device:  sda
        Name:               sdg
      Name:                 de-her-s1-k8sm-1
      Resources:
      Devices:
        Config:
          Metadata Device:  sda
        Name:               sdb
        Config:
          Metadata Device:  sda
        Name:               sdc
        Config:
          Metadata Device:  sda
        Name:               sdd
        Config:
          Metadata Device:  sda
        Name:               sde
        Config:
          Metadata Device:  sda
        Name:               sdf
        Config:
          Metadata Device:  sda
        Name:               sdg
      Name:                 de-her-s2-k8sm-1
      Resources:
      Devices:
        Config:
          Metadata Device:  sda
        Name:               sdb
        Config:
          Metadata Device:  sda
        Name:               sdc
        Config:
          Metadata Device:  sda
        Name:               sdd
        Config:
          Metadata Device:  sda
        Name:               sde
        Config:
          Metadata Device:  sda
        Name:               sdf
        Config:
          Metadata Device:  sda
        Name:               sdg
      Name:    de-her-s2-k8sm-2
      Resources:
    Storage Class Device Sets:  <nil>
    Use All Devices:            false
Status:
  Ceph:
    Health:           HEALTH_OK
    Last Changed:     2020-11-17T22:34:15Z
    Last Checked:     2020-11-17T22:38:17Z
    Previous Health:  HEALTH_WARN
  Conditions:
    Last Heartbeat Time:   2020-11-17T22:33:57Z
    Last Transition Time:  2020-10-02T09:59:34Z
    Message:               Failed to create cluster
    Reason:                ClusterFailure
    Status:                True
    Type:                  Failure
    Last Heartbeat Time:   2020-10-02T09:59:34Z
    Last Transition Time:  2020-10-02T09:59:34Z
    Status:                False
    Type:                  Ignored
    Last Heartbeat Time:   2020-10-02T09:59:34Z
    Last Transition Time:  2020-10-02T09:59:34Z
    Status:                False
    Type:                  Upgrading
    Last Heartbeat Time:   2020-11-17T22:37:16Z
    Last Transition Time:  2020-10-02T10:00:08Z
    Message:               Cluster progression is completed
    Reason:                ProgressingCompleted
    Status:                False
    Type:                  Progressing
    Last Heartbeat Time:   2020-10-02T10:00:42Z
    Last Transition Time:  2020-10-02T10:00:42Z
    Message:               Cluster created successfully
    Reason:                ClusterCreated
    Status:                True
    Type:                  Ready
    Last Heartbeat Time:   2020-10-02T10:01:49Z
    Last Transition Time:  2020-10-02T10:01:49Z
    Message:               Cluster progression is completed
    Reason:                ProgressingCompleted
    Status:                False
    Last Heartbeat Time:   2020-10-02T11:35:31Z
    Last Transition Time:  2020-10-02T10:41:15Z
    Message:               Cluster updating is completed
    Reason:                UpdateCompleted
    Status:                False
    Type:                  Updating
  Message:                 Cluster created successfully
  Phase:                   Ready
  State:                   Created
  Storage:
    Device Classes:
      Name:  hdd
  Version:
    Image:    ceph/ceph:v15.2.5-20200916
    Version:  15.2.5-0
Events:       <none>
Anganthier commented 3 years ago

Ping - sounds familiar.

chrono2002 commented 3 years ago

same here

2020-11-27 18:04:14.160738 D | exec: Running command: stdbuf -oL ceph-volume --log-path /tmp/ceph-log lvm batch --prepare --bluestore --yes --osds-per-device 1 /dev/sdb /dev/sda --db-devices /dev/md0p3 --report 2020-11-27 18:04:21.406855 D | exec: --> passed data devices: 2 physical, 0 LVM 2020-11-27 18:04:21.406920 D | exec: --> relative data size: 1.0 2020-11-27 18:04:21.407166 D | exec: --> passed block_db devices: 0 physical, 1 LVM 2020-11-27 18:04:21.408214 D | exec: Traceback (most recent call last): 2020-11-27 18:04:21.408238 D | exec: File "/usr/sbin/ceph-volume", line 9, in 2020-11-27 18:04:21.408244 D | exec: load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2020-11-27 18:04:21.408250 D | exec: File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 39, in init 2020-11-27 18:04:21.408254 D | exec: self.main(self.argv) 2020-11-27 18:04:21.408258 D | exec: File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2020-11-27 18:04:21.408276 D | exec: return f(*a, *kw) 2020-11-27 18:04:21.408282 D | exec: File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 150, in main 2020-11-27 18:04:21.408324 D | exec: terminal.dispatch(self.mapper, subcommand_args) 2020-11-27 18:04:21.408335 D | exec: File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2020-11-27 18:04:21.408370 D | exec: instance.main() 2020-11-27 18:04:21.408377 D | exec: File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/main.py", line 42, in main 2020-11-27 18:04:21.408385 D | exec: terminal.dispatch(self.mapper, self.argv) 2020-11-27 18:04:21.408400 D | exec: File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2020-11-27 18:04:21.408432 D | exec: instance.main() 2020-11-27 18:04:21.408441 D | exec: File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 16, in is_root 2020-11-27 18:04:21.408459 D | exec: return func(a, **kw) 2020-11-27 18:04:21.408467 D | exec: File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/batch.py", line 402, in main 2020-11-27 18:04:21.408538 D | exec: plan = self.get_plan(self.args) 2020-11-27 18:04:21.408551 D | exec: File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/batch.py", line 440, in get_plan 2020-11-27 18:04:21.408618 D | exec: args.wal_devices) 2020-11-27 18:04:21.408637 D | exec: File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/batch.py", line 472, in get_deployment_layout 2020-11-27 18:04:21.408712 D | exec: fast_type) 2020-11-27 18:04:21.408724 D | exec: File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/batch.py", line 509, in fast_allocations 2020-11-27 18:04:21.408804 D | exec: ret.extend(get_lvm_fast_allocs(lvm_devs)) 2020-11-27 18:04:21.408816 D | exec: File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/batch.py", line 147, in get_lvm_fast_allocs 2020-11-27 18:04:21.408844 D | exec: d.used_by_ceph] 2020-11-27 18:04:21.408868 D | exec: IndexError: list index out of range failed to configure devices: failed to initialize devices: failed ceph-volume report: exit status 1

chrono2002 commented 3 years ago

works without "--yes" in ceph-volume params now trying to build patched version

github-actions[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

obi12341 commented 3 years ago

still no solution for this?

travisn commented 3 years ago

There is a related discussion in #7121. There is a PR in progress in the ceph repo that at a glance will help with this scenario as well.

github-actions[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

github-actions[bot] commented 3 years ago

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

lyind commented 3 years ago

Just in case, I had this issue with LVM volumes for --db-devices (filled by rook's metadataDevice).

Looking through the Ceph ceph-volume source code, in particular class ceph_volume.util.device.Device, I found specifying the logical volume in "vg/lv" format works.

So use for example metadataDevice: "vg-metadata-0/metadata-0-2" (instead of metadataDevice: "/dev/vg-metadata-0/metadata-0-2" or metadataDevice: "/dev/dm-2").

mikabytes commented 2 years ago

@lyind this comment to the rescue! Been trying to find a solution for this for many hours. This trick did it, thanks!

kutovoy commented 2 years ago

I had a similar but different error with IndexError (I was trying to install OSD to lvm with pre-existing lv using ceph-ansible, branch for ceph octopus), I fixed it by using OSD drives config like this (example):

lvm_volumes:

@lyind's answer made me re-think on how to specify vg/lv pairs. Maybe this will help someone since this issue is coming up in google search