search

ceph / ceph-ansible

Ansible playbooks to deploy Ceph, the distributed filesystem.
Apache License 2.0
1.69k stars 1.01k forks source link

need workaround for ceph bug, for OSD nodes and ceph-volume batch mode #4855

Closed ppbrown closed 4 years ago

ppbrown commented 4 years ago

ansible 2.8 ceph nautilus

# ceph-volume lvm --report --bluestore --yes batch /dev/sda /dev/sdb /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdl
--> Aborting because strategy changed from bluestore.MixedType to bluestore.SingleType after filtering

This is similar to issue #4748 In THAT issue, someone commented along the lines of, "well yes its expected to fail, if you initially create a bunch of OSDs, then call ceph-volume with a different set of devices".

Except that I'm NOT using a different set of devices. its the exact same set that was first set up on the machine.

So apparently, theres some kind of bug in ceph, where it happily sets up a set of mixed ssd/hdd devices the first time in batch mode... but then --report will be broken ever after ?

Unless I'm misunderstanding, and the previous bug report is saying that ALL machines in the cluster must have the EXACT SAME device paths as each other. Which would be insane, given that linux can randomly rewrite device paths, and /dev/sda can become /dev/sdl after a reboot.

Right now, the only workaround I would seem to have, is: Add a machine one time to the [osds] ansible host group. get it configured... then take it OUT of the group.

Would be nice to have a better alternative.

(or is that kind of expected ceph-ansible behaviour? to create a dynamic inventory file, and only put in it, what needs to be (re)configured this time ? )

dsavineau commented 4 years ago

Except that I'm NOT using a different set of devices. its the exact same set that was first set up on the machine.

Did you zap/purge/destroy (whatever you use) the devices before the initial deployment ?

So apparently, theres some kind of bug in ceph, where it happily sets up a set of mixed ssd/hdd devices the first time in batch mode... but then --report will be broken ever after ?

Can you share the /var/log/ceph/ceph-volume.log output when running the command ?

On my side I can rerun the ceph-volume batch command without any issue either manually or via new ceph-ansible execution.

# ceph-volume lvm --report --bluestore --yes batch /dev/sdb /dev/sdc
--> All devices are already used by ceph. No OSDs will be created.

Unless I'm misunderstanding, and the previous bug report is saying that ALL machines in the cluster must have the EXACT SAME device paths as each other.

That's not true. If you don't have the same devices configuration on all OSD nodes then instead of using the group_vars you can use the host_vars.

Right now, the only workaround I would seem to have, is: Add a machine one time to the [osds] ansible host group. get it configured... then take it OUT of the group.

You don't need to do that.

Again, share the logs, configuration or anything that could help.

BTW before using stuff like need workaround for ceph bug in the issue title let's investigate to find the root cause.

ppbrown commented 4 years ago

For your ceph-volume test above, was one of those devicdes SSD and the other one HDD ? Thats specifically what needs to be tested.

on the brighter side, looks like I can share the ceph-volume log, after I filtered out the keys. Im just attaching fot the last 24 hours. which is large enough.

ceph-volume.log

ppbrown commented 4 years ago

Okay. I happen to have had to recreate my test cluster from scratch. so here is explicit demo output demonstrating the bug in ceph, for ceph-volume, batch mode,and mixed SSD/HDD reporting. batch --report works fine... until after the devices are actually deployed in ceph.. then it blows up.

[root@XHOSTNAMEX deploy]#
[root@XHOSTNAMEX deploy]# ceph -s
  cluster:
    id:     xxxxxxxx-6d9f-4fbc-9c22-6c189efa1b3f
    health: HEALTH_OK

  services:
    mon: 1 daemons, quorum XHOSTNAMEX (age 8m)
    mgr: XHOSTNAMEX(active, since 8m)
    osd: 3 osds: 3 up (since 17s), 3 in (since 17s)

  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   3.0 GiB used, 5.5 TiB / 5.5 TiB avail
    pgs:

[root@XHOSTNAMEX deploy]# cat /tmp/devs
   /dev/sda /dev/sdb /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh

[root@XHOSTNAMEX deploy]# ceph-volume lvm batch --report `cat /tmp/devs`

Total OSDs: 3

Solid State VG:
  Targets:   block.db                  Total size: 220.00 GB
  Total LVs: 3                         Size per LV: 73.33 GB
  Devices:   /dev/sda, /dev/sdb

  Type            Path                                                    LV Size         % of device
----------------------------------------------------------------------------------------------------
  [data]          /dev/sdd                                                1.82 TB         100%
  [block.db]      vg: vg/lv                                               73.33 GB        33%
----------------------------------------------------------------------------------------------------
  [data]          /dev/sde                                                1.82 TB         100%
  [block.db]      vg: vg/lv                                               73.33 GB        33%
----------------------------------------------------------------------------------------------------
  [data]          /dev/sdf                                                1.82 TB         100%
  [block.db]      vg: vg/lv                                               73.33 GB        33%
[root@XHOSTNAMEX deploy]# ceph-volume lvm batch `cat /tmp/devs`

Total OSDs: 3

Solid State VG:
  Targets:   block.db                  Total size: 220.00 GB
  Total LVs: 3                         Size per LV: 73.33 GB
  Devices:   /dev/sda, /dev/sdb

  Type            Path                                                    LV Size         % of device
----------------------------------------------------------------------------------------------------
  [data]          /dev/sdd                                                1.82 TB         100%
  [block.db]      vg: vg/lv                                               73.33 GB        33%
----------------------------------------------------------------------------------------------------
  [data]          /dev/sde                                                1.82 TB         100%
  [block.db]      vg: vg/lv                                               73.33 GB        33%
----------------------------------------------------------------------------------------------------
  [data]          /dev/sdf                                                1.82 TB         100%
  [block.db]      vg: vg/lv                                               73.33 GB        33%
--> The above OSDs would be created if the operation continues
--> do you want to proceed? (yes/no) yes
Running command: /usr/sbin/vgcreate -s 1G --force --yes ceph-block-499b12ab-639c-4428-b833-186f346c1fef /dev/sdd
 stdout: Physical volume "/dev/sdd" successfully created.
 stdout: Volume group "ceph-block-499b12ab-639c-4428-b833-186f346c1fef" successfully created
Running command: /usr/sbin/vgcreate -s 1G --force --yes ceph-block-073e90c8-9440-4e97-8b5e-121ea710219c /dev/sde
 stdout: Physical volume "/dev/sde" successfully created.
 stdout: Volume group "ceph-block-073e90c8-9440-4e97-8b5e-121ea710219c" successfully created
Running command: /usr/sbin/vgcreate -s 1G --force --yes ceph-block-042c6671-6205-40d0-962c-aec63721b29c /dev/sdf
 stdout: Physical volume "/dev/sdf" successfully created.
 stdout: Volume group "ceph-block-042c6671-6205-40d0-962c-aec63721b29c" successfully created
Running command: /usr/sbin/vgcreate -s 1G --force --yes ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465 /dev/sdb /dev/sda
 stdout: Physical volume "/dev/sdb" successfully created.
 stdout: Physical volume "/dev/sda" successfully created.
 stdout: Volume group "ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465" successfully created
Running command: /usr/sbin/lvcreate --yes -l 1863 -n osd-block-3ea03e63-1477-4361-8772-b4cf2b6ec68b ceph-block-499b12ab-639c-4428-b833-186f346c1fef
 stdout: Logical volume "osd-block-3ea03e63-1477-4361-8772-b4cf2b6ec68b" created.
Running command: /usr/sbin/lvcreate --yes -l 74 -n osd-block-db-87563fce-3548-480f-940f-51975026030d ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465
 stdout: Logical volume "osd-block-db-87563fce-3548-480f-940f-51975026030d" created.
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 826d858d-b5d0-43f7-869e-b446125513e3
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-3
Running command: /usr/sbin/restorecon /var/lib/ceph/osd/ceph-3
Running command: /bin/chown -h ceph:ceph /dev/ceph-block-499b12ab-639c-4428-b833-186f346c1fef/osd-block-3ea03e63-1477-4361-8772-b4cf2b6ec68b
Running command: /bin/chown -R ceph:ceph /dev/dm-5
Running command: /bin/ln -s /dev/ceph-block-499b12ab-639c-4428-b833-186f346c1fef/osd-block-3ea03e63-1477-4361-8772-b4cf2b6ec68b /var/lib/ceph/osd/ceph-3/block
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-3/activate.monmap
 stderr: 2019-12-11 23:34:34.221 7f5b6c1c2700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-12-11 23:34:34.221 7f5b6c1c2700 -1 AuthRegistry(0x7f5b64064e08) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
 stderr: got monmap epoch 1
Running command: /bin/ceph-authtool /var/lib/ceph/osd/ceph-3/keyring --create-keyring --name osd.3 --add-key AQBWw/Fd0o72BRAAtQx8BhxPHMS9Msi4LsMFLA==
 stdout: creating /var/lib/ceph/osd/ceph-3/keyring
added entity osd.3 auth(key=AQBWw/Fd0o72BRAAtQx8BhxPHMS9Msi4LsMFLA==)
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3/keyring
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3/
Running command: /bin/chown -h ceph:ceph /dev/ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465/osd-block-db-87563fce-3548-480f-940f-51975026030d
Running command: /bin/chown -R ceph:ceph /dev/dm-6
Running command: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 3 --monmap /var/lib/ceph/osd/ceph-3/activate.monmap --keyfile - --bluestore-block-db-path /dev/ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465/osd-block-db-87563fce-3548-480f-940f-51975026030d --osd-data /var/lib/ceph/osd/ceph-3/ --osd-uuid 826d858d-b5d0-43f7-869e-b446125513e3 --setuser ceph --setgroup ceph
--> ceph-volume lvm prepare successful for: ceph-block-499b12ab-639c-4428-b833-186f346c1fef/osd-block-3ea03e63-1477-4361-8772-b4cf2b6ec68b
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3
Running command: /bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-block-499b12ab-639c-4428-b833-186f346c1fef/osd-block-3ea03e63-1477-4361-8772-b4cf2b6ec68b --path /var/lib/ceph/osd/ceph-3 --no-mon-config
Running command: /bin/ln -snf /dev/ceph-block-499b12ab-639c-4428-b833-186f346c1fef/osd-block-3ea03e63-1477-4361-8772-b4cf2b6ec68b /var/lib/ceph/osd/ceph-3/block
Running command: /bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block
Running command: /bin/chown -R ceph:ceph /dev/dm-5
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3
Running command: /bin/ln -snf /dev/ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465/osd-block-db-87563fce-3548-480f-940f-51975026030d /var/lib/ceph/osd/ceph-3/block.db
Running command: /bin/chown -h ceph:ceph /dev/ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465/osd-block-db-87563fce-3548-480f-940f-51975026030d
Running command: /bin/chown -R ceph:ceph /dev/dm-6
Running command: /bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block.db
Running command: /bin/chown -R ceph:ceph /dev/dm-6
Running command: /bin/systemctl enable ceph-volume@lvm-3-826d858d-b5d0-43f7-869e-b446125513e3
 stderr: Created symlink from /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-3-826d858d-b5d0-43f7-869e-b446125513e3.service to /usr/lib/systemd/system/ceph-volume@.service.
Running command: /bin/systemctl enable --runtime ceph-osd@3
 stderr: Created symlink from /run/systemd/system/ceph-osd.target.wants/ceph-osd@3.service to /usr/lib/systemd/system/ceph-osd@.service.
Running command: /bin/systemctl start ceph-osd@3
--> ceph-volume lvm activate successful for osd ID: 3
--> ceph-volume lvm create successful for: ceph-block-499b12ab-639c-4428-b833-186f346c1fef/osd-block-3ea03e63-1477-4361-8772-b4cf2b6ec68b
Running command: /usr/sbin/lvcreate --yes -l 1863 -n osd-block-7001cf27-4173-4251-931d-beaf6505a9c2 ceph-block-073e90c8-9440-4e97-8b5e-121ea710219c
 stdout: Logical volume "osd-block-7001cf27-4173-4251-931d-beaf6505a9c2" created.
Running command: /usr/sbin/lvcreate --yes -l 74 -n osd-block-db-7ad12425-bba7-4b64-a207-48c602534a6d ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465
 stdout: Wiping ext3 signature on /dev/ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465/osd-block-db-7ad12425-bba7-4b64-a207-48c602534a6d.
 stdout: Logical volume "osd-block-db-7ad12425-bba7-4b64-a207-48c602534a6d" created.
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new ad832476-effc-4788-bb6b-4db933117622
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-4
Running command: /usr/sbin/restorecon /var/lib/ceph/osd/ceph-4
Running command: /bin/chown -h ceph:ceph /dev/ceph-block-073e90c8-9440-4e97-8b5e-121ea710219c/osd-block-7001cf27-4173-4251-931d-beaf6505a9c2
Running command: /bin/chown -R ceph:ceph /dev/dm-7
Running command: /bin/ln -s /dev/ceph-block-073e90c8-9440-4e97-8b5e-121ea710219c/osd-block-7001cf27-4173-4251-931d-beaf6505a9c2 /var/lib/ceph/osd/ceph-4/block
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-4/activate.monmap
 stderr: 2019-12-11 23:34:44.287 7f6ed473c700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-12-11 23:34:44.287 7f6ed473c700 -1 AuthRegistry(0x7f6ecc064e08) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
 stderr: got monmap epoch 1
Running command: /bin/ceph-authtool /var/lib/ceph/osd/ceph-4/keyring --create-keyring --name osd.4 --add-key AQBgw/FdO6pbCRAA5PGM/qJdxqmM4RwFIkm3Ww==
 stdout: creating /var/lib/ceph/osd/ceph-4/keyring
added entity osd.4 auth(key=AQBgw/FdO6pbCRAA5PGM/qJdxqmM4RwFIkm3Ww==)
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4/keyring
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4/
Running command: /bin/chown -h ceph:ceph /dev/ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465/osd-block-db-7ad12425-bba7-4b64-a207-48c602534a6d
Running command: /bin/chown -R ceph:ceph /dev/dm-8
Running command: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 4 --monmap /var/lib/ceph/osd/ceph-4/activate.monmap --keyfile - --bluestore-block-db-path /dev/ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465/osd-block-db-7ad12425-bba7-4b64-a207-48c602534a6d --osd-data /var/lib/ceph/osd/ceph-4/ --osd-uuid ad832476-effc-4788-bb6b-4db933117622 --setuser ceph --setgroup ceph
--> ceph-volume lvm prepare successful for: ceph-block-073e90c8-9440-4e97-8b5e-121ea710219c/osd-block-7001cf27-4173-4251-931d-beaf6505a9c2
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4
Running command: /bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-block-073e90c8-9440-4e97-8b5e-121ea710219c/osd-block-7001cf27-4173-4251-931d-beaf6505a9c2 --path /var/lib/ceph/osd/ceph-4 --no-mon-config
Running command: /bin/ln -snf /dev/ceph-block-073e90c8-9440-4e97-8b5e-121ea710219c/osd-block-7001cf27-4173-4251-931d-beaf6505a9c2 /var/lib/ceph/osd/ceph-4/block
Running command: /bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block
Running command: /bin/chown -R ceph:ceph /dev/dm-7
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4
Running command: /bin/ln -snf /dev/ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465/osd-block-db-7ad12425-bba7-4b64-a207-48c602534a6d /var/lib/ceph/osd/ceph-4/block.db
Running command: /bin/chown -h ceph:ceph /dev/ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465/osd-block-db-7ad12425-bba7-4b64-a207-48c602534a6d
Running command: /bin/chown -R ceph:ceph /dev/dm-8
Running command: /bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block.db
Running command: /bin/chown -R ceph:ceph /dev/dm-8
Running command: /bin/systemctl enable ceph-volume@lvm-4-ad832476-effc-4788-bb6b-4db933117622
 stderr: Created symlink from /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-4-ad832476-effc-4788-bb6b-4db933117622.service to /usr/lib/systemd/system/ceph-volume@.service.
Running command: /bin/systemctl enable --runtime ceph-osd@4
 stderr: Created symlink from /run/systemd/system/ceph-osd.target.wants/ceph-osd@4.service to /usr/lib/systemd/system/ceph-osd@.service.
Running command: /bin/systemctl start ceph-osd@4
--> ceph-volume lvm activate successful for osd ID: 4
--> ceph-volume lvm create successful for: ceph-block-073e90c8-9440-4e97-8b5e-121ea710219c/osd-block-7001cf27-4173-4251-931d-beaf6505a9c2
Running command: /usr/sbin/lvcreate --yes -l 1863 -n osd-block-03f9931a-493f-4501-a7f0-8df5177c1335 ceph-block-042c6671-6205-40d0-962c-aec63721b29c
 stdout: Logical volume "osd-block-03f9931a-493f-4501-a7f0-8df5177c1335" created.
Running command: /usr/sbin/lvcreate --yes -l 74 -n osd-block-db-af156afc-c3a1-4cc5-ac1f-7fdd19446b77 ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465
 stdout: Logical volume "osd-block-db-af156afc-c3a1-4cc5-ac1f-7fdd19446b77" created.
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 7adf16c5-efae-4a13-81fe-e3827e48265e
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-5
Running command: /usr/sbin/restorecon /var/lib/ceph/osd/ceph-5
Running command: /bin/chown -h ceph:ceph /dev/ceph-block-042c6671-6205-40d0-962c-aec63721b29c/osd-block-03f9931a-493f-4501-a7f0-8df5177c1335
Running command: /bin/chown -R ceph:ceph /dev/dm-9
Running command: /bin/ln -s /dev/ceph-block-042c6671-6205-40d0-962c-aec63721b29c/osd-block-03f9931a-493f-4501-a7f0-8df5177c1335 /var/lib/ceph/osd/ceph-5/block
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-5/activate.monmap
 stderr: 2019-12-11 23:34:54.467 7f76d569e700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2019-12-11 23:34:54.467 7f76d569e700 -1 AuthRegistry(0x7f76d0064e08) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
 stderr: got monmap epoch 1
Running command: /bin/ceph-authtool /var/lib/ceph/osd/ceph-5/keyring --create-keyring --name osd.5 --add-key AQBqw/Fdk5ZbBxAA6zW9RYSExF2OFWYtl0j6XA==
 stdout: creating /var/lib/ceph/osd/ceph-5/keyring
added entity osd.5 auth(key=AQBqw/Fdk5ZbBxAA6zW9RYSExF2OFWYtl0j6XA==)
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5/keyring
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5/
Running command: /bin/chown -h ceph:ceph /dev/ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465/osd-block-db-af156afc-c3a1-4cc5-ac1f-7fdd19446b77
Running command: /bin/chown -R ceph:ceph /dev/dm-10
Running command: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 5 --monmap /var/lib/ceph/osd/ceph-5/activate.monmap --keyfile - --bluestore-block-db-path /dev/ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465/osd-block-db-af156afc-c3a1-4cc5-ac1f-7fdd19446b77 --osd-data /var/lib/ceph/osd/ceph-5/ --osd-uuid 7adf16c5-efae-4a13-81fe-e3827e48265e --setuser ceph --setgroup ceph
--> ceph-volume lvm prepare successful for: ceph-block-042c6671-6205-40d0-962c-aec63721b29c/osd-block-03f9931a-493f-4501-a7f0-8df5177c1335
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5
Running command: /bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-block-042c6671-6205-40d0-962c-aec63721b29c/osd-block-03f9931a-493f-4501-a7f0-8df5177c1335 --path /var/lib/ceph/osd/ceph-5 --no-mon-config
Running command: /bin/ln -snf /dev/ceph-block-042c6671-6205-40d0-962c-aec63721b29c/osd-block-03f9931a-493f-4501-a7f0-8df5177c1335 /var/lib/ceph/osd/ceph-5/block
Running command: /bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-5/block
Running command: /bin/chown -R ceph:ceph /dev/dm-9
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5
Running command: /bin/ln -snf /dev/ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465/osd-block-db-af156afc-c3a1-4cc5-ac1f-7fdd19446b77 /var/lib/ceph/osd/ceph-5/block.db
Running command: /bin/chown -h ceph:ceph /dev/ceph-block-dbs-34fb0c98-5da3-4c2c-bd90-56cc74741465/osd-block-db-af156afc-c3a1-4cc5-ac1f-7fdd19446b77
Running command: /bin/chown -R ceph:ceph /dev/dm-10
Running command: /bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-5/block.db
Running command: /bin/chown -R ceph:ceph /dev/dm-10
Running command: /bin/systemctl enable ceph-volume@lvm-5-7adf16c5-efae-4a13-81fe-e3827e48265e
 stderr: Created symlink from /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-5-7adf16c5-efae-4a13-81fe-e3827e48265e.service to /usr/lib/systemd/system/ceph-volume@.service.
Running command: /bin/systemctl enable --runtime ceph-osd@5
Running command: /bin/systemctl start ceph-osd@5
--> ceph-volume lvm activate successful for osd ID: 5
--> ceph-volume lvm create successful for: ceph-block-042c6671-6205-40d0-962c-aec63721b29c/osd-block-03f9931a-493f-4501-a7f0-8df5177c1335
[root@XHOSTNAMEX deploy]#
[root@XHOSTNAMEX deploy]#
[root@XHOSTNAMEX deploy]#
[root@XHOSTNAMEX deploy]#
[root@XHOSTNAMEX deploy]# ceph-volume lvm batch --report `cat /tmp/devs`
--> Aborting because strategy changed from bluestore.MixedType to bluestore.SingleType after filtering

$
stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

dimm0 commented 4 years ago

Still an issue

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

madkiss commented 4 years ago

This is still an issue. I just ran into this on a RHOP 16 cluster.

dsavineau commented 4 years ago

This has been fixed by https://github.com/ceph/ceph-ansible/commit/e4c24f3407eb035c79fc64065fd5274d787aa6d5

foysalkayum commented 4 years ago

ansible 2.8 ceph nautilus

# ceph-volume lvm --report --bluestore --yes batch /dev/sda /dev/sdb /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdl
--> Aborting because strategy changed from bluestore.MixedType to bluestore.SingleType after filtering

This is similar to issue #4748 In THAT issue, someone commented along the lines of, "well yes its expected to fail, if you initially create a bunch of OSDs, then call ceph-volume with a different set of devices".

Except that I'm NOT using a different set of devices. its the exact same set that was first set up on the machine.

So apparently, theres some kind of bug in ceph, where it happily sets up a set of mixed ssd/hdd devices the first time in batch mode... but then --report will be broken ever after ?

Unless I'm misunderstanding, and the previous bug report is saying that ALL machines in the cluster must have the EXACT SAME device paths as each other. Which would be insane, given that linux can randomly rewrite device paths, and /dev/sda can become /dev/sdl after a reboot.

Right now, the only workaround I would seem to have, is: Add a machine one time to the [osds] ansible host group. get it configured... then take it OUT of the group.

You mean just take out all the OSD nodes from host group and only running ansible script adding the new node? We have 9 active node on the cluster, Is there have any disastrous issue doing so?

Would be nice to have a better alternative.

(or is that kind of expected ceph-ansible behaviour? to create a dynamic inventory file, and only put in it, what needs to be (re)configured this time ? )