search

osism / issues

This repository is used for bug reports that are cross-project or not bound to a specific repository (or to an unknown repository).
https://www.osism.tech
1 stars 1 forks source link

Ceph-Ansible mixed OSDs with HDD and NVME don't work #406

Closed zachelnet closed 1 year ago

zachelnet commented 1 year ago

If I add HDD and NVME to devices in the Inventory, don't created osds with "osism-ceph osds". node.yml:

...
devices:
  - /dev/disk/by-path/pci-0000:01:00.0-sas-phy0-lun-0
  - /dev/disk/by-path/pci-0000:01:00.0-sas-phy1-lun-0
  - /dev/disk/by-path/pci-0000:01:00.0-sas-phy2-lun-0
  - /dev/disk/by-path/pci-0000:01:00.0-sas-phy3-lun-0
  - /dev/disk/by-path/pci-0000:01:00.0-sas-phy4-lun-0
  - /dev/disk/by-path/pci-0000:01:00.0-sas-phy5-lun-0
  - /dev/disk/by-path/pci-0000:01:00.0-sas-phy6-lun-0
  - /dev/disk/by-path/pci-0000:01:00.0-sas-phy7-lun-0
  - /dev/nvme0n1

Following messages is coming:

changed: [ospoc-2] => {
    "changed": true,
    "cmd": [
        "docker",
        "run",
        "--rm",
        "--privileged",
        "--net=host",
        "--ipc=host",
        "-v",
        "/run/lock/lvm:/run/lock/lvm:z",
        "-v",
        "/var/run/udev:/var/run/udev:z",
        "-v",
        "/dev:/dev",
        "-v",
        "/etc/ceph:/etc/ceph:z",
        "-v",
        "/run/lvm:/run/lvm",
        "-v",
        "/var/lib/ceph:/var/lib/ceph:z",
        "-v",
        "/var/log/ceph:/var/log/ceph:z",
        "--entrypoint=ceph-volume",
        "quay.io/osism/ceph-daemon:pacific",
        "--cluster",
        "ceph",
        "lvm",
        "batch",
        "--bluestore",
        "--yes",
        "--prepare",
        "/dev/nvme0n1",
        "/dev/sdi",
        "/dev/sdh",
        "/dev/sdg",
        "/dev/sdf",
        "/dev/sde",
        "/dev/sdc",
        "/dev/sdd",
        "/dev/sdb"
    ],
    "delta": "0:00:05.477316",
    "end": "2023-01-17 08:10:24.250695",
    "invocation": {
        "module_args": {
            "action": "batch",
            "batch_devices": [
                "/dev/nvme0n1",
                "/dev/sdi",
                "/dev/sdh",
                "/dev/sdg",
                "/dev/sdf",
                "/dev/sde",
                "/dev/sdc",
                "/dev/sdd",
                "/dev/sdb"
            ],
            "block_db_devices": [],
            "block_db_size": "-1",
            "cluster": "ceph",
            "crush_device_class": "",
            "data": null,
            "data_vg": null,
            "db": null,
            "db_vg": null,
            "destroy": true,
            "dmcrypt": false,
            "journal": null,
            "journal_devices": [],
            "journal_size": "5120",
            "journal_vg": null,
            "objectstore": "bluestore",
            "osd_fsid": null,
            "osd_id": null,
            "osds_per_device": 1,
            "report": false,
            "wal": null,
            "wal_devices": [],
            "wal_vg": null
        }
    },
    "rc": 0,
    "start": "2023-01-17 08:10:18.773379",
    "stderr": "--> DEPRECATION NOTICE\n--> You are using the legacy automatic disk sorting behavior\n--> The Pacific release will change the default to --no-auto\n--> passed data devices: 8 physical, 0 LVM\n--> relative data size: 1.0\n--> passed block_db devices: 1 physical, 0 LVM\n--> 1 fast devices were passed, but none are available",
    "stderr_lines": [
        "--> DEPRECATION NOTICE",
        "--> You are using the legacy automatic disk sorting behavior",
        "--> The Pacific release will change the default to --no-auto",
        "--> passed data devices: 8 physical, 0 LVM",
        "--> relative data size: 1.0",
        "--> passed block_db devices: 1 physical, 0 LVM",
        "--> 1 fast devices were passed, but none are available"
    ],
    "stdout": "",
    "stdout_lines": []
}

After I comment out the NVME, it's worked:

changed: [ospoc-2] => {
    "changed": true,
    "cmd": [
        "docker",
        "run",
        "--rm",
        "--privileged",
        "--net=host",
        "--ipc=host",
        "-v",
        "/run/lock/lvm:/run/lock/lvm:z",
        "-v",
        "/var/run/udev:/var/run/udev:z",
        "-v",
        "/dev:/dev",
        "-v",
        "/etc/ceph:/etc/ceph:z",
        "-v",
        "/run/lvm:/run/lvm",
        "-v",
        "/var/lib/ceph:/var/lib/ceph:z",
        "-v",
        "/var/log/ceph:/var/log/ceph:z",
        "--entrypoint=ceph-volume",
        "quay.io/osism/ceph-daemon:pacific",
        "--cluster",
        "ceph",
        "lvm",
        "batch",
        "--bluestore",
        "--yes",
        "--prepare",
        "/dev/sdi",
        "/dev/sdh",
        "/dev/sdg",
        "/dev/sdf",
        "/dev/sde",
        "/dev/sdc",
        "/dev/sdd",
        "/dev/sdb"
    ],
    "delta": "0:01:04.719803",
    "end": "2023-01-17 09:07:09.638166",
    "invocation": {
        "module_args": {
            "action": "batch",
            "batch_devices": [
                "/dev/sdi",
                "/dev/sdh",
                "/dev/sdg",
                "/dev/sdf",
                "/dev/sde",
                "/dev/sdc",
                "/dev/sdd",
                "/dev/sdb"
            ],
            "block_db_devices": [],
            "block_db_size": "-1",
            "cluster": "ceph",
            "crush_device_class": "",
            "data": null,
            "data_vg": null,
            "db": null,
            "db_vg": null,
            "destroy": true,
            "dmcrypt": false,
            "journal": null,
            "journal_devices": [],
            "journal_size": "5120",
            "journal_vg": null,
            "objectstore": "bluestore",
            "osd_fsid": null,
            "osd_id": null,
            "osds_per_device": 1,
            "report": false,
            "wal": null,
            "wal_devices": [],
            "wal_vg": null
        }
    },
    "rc": 0,
    "start": "2023-01-17 09:06:04.918363",
    "stderr": "--> DEPRECATION NOTICE\n--> You are using the legacy automatic disk sorting behavior\n--> The Pacific release will change the default to --no-auto\n--> passed data devices: 8 physical, 0 LVM\n--> relative data size: 1.0\nRunning command: /usr/bin/ceph-authtool --gen-print-key\nRunning command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4ffe1bda-c169-4602-857a-5c67c03050c5\nRunning command: vgcreate --force --yes ceph-50199072-100c-4e52-a3bf-bc1bf15190b3 /dev/sdi\n stdout: Physical volume \"/dev/sdi\" successfully created.\n stdout: Volume group \"ceph-50199072-100c-4e52-a3bf-bc1bf15190b3\" successfully created\nRunning command: lvcreate --yes -l 1907721 -n osd-block-4ffe1bda-c169-4602-857a-5c67c03050c5 ceph-50199072-100c-4e52-a3bf-bc1bf15190b3\n stdout: Logical volume \"osd-block-4ffe1bda-c169-4602-857a-5c67c03050c5\" created.\nRunning command: /usr/bin/ceph-authtool --gen-print-key\nRunning command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-22\nRunning command: /usr/bin/chown -h ceph:ceph /dev/ceph-50199072-100c-4e52-a3bf-bc1bf15190b3/osd-block-4ffe1bda-c169-4602-857a-5c67c03050c5\nRunning command: /usr/bin/chown -R ceph:ceph /dev/dm-9\nRunning command: /usr/bin/ln -s /dev/ceph-50199072-100c-4e52-a3bf-bc1bf15190b3/osd-block-4ffe1bda-c169-4602-857a-5c67c03050c5 /var/lib/ceph/osd/ceph-22/block\nRunning command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-22/activate.monmap\n stderr: got monmap epoch 1\nRunning command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-22/keyring --create-keyring --name osd.22 --add-key AQACZcZjfpl4ARAAAASc4EK9IrdS8MQh3xMaQw==\n stdout: creating /var/lib/ceph/osd/ceph-22/keyring\nadded entity osd.22 auth(key=AQACZcZjfpl4ARAAAASc4EK9IrdS8MQh3xMaQw==)\nRunning command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-22/keyring\nRunning command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-22/\nRunning command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 22 --monmap /var/lib/ceph/osd/ceph-22/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-22/ --osd-uuid 4ffe1bda-c169-4602-857a-5c67c03050c5 --setuser ceph --setgroup ceph\n stderr: 2023-01-17T09:06:12.300+0000 7fd96b3c9200 -1 bluestore(/var/lib/ceph/osd/ceph-22/) _read_fsid unparsable uuid\n stderr: tcmalloc: large alloc 1314775040 bytes == 0x55a853606000 @  0x7fd969e95760 0x7fd969eb6c64 0x55a84774d7f6 0x55a847720730 0x55a84772ed5c 0x55a847731a24 0x55a8471a3fef 0x55a84712a898 0x7fd967fcaca3 0x55a8471762be\n stderr: tcmalloc: large alloc 1314775040 bytes == 0x55a8a1be4000 @  0x7fd969e95760 0x7fd969eb6c64 0x55a84774d7f6 0x55a8477201b5 0x55a84772ed5c 0x55a847731a24 0x55a8471a3fef 0x55a84712a898 0x7fd967fcaca3 0x55a8471762be\n--> ceph-volume lvm prepare successful for: /dev/sdi\nRunning command: /usr/bin/ceph-authtool --gen-print-key\nRunning command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e0b9e1d7-a9e0-4728-b3a3-f3aa9954d8dd\nRunning command: vgcreate --force --yes ceph-9507a566-4ab9-4dbf-a20f-cec3de2e3da4 /dev/sdh\n stdout: Physical volume \"/dev/sdh\" successfully created.\n stdout: Volume group \"ceph-9507a566-4ab9-4dbf-a20f-cec3de2e3da4\" successfully created\nRunning command: lvcreate --yes -l 1907721 -n osd-block-e0b9e1d7-a9e0-4728-b3a3-f3aa9954d8dd ceph-9507a566-4ab9-4dbf-a20f-cec3de2e3da4\n stdout: Logical volume \"osd-block-e0b9e1d7-a9e0-4728-b3a3-f3aa9954d8dd\" created.\nRunning command: /usr/bin/ceph-authtool --gen-print-key\nRunning command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-23\nRunning command: /usr/bin/chown -h ceph:ceph /dev/ceph-9507a566-4ab9-4dbf-a20f-cec3de2e3da4/osd-block-e0b9e1d7-a9e0-4728-b3a3-f3aa9954d8dd\nRunning command: /usr/bin/chown -R ceph:ceph /dev/dm-10\nRunning command: /usr/bin/ln -s /dev/ceph-9507a566-4ab9-4dbf-a20f-cec3de2e3da4/osd-block-e0b9e1d7-a9e0-4728-b3a3-f3aa9954d8dd /var/lib/ceph/osd/ceph-23/block\nRunning command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-23/activate.monmap\n stderr: got monmap epoch 1\nRunning command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-23/keyring --create-keyring --name osd.23 --add-key AQAOZcZjzDD5CxAA5YBiwlfWIF1RYkqrElzmkQ==\n stdout: creating /var/lib/ceph/osd/ceph-23/keyring\nadded entity osd.23 auth(key=AQAOZcZjzDD5CxAA5YBiwlfWIF1RYkqrElzmkQ==)\nRunning command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-23/keyring\nRunning command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-23/\nRunning command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 23 --monmap /var/lib/ceph/osd/ceph-23/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-23/ --osd-uuid e0b9e1d7-a9e0-4728-b3a3-f3aa9954d8dd --setuser ceph --setgroup ceph\n stderr: 2023-01-17T09:06:23.916+0000 7f5462f44200 -1 bluestore(/var/lib/ceph/osd/ceph-23/) _read_fsid unparsable uuid\n stderr: tcmalloc: large alloc 1314775040 bytes == 0x5623c68fe000 @  0x7f5461a10760 0x7f5461a31c64 0x5623bb9c37f6 0x5623bb996730 0x5623bb9a4d5c 0x5623bb9a7a24 0x5623bb419fef 0x5623bb3a0898 0x7f545fb45ca3 0x5623bb3ec2be\n stderr: tcmalloc: large alloc 1314775040 bytes == 0x562414edc000 @  0x7f5461a10760 0x7f5461a31c64 0x5623bb9c37f6 0x5623bb9961b5 0x5623bb9a4d5c 0x5623bb9a7a24 0x5623bb419fef 0x5623bb3a0898 0x7f545fb45ca3 0x5623bb3ec2be\n--> ceph-volume lvm prepare successful for: /dev/sdh\nRunning command: /usr/bin/ceph-authtool --gen-print-key\nRunning command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 7fcf2514-b136-44c5-b7bf-5012eb735b0f\nRunning command: vgcreate --force --yes ceph-3e89978d-cc45-4b3a-9a5a-0802a376e29c /dev/sdg\n stdout: Physical volume \"/dev/sdg\" successfully created.\n stdout: Volume group \"ceph-3e89978d-cc45-4b3a-9a5a-0802a376e29c\" successfully created\nRunning command: lvcreate --yes -l 1907721 -n osd-block-7fcf2514-b136-44c5-b7bf-5012eb735b0f ceph-3e89978d-cc45-4b3a-9a5a-0802a376e29c\n stdout: Logical volume \"osd-block-7fcf2514-b136-44c5-b7bf-5012eb735b0f\" created.\nRunning command: /usr/bin/ceph-authtool --gen-print-key\nRunning command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-24\nRunning command: /usr/bin/chown -h ceph:ceph /dev/ceph-3e89978d-cc45-4b3a-9a5a-0802a376e29c/osd-block-7fcf2514-b136-44c5-b7bf-5012eb735b0f\nRunning command: /usr/bin/chown -R ceph:ceph /dev/dm-11\nRunning command: /usr/bin/ln -s /dev/ceph-3e89978d-cc45-4b3a-9a5a-0802a376e29c/osd-block-7fcf2514-b136-44c5-b7bf-5012eb735b0f /var/lib/ceph/osd/ceph-24/block\nRunning command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-24/activate.monmap\n stderr: got monmap epoch 1\nRunning command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-24/keyring --create-keyring --name osd.24 --add-key AQAaZcZjOo33FxAAoTvpOSFzz7dkv4L51l50XQ==\n stdout: creating /var/lib/ceph/osd/ceph-24/keyring\nadded entity osd.24 auth(key=AQAaZcZjOo33FxAAoTvpOSFzz7dkv4L51l50XQ==)\nRunning command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-24/keyring\nRunning command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-24/\nRunning command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 24 --monmap /var/lib/ceph/osd/ceph-24/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-24/ --osd-uuid 7fcf2514-b136-44c5-b7bf-5012eb735b0f --setuser ceph --setgroup ceph\n stderr: 2023-01-17T09:06:36.236+0000 7f10392e7200 -1 bluestore(/var/lib/ceph/osd/ceph-24/) _read_fsid unparsable uuid\n stderr: tcmalloc: large alloc 1314775040 bytes == 0x561147164000 @  0x7f1037db3760 0x7f1037dd4c64 0x5611391957f6 0x561139168730 0x561139176d5c 0x561139179a24 0x561138bebfef 0x561138b72898 0x7f1035ee8ca3 0x561138bbe2be\n stderr: tcmalloc: large alloc 1314775040 bytes == 0x561195742000 @  0x7f1037db3760 0x7f1037dd4c64 0x5611391957f6 0x5611391681b5 0x561139176d5c 0x561139179a24 0x561138bebfef 0x561138b72898 0x7f1035ee8ca3 0x561138bbe2be\n--> ceph-volume lvm prepare successful for: /dev/sdg\nRunning command: /usr/bin/ceph-authtool --gen-print-key\nRunning command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new f92c2c87-9e75-4332-ad71-8adf425dca32\nRunning command: vgcreate --force --yes ceph-95b3ade9-6cad-4544-b735-092c35fc759c /dev/sdf\n stdout: Physical volume \"/dev/sdf\" successfully created.\n stdout: Volume group \"ceph-95b3ade9-6cad-4544-b735-092c35fc759c\" successfully created\nRunning command: lvcreate --yes -l 1907721 -n osd-block-f92c2c87-9e75-4332-ad71-8adf425dca32 ceph-95b3ade9-6cad-4544-b735-092c35fc759c\n stdout: Logical volume \"osd-block-f92c2c87-9e75-4332-ad71-8adf425dca32\" created.\nRunning command: /usr/bin/ceph-authtool --gen-print-key\nRunning command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-25\nRunning command: /usr/bin/chown -h ceph:ceph /dev/ceph-95b3ade9-6cad-4544-b735-092c35fc759c/osd-block-f92c2c87-9e75-4332-ad71-8adf425dca32\nRunning command: /usr/bin/chown -R ceph:ceph /dev/dm-12\nRunning command: /usr/bin/ln -s /dev/ceph-95b3ade9-6cad-4544-b735-092c35fc759c/osd-block-f92c2c87-9e75-4332-ad71-8adf425dca32 /var/lib/ceph/osd/ceph-25/block\nRunning command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-25/activate.monmap\n stderr: got monmap epoch 1\nRunning command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-25/keyring --create-keyring --name osd.25 --add-key AQAmZcZjZLVGBxAAx5XL+k8pruTqDAo4udgeoQ==\n stdout: creating /var/lib/ceph/osd/ceph-25/keyring\nadded entity osd.25 auth(key=AQAmZcZjZLVGBxAAx5XL+k8pruTqDAo4udgeoQ==)\nRunning command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-25/keyring\nRunning command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-25/\nRunning command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 25 --monmap /var/lib/ceph/osd/ceph-25/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-25/ --osd-uuid f92c2c87-9e75-4332-ad71-8adf425dca32 --setuser ceph --setgroup ceph\n stderr: 2023-01-17T09:06:48.016+0000 7f47006e4200 -1 bluestore(/var/lib/ceph/osd/ceph-25/) _read_fsid unparsable uuid\n stderr: tcmalloc: large alloc 1314775040 bytes == 0x557918b22000 @  0x7f46ff1b0760 0x7f46ff1d1c64 0x55790d88f7f6 0x55790d862730 0x55790d870d5c 0x55790d873a24 0x55790d2e5fef 0x55790d26c898 0x7f46fd2e5ca3 0x55790d2b82be\n stderr: tcmalloc: large alloc 1314775040 bytes == 0x557967100000 @  0x7f46ff1b0760 0x7f46ff1d1c64 0x55790d88f7f6 0x55790d8621b5 0x55790d870d5c 0x55790d873a24 0x55790d2e5fef 0x55790d26c898 0x7f46fd2e5ca3 0x55790d2b82be\n--> ceph-volume lvm prepare successful for: /dev/sdf\nRunning command: /usr/bin/ceph-authtool --gen-print-key\nRunning command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new f73d8c62-9e52-49dd-b52c-a7f16acc265b\nRunning command: vgcreate --force --yes ceph-b38b62a5-dcf3-4d28-9fa4-2d9aa1dda9be /dev/sde\n stdout: Physical volume \"/dev/sde\" successfully created.\n stdout: Volume group \"ceph-b38b62a5-dcf3-4d28-9fa4-2d9aa1dda9be\" successfully created\nRunning command: lvcreate --yes -l 1907721 -n osd-block-f73d8c62-9e52-49dd-b52c-a7f16acc265b ceph-b38b62a5-dcf3-4d28-9fa4-2d9aa1dda9be\n stdout: Logical volume \"osd-block-f73d8c62-9e52-49dd-b52c-a7f16acc265b\" created.\nRunning command: /usr/bin/ceph-authtool --gen-print-key\nRunning command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-26\nRunning command: /usr/bin/chown -h ceph:ceph /dev/ceph-b38b62a5-dcf3-4d28-9fa4-2d9aa1dda9be/osd-block-f73d8c62-9e52-49dd-b52c-a7f16acc265b\nRunning command: /usr/bin/chown -R ceph:ceph /dev/dm-13\nRunning command: /usr/bin/ln -s /dev/ceph-b38b62a5-dcf3-4d28-9fa4-2d9aa1dda9be/osd-block-f73d8c62-9e52-49dd-b52c-a7f16acc265b /var/lib/ceph/osd/ceph-26/block\nRunning command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-26/activate.monmap\n stderr: got monmap epoch 1\nRunning command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-26/keyring --create-keyring --name osd.26 --add-key AQAxZcZjq/fONRAAVHoZ+1gkB2NEhJOuq4dBkw==\n stdout: creating /var/lib/ceph/osd/ceph-26/keyring\nadded entity osd.26 auth(key=AQAxZcZjq/fONRAAVHoZ+1gkB2NEhJOuq4dBkw==)\nRunning command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-26/keyring\nRunning command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-26/\nRunning command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 26 --monmap /var/lib/ceph/osd/ceph-26/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-26/ --osd-uuid f73d8c62-9e52-49dd-b52c-a7f16acc265b --setuser ceph --setgroup ceph\n stderr: 2023-01-17T09:06:59.680+0000 7f070d2c3200 -1 bluestore(/var/lib/ceph/osd/ceph-26/) _read_fsid unparsable uuid\n stderr: tcmalloc: large alloc 1314775040 bytes == 0x5599b619e000 @  0x7f070bd8f760 0x7f070bdb0c64 0x5599ab8477f6 0x5599ab81a730 0x5599ab828d5c 0x5599ab82ba24 0x5599ab29dfef 0x5599ab224898 0x7f0709ec4ca3 0x5599ab2702be\n stderr: tcmalloc: large alloc 1314775040 bytes == 0x559a0477c000 @  0x7f070bd8f760 0x7f070bdb0c64 0x5599ab8477f6 0x5599ab81a1b5 0x5599ab828d5c 0x5599ab82ba24 0x5599ab29dfef 0x5599ab224898 0x7f0709ec4ca3 0x5599ab2702be\n--> ceph-volume lvm prepare successful for: /dev/sde",
    "stderr_lines": [
        "--> DEPRECATION NOTICE",
        "--> You are using the legacy automatic disk sorting behavior",
        "--> The Pacific release will change the default to --no-auto",
        "--> passed data devices: 8 physical, 0 LVM",
        "--> relative data size: 1.0",
        "Running command: /usr/bin/ceph-authtool --gen-print-key",
        "Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4ffe1bda-c169-4602-857a-5c67c03050c5",
        "Running command: vgcreate --force --yes ceph-50199072-100c-4e52-a3bf-bc1bf15190b3 /dev/sdi",
        " stdout: Physical volume \"/dev/sdi\" successfully created.",
        " stdout: Volume group \"ceph-50199072-100c-4e52-a3bf-bc1bf15190b3\" successfully created",
        "Running command: lvcreate --yes -l 1907721 -n osd-block-4ffe1bda-c169-4602-857a-5c67c03050c5 ceph-50199072-100c-4e52-a3bf-bc1bf15190b3",
        " stdout: Logical volume \"osd-block-4ffe1bda-c169-4602-857a-5c67c03050c5\" created.",
        "Running command: /usr/bin/ceph-authtool --gen-print-key",
        "Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-22",
        "Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-50199072-100c-4e52-a3bf-bc1bf15190b3/osd-block-4ffe1bda-c169-4602-857a-5c67c03050c5",
        "Running command: /usr/bin/chown -R ceph:ceph /dev/dm-9",
        "Running command: /usr/bin/ln -s /dev/ceph-50199072-100c-4e52-a3bf-bc1bf15190b3/osd-block-4ffe1bda-c169-4602-857a-5c67c03050c5 /var/lib/ceph/osd/ceph-22/block",
        "Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-22/activate.monmap",
        " stderr: got monmap epoch 1",
        "Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-22/keyring --create-keyring --name osd.22 --add-key AQACZcZjfpl4ARAAAASc4EK9IrdS8MQh3xMaQw==",
        " stdout: creating /var/lib/ceph/osd/ceph-22/keyring",
        "added entity osd.22 auth(key=AQACZcZjfpl4ARAAAASc4EK9IrdS8MQh3xMaQw==)",
        "Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-22/keyring",
        "Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-22/",
        "Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 22 --monmap /var/lib/ceph/osd/ceph-22/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-22/ --osd-uuid 4ffe1bda-c169-4602-857a-5c67c03050c5 --setuser ceph --setgroup ceph",
        " stderr: 2023-01-17T09:06:12.300+0000 7fd96b3c9200 -1 bluestore(/var/lib/ceph/osd/ceph-22/) _read_fsid unparsable uuid",
        " stderr: tcmalloc: large alloc 1314775040 bytes == 0x55a853606000 @  0x7fd969e95760 0x7fd969eb6c64 0x55a84774d7f6 0x55a847720730 0x55a84772ed5c 0x55a847731a24 0x55a8471a3fef 0x55a84712a898 0x7fd967fcaca3 0x55a8471762be",
        " stderr: tcmalloc: large alloc 1314775040 bytes == 0x55a8a1be4000 @  0x7fd969e95760 0x7fd969eb6c64 0x55a84774d7f6 0x55a8477201b5 0x55a84772ed5c 0x55a847731a24 0x55a8471a3fef 0x55a84712a898 0x7fd967fcaca3 0x55a8471762be",
        "--> ceph-volume lvm prepare successful for: /dev/sdi",
        "Running command: /usr/bin/ceph-authtool --gen-print-key",
        "Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e0b9e1d7-a9e0-4728-b3a3-f3aa9954d8dd",
        "Running command: vgcreate --force --yes ceph-9507a566-4ab9-4dbf-a20f-cec3de2e3da4 /dev/sdh",
        " stdout: Physical volume \"/dev/sdh\" successfully created.",
        " stdout: Volume group \"ceph-9507a566-4ab9-4dbf-a20f-cec3de2e3da4\" successfully created",
        "Running command: lvcreate --yes -l 1907721 -n osd-block-e0b9e1d7-a9e0-4728-b3a3-f3aa9954d8dd ceph-9507a566-4ab9-4dbf-a20f-cec3de2e3da4",
        " stdout: Logical volume \"osd-block-e0b9e1d7-a9e0-4728-b3a3-f3aa9954d8dd\" created.",
        "Running command: /usr/bin/ceph-authtool --gen-print-key",
        "Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-23",
        "Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-9507a566-4ab9-4dbf-a20f-cec3de2e3da4/osd-block-e0b9e1d7-a9e0-4728-b3a3-f3aa9954d8dd",
        "Running command: /usr/bin/chown -R ceph:ceph /dev/dm-10",
        "Running command: /usr/bin/ln -s /dev/ceph-9507a566-4ab9-4dbf-a20f-cec3de2e3da4/osd-block-e0b9e1d7-a9e0-4728-b3a3-f3aa9954d8dd /var/lib/ceph/osd/ceph-23/block",
        "Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-23/activate.monmap",
        " stderr: got monmap epoch 1",
        "Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-23/keyring --create-keyring --name osd.23 --add-key AQAOZcZjzDD5CxAA5YBiwlfWIF1RYkqrElzmkQ==",
        " stdout: creating /var/lib/ceph/osd/ceph-23/keyring",
        "added entity osd.23 auth(key=AQAOZcZjzDD5CxAA5YBiwlfWIF1RYkqrElzmkQ==)",
        "Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-23/keyring",
        "Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-23/",
        "Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 23 --monmap /var/lib/ceph/osd/ceph-23/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-23/ --osd-uuid e0b9e1d7-a9e0-4728-b3a3-f3aa9954d8dd --setuser ceph --setgroup ceph",
        " stderr: 2023-01-17T09:06:23.916+0000 7f5462f44200 -1 bluestore(/var/lib/ceph/osd/ceph-23/) _read_fsid unparsable uuid",
        " stderr: tcmalloc: large alloc 1314775040 bytes == 0x5623c68fe000 @  0x7f5461a10760 0x7f5461a31c64 0x5623bb9c37f6 0x5623bb996730 0x5623bb9a4d5c 0x5623bb9a7a24 0x5623bb419fef 0x5623bb3a0898 0x7f545fb45ca3 0x5623bb3ec2be",
        " stderr: tcmalloc: large alloc 1314775040 bytes == 0x562414edc000 @  0x7f5461a10760 0x7f5461a31c64 0x5623bb9c37f6 0x5623bb9961b5 0x5623bb9a4d5c 0x5623bb9a7a24 0x5623bb419fef 0x5623bb3a0898 0x7f545fb45ca3 0x5623bb3ec2be",
        "--> ceph-volume lvm prepare successful for: /dev/sdh",
        "Running command: /usr/bin/ceph-authtool --gen-print-key",
        "Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 7fcf2514-b136-44c5-b7bf-5012eb735b0f",
        "Running command: vgcreate --force --yes ceph-3e89978d-cc45-4b3a-9a5a-0802a376e29c /dev/sdg",
        " stdout: Physical volume \"/dev/sdg\" successfully created.",
        " stdout: Volume group \"ceph-3e89978d-cc45-4b3a-9a5a-0802a376e29c\" successfully created",
        "Running command: lvcreate --yes -l 1907721 -n osd-block-7fcf2514-b136-44c5-b7bf-5012eb735b0f ceph-3e89978d-cc45-4b3a-9a5a-0802a376e29c",
        " stdout: Logical volume \"osd-block-7fcf2514-b136-44c5-b7bf-5012eb735b0f\" created.",
        "Running command: /usr/bin/ceph-authtool --gen-print-key",
        "Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-24",
        "Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-3e89978d-cc45-4b3a-9a5a-0802a376e29c/osd-block-7fcf2514-b136-44c5-b7bf-5012eb735b0f",
        "Running command: /usr/bin/chown -R ceph:ceph /dev/dm-11",
        "Running command: /usr/bin/ln -s /dev/ceph-3e89978d-cc45-4b3a-9a5a-0802a376e29c/osd-block-7fcf2514-b136-44c5-b7bf-5012eb735b0f /var/lib/ceph/osd/ceph-24/block",
        "Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-24/activate.monmap",
        " stderr: got monmap epoch 1",
        "Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-24/keyring --create-keyring --name osd.24 --add-key AQAaZcZjOo33FxAAoTvpOSFzz7dkv4L51l50XQ==",
        " stdout: creating /var/lib/ceph/osd/ceph-24/keyring",
        "added entity osd.24 auth(key=AQAaZcZjOo33FxAAoTvpOSFzz7dkv4L51l50XQ==)",
        "Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-24/keyring",
        "Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-24/",
        "Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 24 --monmap /var/lib/ceph/osd/ceph-24/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-24/ --osd-uuid 7fcf2514-b136-44c5-b7bf-5012eb735b0f --setuser ceph --setgroup ceph",
        " stderr: 2023-01-17T09:06:36.236+0000 7f10392e7200 -1 bluestore(/var/lib/ceph/osd/ceph-24/) _read_fsid unparsable uuid",
        " stderr: tcmalloc: large alloc 1314775040 bytes == 0x561147164000 @  0x7f1037db3760 0x7f1037dd4c64 0x5611391957f6 0x561139168730 0x561139176d5c 0x561139179a24 0x561138bebfef 0x561138b72898 0x7f1035ee8ca3 0x561138bbe2be",
        " stderr: tcmalloc: large alloc 1314775040 bytes == 0x561195742000 @  0x7f1037db3760 0x7f1037dd4c64 0x5611391957f6 0x5611391681b5 0x561139176d5c 0x561139179a24 0x561138bebfef 0x561138b72898 0x7f1035ee8ca3 0x561138bbe2be",
        "--> ceph-volume lvm prepare successful for: /dev/sdg",
        "Running command: /usr/bin/ceph-authtool --gen-print-key",
        "Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new f92c2c87-9e75-4332-ad71-8adf425dca32",
        "Running command: vgcreate --force --yes ceph-95b3ade9-6cad-4544-b735-092c35fc759c /dev/sdf",
        " stdout: Physical volume \"/dev/sdf\" successfully created.",
        " stdout: Volume group \"ceph-95b3ade9-6cad-4544-b735-092c35fc759c\" successfully created",
        "Running command: lvcreate --yes -l 1907721 -n osd-block-f92c2c87-9e75-4332-ad71-8adf425dca32 ceph-95b3ade9-6cad-4544-b735-092c35fc759c",
        " stdout: Logical volume \"osd-block-f92c2c87-9e75-4332-ad71-8adf425dca32\" created.",
        "Running command: /usr/bin/ceph-authtool --gen-print-key",
        "Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-25",
        "Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-95b3ade9-6cad-4544-b735-092c35fc759c/osd-block-f92c2c87-9e75-4332-ad71-8adf425dca32",
        "Running command: /usr/bin/chown -R ceph:ceph /dev/dm-12",
        "Running command: /usr/bin/ln -s /dev/ceph-95b3ade9-6cad-4544-b735-092c35fc759c/osd-block-f92c2c87-9e75-4332-ad71-8adf425dca32 /var/lib/ceph/osd/ceph-25/block",
        "Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-25/activate.monmap",
        " stderr: got monmap epoch 1",
        "Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-25/keyring --create-keyring --name osd.25 --add-key AQAmZcZjZLVGBxAAx5XL+k8pruTqDAo4udgeoQ==",
        " stdout: creating /var/lib/ceph/osd/ceph-25/keyring",
        "added entity osd.25 auth(key=AQAmZcZjZLVGBxAAx5XL+k8pruTqDAo4udgeoQ==)",
        "Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-25/keyring",
        "Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-25/",
        "Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 25 --monmap /var/lib/ceph/osd/ceph-25/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-25/ --osd-uuid f92c2c87-9e75-4332-ad71-8adf425dca32 --setuser ceph --setgroup ceph",
        " stderr: 2023-01-17T09:06:48.016+0000 7f47006e4200 -1 bluestore(/var/lib/ceph/osd/ceph-25/) _read_fsid unparsable uuid",
        " stderr: tcmalloc: large alloc 1314775040 bytes == 0x557918b22000 @  0x7f46ff1b0760 0x7f46ff1d1c64 0x55790d88f7f6 0x55790d862730 0x55790d870d5c 0x55790d873a24 0x55790d2e5fef 0x55790d26c898 0x7f46fd2e5ca3 0x55790d2b82be",
        " stderr: tcmalloc: large alloc 1314775040 bytes == 0x557967100000 @  0x7f46ff1b0760 0x7f46ff1d1c64 0x55790d88f7f6 0x55790d8621b5 0x55790d870d5c 0x55790d873a24 0x55790d2e5fef 0x55790d26c898 0x7f46fd2e5ca3 0x55790d2b82be",
        "--> ceph-volume lvm prepare successful for: /dev/sdf",
        "Running command: /usr/bin/ceph-authtool --gen-print-key",
        "Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new f73d8c62-9e52-49dd-b52c-a7f16acc265b",
        "Running command: vgcreate --force --yes ceph-b38b62a5-dcf3-4d28-9fa4-2d9aa1dda9be /dev/sde",
        " stdout: Physical volume \"/dev/sde\" successfully created.",
        " stdout: Volume group \"ceph-b38b62a5-dcf3-4d28-9fa4-2d9aa1dda9be\" successfully created",
        "Running command: lvcreate --yes -l 1907721 -n osd-block-f73d8c62-9e52-49dd-b52c-a7f16acc265b ceph-b38b62a5-dcf3-4d28-9fa4-2d9aa1dda9be",
        " stdout: Logical volume \"osd-block-f73d8c62-9e52-49dd-b52c-a7f16acc265b\" created.",
        "Running command: /usr/bin/ceph-authtool --gen-print-key",
        "Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-26",
        "Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-b38b62a5-dcf3-4d28-9fa4-2d9aa1dda9be/osd-block-f73d8c62-9e52-49dd-b52c-a7f16acc265b",
        "Running command: /usr/bin/chown -R ceph:ceph /dev/dm-13",
        "Running command: /usr/bin/ln -s /dev/ceph-b38b62a5-dcf3-4d28-9fa4-2d9aa1dda9be/osd-block-f73d8c62-9e52-49dd-b52c-a7f16acc265b /var/lib/ceph/osd/ceph-26/block",
        "Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-26/activate.monmap",
        " stderr: got monmap epoch 1",
        "Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-26/keyring --create-keyring --name osd.26 --add-key AQAxZcZjq/fONRAAVHoZ+1gkB2NEhJOuq4dBkw==",
        " stdout: creating /var/lib/ceph/osd/ceph-26/keyring",
        "added entity osd.26 auth(key=AQAxZcZjq/fONRAAVHoZ+1gkB2NEhJOuq4dBkw==)",
        "Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-26/keyring",
        "Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-26/",
        "Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 26 --monmap /var/lib/ceph/osd/ceph-26/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-26/ --osd-uuid f73d8c62-9e52-49dd-b52c-a7f16acc265b --setuser ceph --setgroup ceph",
        " stderr: 2023-01-17T09:06:59.680+0000 7f070d2c3200 -1 bluestore(/var/lib/ceph/osd/ceph-26/) _read_fsid unparsable uuid",
        " stderr: tcmalloc: large alloc 1314775040 bytes == 0x5599b619e000 @  0x7f070bd8f760 0x7f070bdb0c64 0x5599ab8477f6 0x5599ab81a730 0x5599ab828d5c 0x5599ab82ba24 0x5599ab29dfef 0x5599ab224898 0x7f0709ec4ca3 0x5599ab2702be",
        " stderr: tcmalloc: large alloc 1314775040 bytes == 0x559a0477c000 @  0x7f070bd8f760 0x7f070bdb0c64 0x5599ab8477f6 0x5599ab81a1b5 0x5599ab828d5c 0x5599ab82ba24 0x5599ab29dfef 0x5599ab224898 0x7f0709ec4ca3 0x5599ab2702be",
        "--> ceph-volume lvm prepare successful for: /dev/sde"
    ],
    "stdout": "",
    "stdout_lines": []
}
berendt commented 1 year ago

What is the expected result? Should the NVMe device be used for the journals from the spinning drives?

zachelnet commented 1 year ago

What is the expected result? Should the NVMe device be used for the journals from the spinning drives?

No, the NVMe should be used for osds to create a separate pool/storage type

berendt commented 1 year ago

Please have a look at https://docs.ceph.com/en/latest/ceph-volume/lvm/batch/#batch and https://docs.ceph.com/projects/ceph-ansible/en/latest/osds/scenarios.html#simple-configuration.

devices is the input for the batch creation. This does not work for using the NVMe as a dedicated pool and not as a cache device for the HDDs.

Please have a look at https://docs.ceph.com/projects/ceph-ansible/en/latest/osds/scenarios.html#advanced-configuration.

berendt commented 1 year ago

I am just clearing the issues. I think that with the reference to the Advanced Configuration, this is done so far. If not, please reopen with additional questions.