search

ceph / ceph-salt

Deploy Ceph clusters using cephadm
MIT License
31 stars 20 forks source link

After "ceph-bootstrap deploy", cluster is in HEALTH_WARN ("CEPHADM_STRAY_HOST: 3 stray host(s) with 10 service(s) not managed by cephadm") #56

Closed smithfarm closed 4 years ago

smithfarm commented 4 years ago 
$ sesdev ssh octopus_test1
Warning: Permanently added '192.168.121.4' (ECDSA) to the list of known hosts.
Have a lot of fun...
admin:~ # ceph -s
  cluster:
    id:     be09d766-42d5-11ea-bbb8-52540088717e
    health: HEALTH_WARN
            3 stray host(s) with 10 service(s) not managed by cephadm

  services:
    mon: 3 daemons, quorum node1.octopus_test1.com,node2,node3 (age 23m)
    mgr: hwboak(active, since 23m), standbys: icdsnv, ytvgxh
    osd: 6 osds: 6 up (since 22m), 6 in (since 22m)

  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   6.0 GiB used, 42 GiB / 48 GiB avail
    pgs:     

admin:~ # ceph health detail
HEALTH_WARN 3 stray host(s) with 10 service(s) not managed by cephadm
[WRN] CEPHADM_STRAY_HOST: 3 stray host(s) with 10 service(s) not managed by cephadm
    stray host node1.octopus_test1.com has 4 stray daemons: ['mgr.hwboak', 'mon.node1.octopus_test1.com', 'osd.0', 'osd.1']
    stray host node2.octopus_test1.com has 3 stray daemons: ['mgr.icdsnv', 'osd.2', 'osd.3']
    stray host node3.octopus_test1.com has 3 stray daemons: ['mgr.ytvgxh', 'osd.4', 'osd.5']
admin:~ # ceph versions
{
    "mon": {
        "ceph version 15.0.0-9865-gd2c6620fea (d2c6620fea8e44e5b0bc24a0effaa6347315be7e) octopus (dev)": 3
    },
    "mgr": {
        "ceph version 15.0.0-9865-gd2c6620fea (d2c6620fea8e44e5b0bc24a0effaa6347315be7e) octopus (dev)": 3
    },
    "osd": {
        "ceph version 15.0.0-9865-gd2c6620fea (d2c6620fea8e44e5b0bc24a0effaa6347315be7e) octopus (dev)": 6
    },
    "mds": {},
    "overall": {
        "ceph version 15.0.0-9865-gd2c6620fea (d2c6620fea8e44e5b0bc24a0effaa6347315be7e) octopus (dev)": 12
    }
}
admin:~ # ceph --version
ceph version 15.0.0-9865-gd2c6620fea (d2c6620fea8e44e5b0bc24a0effaa6347315be7e) octopus (dev)

UPDATE: The workaround is to explicitly add the hosts - e.g.:

# ceph orchestrator host add admin.octopus_test1.com
Added host 'admin.octopus_test1.com'
rjfd commented 4 years ago

@sebastian-philipp could you take a look at this. I believe the problem is due to a mismatch between short hostnames and FQDNs that need to be solved in the cephadm mgr module.

sebastian-philipp commented 4 years ago

Could you please add the output of

ceph orchestrator host ls
ceph {osd|mon|mgr} metadata

?

rjfd commented 4 years ago

I have a cluster with the same error and here's the output of the commands:

HEALTH_WARN 3 stray host(s) with 12 service(s) not managed by cephadm
[WRN] CEPHADM_STRAY_HOST: 3 stray host(s) with 12 service(s) not managed by cephadm
    stray host node1.test.com has 4 stray daemons: ['mgr.zbxtqs', 'mon.node1.test.com', 'osd.0', 'osd.1']
    stray host node2.test.com has 4 stray daemons: ['mgr.kjsbck', 'mon.node2', 'osd.2', 'osd.3']
    stray host node3.test.com has 4 stray daemons: ['mgr.awsyva', 'mon.node3', 'osd.4', 'osd.5']

node1:~ # ceph orchestrator host ls
HOST  LABELS 
node2        
admin        
node3        
node1        

ceph mon metadata
[
    {
        "name": "node1.test.com",
        "addrs": "[v2:10.20.163.201:3300/0,v1:10.20.163.201:6789/0]",
        "arch": "x86_64",
        "ceph_release": "octopus",
        "ceph_version": "ceph version 15.0.0-10021-g36f5e3b (36f5e3b115584df022186965dbcb29282376d091) octopus (dev)",
        "ceph_version_short": "15.0.0-10021-g36f5e3b",
        "compression_algorithms": "none, snappy, zlib, zstd, lz4",
        "container_hostname": "node1",
        "container_image": "docker.io/ceph/daemon-base:latest-master-devel",
        "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz",
        "device_ids": "",
        "device_paths": "vda=/dev/disk/by-path/pci-0000:00:03.0",
        "devices": "vda",
        "distro": "centos",
        "distro_description": "CentOS Linux 8 (Core)",
        "distro_version": "8",
        "hostname": "node1.test.com",
        "kernel_description": "#1 SMP Mon Nov 4 04:30:26 UTC 2019 (242bd27)",
        "kernel_version": "5.3.8-lp152.2-default",
        "mem_swap_kb": "0",
        "mem_total_kb": "2027176",
        "os": "Linux"
    },
    {
        "name": "node2",
        "addrs": "[v2:10.20.163.202:3300/0,v1:10.20.163.202:6789/0]",
        "arch": "x86_64",
        "ceph_release": "octopus",
        "ceph_version": "ceph version 15.0.0-10021-g36f5e3b (36f5e3b115584df022186965dbcb29282376d091) octopus (dev)",
        "ceph_version_short": "15.0.0-10021-g36f5e3b",
        "compression_algorithms": "none, snappy, zlib, zstd, lz4",
        "container_hostname": "node2",
        "container_image": "docker.io/ceph/daemon-base:latest-master-devel",
        "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz",
        "device_ids": "",
        "device_paths": "vda=/dev/disk/by-path/pci-0000:00:03.0",
        "devices": "vda",
        "distro": "centos",
        "distro_description": "CentOS Linux 8 (Core)",
        "distro_version": "8",
        "hostname": "node2.test.com",
        "kernel_description": "#1 SMP Mon Nov 4 04:30:26 UTC 2019 (242bd27)",
        "kernel_version": "5.3.8-lp152.2-default",
        "mem_swap_kb": "0",
        "mem_total_kb": "2027176",
        "os": "Linux"
    },
    {
        "name": "node3",
        "addrs": "[v2:10.20.163.203:3300/0,v1:10.20.163.203:6789/0]",
        "arch": "x86_64",
        "ceph_release": "octopus",
        "ceph_version": "ceph version 15.0.0-10021-g36f5e3b (36f5e3b115584df022186965dbcb29282376d091) octopus (dev)",
        "ceph_version_short": "15.0.0-10021-g36f5e3b",
        "compression_algorithms": "none, snappy, zlib, zstd, lz4",
        "container_hostname": "node3",
        "container_image": "docker.io/ceph/daemon-base:latest-master-devel",
        "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz",
        "device_ids": "",
        "device_paths": "vda=/dev/disk/by-path/pci-0000:00:03.0",
        "devices": "vda",
        "distro": "centos",
        "distro_description": "CentOS Linux 8 (Core)",
        "distro_version": "8",
        "hostname": "node3.test.com",
        "kernel_description": "#1 SMP Mon Nov 4 04:30:26 UTC 2019 (242bd27)",
        "kernel_version": "5.3.8-lp152.2-default",
        "mem_swap_kb": "0",
        "mem_total_kb": "2027176",
        "os": "Linux"
    }
]
node1:~ # 

node1:~ # ceph mgr metadata
[
    {
        "name": "awsyva",
        "addr": "10.20.163.203",
        "addrs": "10.20.163.203:0/1",
        "arch": "x86_64",
        "ceph_release": "octopus",
        "ceph_version": "ceph version 15.0.0-10021-g36f5e3b (36f5e3b115584df022186965dbcb29282376d091) octopus (dev)",
        "ceph_version_short": "15.0.0-10021-g36f5e3b",
        "container_hostname": "node3",
        "container_image": "docker.io/ceph/daemon-base:latest-master-devel",
        "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz",
        "distro": "centos",
        "distro_description": "CentOS Linux 8 (Core)",
        "distro_version": "8",
        "hostname": "node3.test.com",
        "kernel_description": "#1 SMP Mon Nov 4 04:30:26 UTC 2019 (242bd27)",
        "kernel_version": "5.3.8-lp152.2-default",
        "mem_swap_kb": "0",
        "mem_total_kb": "2027176",
        "os": "Linux"
    },
    {
        "name": "kjsbck",
        "addr": "10.20.163.202",
        "addrs": "10.20.163.202:0/1",
        "arch": "x86_64",
        "ceph_release": "octopus",
        "ceph_version": "ceph version 15.0.0-10021-g36f5e3b (36f5e3b115584df022186965dbcb29282376d091) octopus (dev)",
        "ceph_version_short": "15.0.0-10021-g36f5e3b",
        "container_hostname": "node2",
        "container_image": "docker.io/ceph/daemon-base:latest-master-devel",
        "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz",
        "distro": "centos",
        "distro_description": "CentOS Linux 8 (Core)",
        "distro_version": "8",
        "hostname": "node2.test.com",
        "kernel_description": "#1 SMP Mon Nov 4 04:30:26 UTC 2019 (242bd27)",
        "kernel_version": "5.3.8-lp152.2-default",
        "mem_swap_kb": "0",
        "mem_total_kb": "2027176",
        "os": "Linux"
    },
    {
        "name": "zbxtqs",
        "addr": "10.20.163.201",
        "addrs": "10.20.163.201:0/1",
        "arch": "x86_64",
        "ceph_release": "octopus",
        "ceph_version": "ceph version 15.0.0-10021-g36f5e3b (36f5e3b115584df022186965dbcb29282376d091) octopus (dev)",
        "ceph_version_short": "15.0.0-10021-g36f5e3b",
        "container_hostname": "node1",
        "container_image": "docker.io/ceph/daemon-base:latest-master-devel",
        "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz",
        "distro": "centos",
        "distro_description": "CentOS Linux 8 (Core)",
        "distro_version": "8",
        "hostname": "node1.test.com",
        "kernel_description": "#1 SMP Mon Nov 4 04:30:26 UTC 2019 (242bd27)",
        "kernel_version": "5.3.8-lp152.2-default",
        "mem_swap_kb": "0",
        "mem_total_kb": "2027176",
        "os": "Linux"
    }
]
node1:~ # 

node1:~ # ceph osd metadata
[
    {
        "id": 0,
        "arch": "x86_64",
        "back_addr": "[v2:10.20.163.201:6802/1,v1:10.20.163.201:6803/1]",
        "back_iface": "eth1",
        "bluefs": "1",
        "bluefs_dedicated_db": "0",
        "bluefs_dedicated_wal": "0",
        "bluefs_single_shared_device": "1",
        "bluestore_bdev_access_mode": "blk",
        "bluestore_bdev_block_size": "4096",
        "bluestore_bdev_dev_node": "/dev/dm-1",
        "bluestore_bdev_devices": "vdb",
        "bluestore_bdev_driver": "KernelDevice",
        "bluestore_bdev_partition_path": "/dev/dm-1",
        "bluestore_bdev_rotational": "1",
        "bluestore_bdev_size": "8585740288",
        "bluestore_bdev_support_discard": "0",
        "bluestore_bdev_type": "hdd",
        "ceph_release": "octopus",
        "ceph_version": "ceph version 15.0.0-10021-g36f5e3b (36f5e3b115584df022186965dbcb29282376d091) octopus (dev)",
        "ceph_version_short": "15.0.0-10021-g36f5e3b",
        "container_hostname": "node1",
        "container_image": "docker.io/ceph/daemon-base:latest-master-devel",
        "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz",
        "default_device_class": "hdd",
        "device_ids": "vdb=590749",
        "device_paths": "vdb=/dev/disk/by-path/pci-0000:00:04.0",
        "devices": "vdb",
        "distro": "centos",
        "distro_description": "CentOS Linux 8 (Core)",
        "distro_version": "8",
        "front_addr": "[v2:10.20.163.201:6800/1,v1:10.20.163.201:6801/1]",
        "front_iface": "eth1",
        "hb_back_addr": "[v2:10.20.163.201:6806/1,v1:10.20.163.201:6807/1]",
        "hb_front_addr": "[v2:10.20.163.201:6804/1,v1:10.20.163.201:6805/1]",
        "hostname": "node1.test.com",
        "journal_rotational": "1",
        "kernel_description": "#1 SMP Mon Nov 4 04:30:26 UTC 2019 (242bd27)",
        "kernel_version": "5.3.8-lp152.2-default",
        "mem_swap_kb": "0",
        "mem_total_kb": "2027176",
        "network_numa_nodes": "-1",
        "objectstore_numa_unknown_devices": "vdb",
        "os": "Linux",
        "osd_data": "/var/lib/ceph/osd/ceph-0",
        "osd_objectstore": "bluestore",
        "rotational": "1"
    },
...
smithfarm commented 4 years ago 
admin:~ # ceph -s
  cluster:
    id:     8dc510f6-434e-11ea-9e8e-525400039d93
    health: HEALTH_WARN
            1 stray host(s) with 6 service(s) not managed by cephadm

  services:
    mon: 1 daemons, quorum admin.octopus_test1.com (age 74m)
    mgr: sikjie(active, since 74m)
    osd: 4 osds: 4 up (since 73m), 4 in (since 73m)

  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   4.0 GiB used, 28 GiB / 32 GiB avail
    pgs:     

admin:~ # ceph orchestrator host ls
HOST  LABELS 
admin        
admin:~ # hostname
admin.octopus_test1.com
admin:~ # ceph osd metadata
[
    {
        "id": 0,
        "arch": "x86_64",
        "back_addr": "[v2:10.20.70.200:6804/1,v1:10.20.70.200:6805/1]",
        "back_iface": "eth1",
        "bluefs": "1",
        "bluefs_dedicated_db": "0",
        "bluefs_dedicated_wal": "0",
        "bluefs_single_shared_device": "1",
        "bluestore_bdev_access_mode": "blk",
        "bluestore_bdev_block_size": "4096",
        "bluestore_bdev_dev_node": "/dev/dm-0",
        "bluestore_bdev_devices": "vdb",
        "bluestore_bdev_driver": "KernelDevice",
        "bluestore_bdev_partition_path": "/dev/dm-0",
        "bluestore_bdev_rotational": "1",
        "bluestore_bdev_size": "8585740288",
        "bluestore_bdev_support_discard": "0",
        "bluestore_bdev_type": "hdd",
        "ceph_release": "octopus",
        "ceph_version": "ceph version 15.0.0-10067-g6bb36f862f (6bb36f862fb7721a65a4f7a705e3ec0fff46a884) octopus (rc)",
        "ceph_version_short": "15.0.0-10067-g6bb36f862f",
        "container_hostname": "admin.octopus_test1.com",
        "container_image": "registry.opensuse.org/filesystems/ceph/master/upstream/images/ceph/ceph",
        "cpu": "Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz",
        "default_device_class": "hdd",
        "device_ids": "vdb=234527",
        "device_paths": "vdb=/dev/disk/by-path/pci-0000:00:04.0",
        "devices": "vdb",
        "distro": "opensuse-leap",
        "distro_description": "openSUSE Leap 15.2 Alpha",
        "distro_version": "15.2",
        "front_addr": "[v2:10.20.70.200:6802/1,v1:10.20.70.200:6803/1]",
        "front_iface": "eth1",
        "hb_back_addr": "[v2:10.20.70.200:6808/1,v1:10.20.70.200:6809/1]",
        "hb_front_addr": "[v2:10.20.70.200:6806/1,v1:10.20.70.200:6807/1]",
        "hostname": "admin.octopus_test1.com",
        "journal_rotational": "1",
        "kernel_description": "#1 SMP Thu Nov 14 18:57:19 UTC 2019 (041cc7d)",
        "kernel_version": "5.3.11-lp152.1-default",
        "mem_swap_kb": "0",
        "mem_total_kb": "4022716",
        "network_numa_nodes": "-1",
        "objectstore_numa_unknown_devices": "vdb",
        "os": "Linux",
        "osd_data": "/var/lib/ceph/osd/ceph-0",
        "osd_objectstore": "bluestore",
        "rotational": "1"
    },
    {
        "id": 1,
        "arch": "x86_64",
        "back_addr": "[v2:10.20.70.200:6812/1,v1:10.20.70.200:6813/1]",
        "back_iface": "eth1",
        "bluefs": "1",
        "bluefs_dedicated_db": "0",
        "bluefs_dedicated_wal": "0",
        "bluefs_single_shared_device": "1",
        "bluestore_bdev_access_mode": "blk",
        "bluestore_bdev_block_size": "4096",
        "bluestore_bdev_dev_node": "/dev/dm-1",
        "bluestore_bdev_devices": "vdc",
        "bluestore_bdev_driver": "KernelDevice",
        "bluestore_bdev_partition_path": "/dev/dm-1",
        "bluestore_bdev_rotational": "1",
        "bluestore_bdev_size": "8585740288",
        "bluestore_bdev_support_discard": "0",
        "bluestore_bdev_type": "hdd",
        "ceph_release": "octopus",
        "ceph_version": "ceph version 15.0.0-10067-g6bb36f862f (6bb36f862fb7721a65a4f7a705e3ec0fff46a884) octopus (rc)",
        "ceph_version_short": "15.0.0-10067-g6bb36f862f",
        "container_hostname": "admin.octopus_test1.com",
        "container_image": "registry.opensuse.org/filesystems/ceph/master/upstream/images/ceph/ceph",
        "cpu": "Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz",
        "default_device_class": "hdd",
        "device_ids": "vdc=944523",
        "device_paths": "vdc=/dev/disk/by-path/pci-0000:00:05.0",
        "devices": "vdc",
        "distro": "opensuse-leap",
        "distro_description": "openSUSE Leap 15.2 Alpha",
        "distro_version": "15.2",
        "front_addr": "[v2:10.20.70.200:6810/1,v1:10.20.70.200:6811/1]",
        "front_iface": "eth1",
        "hb_back_addr": "[v2:10.20.70.200:6816/1,v1:10.20.70.200:6817/1]",
        "hb_front_addr": "[v2:10.20.70.200:6814/1,v1:10.20.70.200:6815/1]",
        "hostname": "admin.octopus_test1.com",
        "journal_rotational": "1",
        "kernel_description": "#1 SMP Thu Nov 14 18:57:19 UTC 2019 (041cc7d)",
        "kernel_version": "5.3.11-lp152.1-default",
        "mem_swap_kb": "0",
        "mem_total_kb": "4022716",
        "network_numa_nodes": "-1",
        "objectstore_numa_unknown_devices": "vdc",
        "os": "Linux",
        "osd_data": "/var/lib/ceph/osd/ceph-1",
        "osd_objectstore": "bluestore",
        "rotational": "1"
    },
    {
        "id": 2,
        "arch": "x86_64",
        "back_addr": "[v2:10.20.70.200:6820/1,v1:10.20.70.200:6821/1]",
        "back_iface": "eth1",
        "bluefs": "1",
        "bluefs_dedicated_db": "0",
        "bluefs_dedicated_wal": "0",
        "bluefs_single_shared_device": "1",
        "bluestore_bdev_access_mode": "blk",
        "bluestore_bdev_block_size": "4096",
        "bluestore_bdev_dev_node": "/dev/dm-2",
        "bluestore_bdev_devices": "vdd",
        "bluestore_bdev_driver": "KernelDevice",
        "bluestore_bdev_partition_path": "/dev/dm-2",
        "bluestore_bdev_rotational": "1",
        "bluestore_bdev_size": "8585740288",
        "bluestore_bdev_support_discard": "0",
        "bluestore_bdev_type": "hdd",
        "ceph_release": "octopus",
        "ceph_version": "ceph version 15.0.0-10067-g6bb36f862f (6bb36f862fb7721a65a4f7a705e3ec0fff46a884) octopus (rc)",
        "ceph_version_short": "15.0.0-10067-g6bb36f862f",
        "container_hostname": "admin.octopus_test1.com",
        "container_image": "registry.opensuse.org/filesystems/ceph/master/upstream/images/ceph/ceph",
        "cpu": "Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz",
        "default_device_class": "hdd",
        "device_ids": "vdd=362968",
        "device_paths": "vdd=/dev/disk/by-path/pci-0000:00:06.0",
        "devices": "vdd",
        "distro": "opensuse-leap",
        "distro_description": "openSUSE Leap 15.2 Alpha",
        "distro_version": "15.2",
        "front_addr": "[v2:10.20.70.200:6818/1,v1:10.20.70.200:6819/1]",
        "front_iface": "eth1",
        "hb_back_addr": "[v2:10.20.70.200:6824/1,v1:10.20.70.200:6825/1]",
        "hb_front_addr": "[v2:10.20.70.200:6822/1,v1:10.20.70.200:6823/1]",
        "hostname": "admin.octopus_test1.com",
        "journal_rotational": "1",
        "kernel_description": "#1 SMP Thu Nov 14 18:57:19 UTC 2019 (041cc7d)",
        "kernel_version": "5.3.11-lp152.1-default",
        "mem_swap_kb": "0",
        "mem_total_kb": "4022716",
        "network_numa_nodes": "-1",
        "objectstore_numa_unknown_devices": "vdd",
        "os": "Linux",
        "osd_data": "/var/lib/ceph/osd/ceph-2",
        "osd_objectstore": "bluestore",
        "rotational": "1"
    },
    {
        "id": 3,
        "arch": "x86_64",
        "back_addr": "[v2:10.20.70.200:6828/1,v1:10.20.70.200:6829/1]",
        "back_iface": "eth1",
        "bluefs": "1",
        "bluefs_dedicated_db": "0",
        "bluefs_dedicated_wal": "0",
        "bluefs_single_shared_device": "1",
        "bluestore_bdev_access_mode": "blk",
        "bluestore_bdev_block_size": "4096",
        "bluestore_bdev_dev_node": "/dev/dm-3",
        "bluestore_bdev_devices": "vde",
        "bluestore_bdev_driver": "KernelDevice",
        "bluestore_bdev_partition_path": "/dev/dm-3",
        "bluestore_bdev_rotational": "1",
        "bluestore_bdev_size": "8585740288",
        "bluestore_bdev_support_discard": "0",
        "bluestore_bdev_type": "hdd",
        "ceph_release": "octopus",
        "ceph_version": "ceph version 15.0.0-10067-g6bb36f862f (6bb36f862fb7721a65a4f7a705e3ec0fff46a884) octopus (rc)",
daemonperf {type.id | path} list|ls [stat-pats] [priority]
        "ceph_version_short": "15.0.0-10067-g6bb36f862f",
        "container_hostname": "admin.octopus_test1.com",
        "container_image": "registry.opensuse.org/filesystems/ceph/master/upstream/images/ceph/ceph",
        "cpu": "Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz",
        "default_device_class": "hdd",
        "device_ids": "vde=608188",
        "device_paths": "vde=/dev/disk/by-path/pci-0000:00:07.0",
        "devices": "vde",
        "distro": "opensuse-leap",
        "distro_description": "openSUSE Leap 15.2 Alpha",
        "distro_version": "15.2",
        "front_addr": "[v2:10.20.70.200:6826/1,v1:10.20.70.200:6827/1]",
        "front_iface": "eth1",
        "hb_back_addr": "[v2:10.20.70.200:6832/1,v1:10.20.70.200:6833/1]",
        "hb_front_addr": "[v2:10.20.70.200:6830/1,v1:10.20.70.200:6831/1]",
        "hostname": "admin.octopus_test1.com",
        "journal_rotational": "1",
        "kernel_description": "#1 SMP Thu Nov 14 18:57:19 UTC 2019 (041cc7d)",
        "kernel_version": "5.3.11-lp152.1-default",
        "mem_swap_kb": "0",
        "mem_total_kb": "4022716",
        "network_numa_nodes": "-1",
        "objectstore_numa_unknown_devices": "vde",
        "os": "Linux",
        "osd_data": "/var/lib/ceph/osd/ceph-3",
        "osd_objectstore": "bluestore",
        "rotational": "1"
    }
]
admin:~ # ceph mon metadata
[
    {
        "name": "admin.octopus_test1.com",
        "addrs": "[v2:10.20.70.200:3300/0,v1:10.20.70.200:6789/0]",
        "arch": "x86_64",
        "ceph_release": "octopus",
        "ceph_version": "ceph version 15.0.0-10067-g6bb36f862f (6bb36f862fb7721a65a4f7a705e3ec0fff46a884) octopus (rc)",
        "ceph_version_short": "15.0.0-10067-g6bb36f862f",
        "compression_algorithms": "none, snappy, zlib, zstd, lz4",
        "container_hostname": "admin.octopus_test1.com",
        "container_image": "registry.opensuse.org/filesystems/ceph/master/upstream/images/ceph/ceph",
        "cpu": "Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz",
        "device_ids": "",
        "device_paths": "vda=/dev/disk/by-path/pci-0000:00:03.0",
        "devices": "vda",
        "distro": "opensuse-leap",
        "distro_description": "openSUSE Leap 15.2 Alpha",
        "distro_version": "15.2",
        "hostname": "admin.octopus_test1.com",
        "kernel_description": "#1 SMP Thu Nov 14 18:57:19 UTC 2019 (041cc7d)",
        "kernel_version": "5.3.11-lp152.1-default",
        "mem_swap_kb": "0",
        "mem_total_kb": "4022716",
        "os": "Linux"
    }
]
admin:~ # ceph mgr metadata
[
    {
        "name": "sikjie",
        "addr": "10.20.70.200",
        "addrs": "10.20.70.200:0/1",
        "arch": "x86_64",
        "ceph_release": "octopus",
        "ceph_version": "ceph version 15.0.0-10067-g6bb36f862f (6bb36f862fb7721a65a4f7a705e3ec0fff46a884) octopus (rc)",
        "ceph_version_short": "15.0.0-10067-g6bb36f862f",
        "container_hostname": "admin.octopus_test1.com",
        "container_image": "registry.opensuse.org/filesystems/ceph/master/upstream/images/ceph/ceph",
        "cpu": "Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz",
        "distro": "opensuse-leap",
        "distro_description": "openSUSE Leap 15.2 Alpha",
        "distro_version": "15.2",
        "hostname": "admin.octopus_test1.com",
        "kernel_description": "#1 SMP Thu Nov 14 18:57:19 UTC 2019 (041cc7d)",
        "kernel_version": "5.3.11-lp152.1-default",
        "mem_swap_kb": "0",
        "mem_total_kb": "4022716",
        "os": "Linux"
    }
]
smithfarm commented 4 years ago

@sebastian-philipp This bug is easy to reproduce with:

sesdev create octopus --roles="[admin,mon,mgr,storage]" --ceph-bootstrap-repo https://github.com/SUSE/ceph-bootstrap.git --ceph-bootstrap-branch master --ceph-container-image registry.opensuse.org/filesystems/ceph/master/upstream/images/ceph/ceph --num-disks 4 octopus_test1

sebastian-philipp commented 4 years ago

https://tracker.ceph.com/issues/43898

smithfarm commented 4 years ago

Not happening anymore with ceph v15.2.0