ceph-volume: nvme0n1 is not JSON serializable

[discostur]

What happened:

Tried to deploy ceph-ansible cluster with osd-scenario LVM and mixed HDD / NVMe Setup. I got the following error:

 FAILED! => {"changed": true, "cmd": ["ceph-volume", "--cluster", "ceph", "lvm", "batch", "--bluestore", "--yes", "/dev/sdb", "/dev/sdc", "/dev/sdd", "/dev/sde", "/dev/sdf", "/dev/sdg", "/dev/sdh", "/dev/sdi", "/dev/nvme0n1", "--report", "--format=json"], "msg": "non-zero return code", "rc": 1, "stderr": "Traceback (most recent call last):\n  File \"/sbin/ceph-volume\", line 6, in <module>\n    main.Volume()\n  File \"/usr/lib/python2.7/site-packages/ceph_volume/main.py\", line 40, in __init__\n    self.main(self.argv)\n  File \"/usr/lib/python2.7/site-packages/ceph_volume/decorators.py\", line 59, in newfunc\n    return f(*a, **kw)\n  File \"/usr/lib/python2.7/site-packages/ceph_volume/main.py\", line 156, in main\n    terminal.dispatch(self.mapper, subcommand_args)\n  File \"/usr/lib/python2.7/site-packages/ceph_volume/terminal.py\", line 182, in dispatch\n    instance.main()\n  File \"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/main.py\", line 40, in main\n    terminal.dispatch(self.mapper, self.argv)\n  File \"/usr/lib/python2.7/site-packages/ceph_volume/terminal.py\", line 182, in dispatch\n    instance.main()\n  File \"/usr/lib/python2.7/site-packages/ceph_volume/decorators.py\", line 16, in is_root\n    return func(*a, **kw)\n  File \"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/batch.py\", line 236, in main\n    self.report(args)\n  File \"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/batch.py\", line 135, in report\n    strategy.report_json()\n  File \"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/strategies/bluestore.py\", line 146, in report_json\n    print(json.dumps(self.computed, indent=4, sort_keys=True))\n  File \"/usr/lib64/python2.7/json/__init__.py\", line 250, in dumps\n    sort_keys=sort_keys, **kw).encode(obj)\n  File \"/usr/lib64/python2.7/json/encoder.py\", line 209, in encode\n    chunks = list(chunks)\n  File \"/usr/lib64/python2.7/json/encoder.py\", line 434, in _iterencode\n    for chunk in _iterencode_dict(o, _current_indent_level):\n  File \"/usr/lib64/python2.7/json/encoder.py\", line 408, in _iterencode_dict\n    for chunk in chunks:\n  File \"/usr/lib64/python2.7/json/encoder.py\", line 408, in _iterencode_dict\n    for chunk in chunks:\n  File \"/usr/lib64/python2.7/json/encoder.py\", line 442, in _iterencode\n    o = _default(o)\n  File \"/usr/lib64/python2.7/json/encoder.py\", line 184, in default\n    raise TypeError(repr(o) + \" is not JSON serializable\")\nTypeError: set([<Unknown: /dev/nvme0n1>]) is not JSON serializable", "stderr_lines": ["Traceback (most recent call last):", "  File \"/sbin/ceph-volume\", line 6, in <module>", "    main.Volume()", "  File \"/usr/lib/python2.7/site-packages/ceph_volume/main.py\", line 40, in __init__", "    self.main(self.argv)", "  File \"/usr/lib/python2.7/site-packages/ceph_volume/decorators.py\", line 59, in newfunc", "    return f(*a, **kw)", "  File \"/usr/lib/python2.7/site-packages/ceph_volume/main.py\", line 156, in main", "    terminal.dispatch(self.mapper, subcommand_args)", "  File \"/usr/lib/python2.7/site-packages/ceph_volume/terminal.py\", line 182, in dispatch", "    instance.main()", "  File \"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/main.py\", line 40, in main", "    terminal.dispatch(self.mapper, self.argv)", "  File \"/usr/lib/python2.7/site-packages/ceph_volume/terminal.py\", line 182, in dispatch", "    instance.main()", "  File \"/usr/lib/python2.7/site-packages/ceph_volume/decorators.py\", line 16, in is_root", "    return func(*a, **kw)", "  File \"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/batch.py\", line 236, in main", "    self.report(args)", "  File \"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/batch.py\", line 135, in report", "    strategy.report_json()", "  File \"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/strategies/bluestore.py\", line 146, in report_json", "    print(json.dumps(self.computed, indent=4, sort_keys=True))", "  File \"/usr/lib64/python2.7/json/__init__.py\", line 250, in dumps", "    sort_keys=sort_keys, **kw).encode(obj)", "  File \"/usr/lib64/python2.7/json/encoder.py\", line 209, in encode", "    chunks = list(chunks)", "  File \"/usr/lib64/python2.7/json/encoder.py\", line 434, in _iterencode", "    for chunk in _iterencode_dict(o, _current_indent_level):", "  File \"/usr/lib64/python2.7/json/encoder.py\", line 408, in _iterencode_dict", "    for chunk in chunks:", "  File \"/usr/lib64/python2.7/json/encoder.py\", line 408, in _iterencode_dict", "    for chunk in chunks:", "  File \"/usr/lib64/python2.7/json/encoder.py\", line 442, in _iterencode", "    o = _default(o)", "  File \"/usr/lib64/python2.7/json/encoder.py\", line 184, in default", "    raise TypeError(repr(o) + \" is not JSON serializable\")", "TypeError: set([<Unknown: /dev/nvme0n1>]) is not JSON serializable"], "stdout": "", "stdout_lines": []}

Same as i try the command manually:

# ceph-volume --cluster ceph lvm batch --bluestore --yes /dev/nvme0n1 /dev/sdb --report --format=json
-->  TypeError: set([<Unknown: /dev/nvme0n1>]) is not JSON serializable

However, if i try the ceph-volume command with just the NVMe (no HDDs) or leave the "--format=json" command it works:

# ceph-volume --cluster ceph lvm batch --bluestore --yes /dev/nvme0n1 --report --format=json
{
    "osds": [
        {
            "block.db": {}, 
            "data": {
                "human_readable_size": "260.83 GB", 
                "parts": 1, 
                "path": "/dev/nvme0n1", 
                "percentage": 100, 
                "size": 280065171456
            }
        }
    ], 
    "vgs": []
}

# ceph-volume --cluster ceph lvm batch --bluestore --yes /dev/nvme0n1 /dev/sdb --report

Total OSDs: 1

Solid State VG:
  Targets:   block.db                  Total size: 260.83 GB                
  Total LVs: 1                         Size per LV: 260.83 GB                
  Devices:   /dev/nvme0n1

  Type            Path                                                    LV Size         % of device
----------------------------------------------------------------------------------------------------
  [data]          /dev/sdb                                                3.64 TB         100%
  [block.db]      vg: vg/lv                                               260.83 GB       100%

Environment:

• OS (e.g. from /etc/os-release): centos7
• Kernel (e.g. uname -a): 3.10.0-862.14.4.el7.x86_64
• Ansible version (e.g. ansible-playbook --version): 2.6.x
• ceph-ansible version (e.g. git head or tag or stable branch): stable-3.2
• Ceph version (e.g. ceph -v): mimic 13.2.2

[leseb]

@discostur Could you please open your issue on http://tracker.ceph.com/projects/ceph-volume instead as this appears to be a ceph-volume issue? Thanks, please this once that's done.

@alfredodeza you might be interested in that. Thanks.

[andrewschoen]

This is a bug that was fixed when we made lvm batch idempotent. The upstream mimic branch contains the fix, but it has not made it into a release yet.

mimic backport: https://github.com/ceph/ceph/pull/24588

[alfredodeza]

It is part of Nautilus 14.0.1 though (basically a beta release). Closing this as it has been fixed in Ceph, just not released yet as part of a stable version (also ceph-volume, not ceph-ansible related)