[stable-5.0] Impossible to add an OSD on an existing cluster

[tlebars]

Bug Report

What happened: Cannot add some new OSDs after the first run.

What you expected to happen: New OSDs are created when new devices are added

How to reproduce it (minimal and precise):

1. Launch a first run to deploy the Ceph cluster with some OSDs
2. Add a new device (OSD) to the inventory
3. Replay the playbook and find that the new device (OSD) is not integrated in the cluster.

When using the command executed by Ansible directly on the server, here is the error returned:

root@serv2 # ceph-volume lvm batch --bluestore --yes --block-db-size '33285996544' /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde --db-devices /dev/sdf /dev/sdg --report
-->  RuntimeError: 2 devices were filtered in non-interactive mode, bailing out

Configuration Files

group_vars:

---
# all.yml
dummy:
ntp_service_enabled: false
upgrade_ceph_packages: True

ceph_origin: repository
ceph_repository: community
ceph_stable_release: octopus
public_network: "X.X.X.X/X"
cluster_network: "X.X.X.X/X"
monitor_interface: XXX

generate_fsid: true
block_db_size: 33285996544 
ceph_tcmalloc_max_total_thread_cache: 134217728
dashboard_enabled: False

---
# mons.yml
dummy:
monitor_secret: "{{ monitor_keyring.stdout }}"
admin_secret: 'XXXXX'

---
# osds.yml
dummy:
osd_auto_discovery: false

Inventory:

---
# hosts file
all:
  children:
    mons:
      hosts:
        serv1:
        serv2
        serv3:
    osds:
      hosts:
        serv1:
          devices:
            - /dev/sda
            - /dev/sdb
            - /dev/sdc
            - /dev/sdd
            - /dev/sdg
          dedicated_devices:
            - /dev/sde
            - /dev/sdf
        serv2:
          devices:
            - /dev/sda
            - /dev/sdb
            - /dev/sdc
            - /dev/sdd
            - /dev/sde
          dedicated_devices:
            - /dev/sdf
            - /dev/sdg
        serv3:
          devices:
            - /dev/sda
            - /dev/sdb
            - /dev/sdc
            - /dev/sdd
            - /dev/sde
          dedicated_devices:
            - /dev/sdf
            - /dev/sdg
        serv4:
          devices:
            - /dev/sda
            - /dev/sdb
            - /dev/sdc
            - /dev/sdd
            - /dev/sde
          dedicated_devices:
            - /dev/sdf
            - /dev/sdg

Environment:

• OS (e.g. from /etc/os-release):

  NAME="Ubuntu"
  VERSION="20.04.1 LTS (Focal Fossa)"
  ID=ubuntu
  ID_LIKE=debian
  PRETTY_NAME="Ubuntu 20.04.1 LTS"
  VERSION_ID="20.04"
  HOME_URL="https://www.ubuntu.com/"
  SUPPORT_URL="https://help.ubuntu.com/"
  BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
  PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
  VERSION_CODENAME=focal
  UBUNTU_CODENAME=focal

• Kernel (e.g. uname -a):

  Linux serv1 5.4.0-51-generic #56-Ubuntu SMP Mon Oct 5 14:28:49 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

• Ansible version (e.g. ansible-playbook --version):

  ansible-playbook 2.9.6
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  executable location = /usr/bin/ansible-playbook
  python version = 3.8.5 (default, Jul 28 2020, 12:59:40) [GCC 9.3.0]

• ceph-ansible version (e.g. git head or tag or stable branch):

  stable-5.0

• Ceph version (e.g. ceph -v):

  ceph version 15.2.5 (2c93eff00150f0cc5f106a559557a58d3d7b6f1f) octopus (stable)

  Logs: There are no errors in the ceph-ansible logs. The problem seems to be at this level, where nothing is happening while a new device has just been added (/dev/sde):

  TASK [ceph-osd : use ceph-volume lvm batch to create bluestore osds] *************************************************************************************************************************
  Friday 16 October 2020  16:59:30 +0200 (0:00:00.122)       0:02:05.850 ******** 
  ok: [serv2] => changed=false 
  cmd:
  - ceph-volume
  - --cluster
  - ceph
  - lvm
  - batch
  - --bluestore
  - --yes
  - --block-db-size
  - '33285996544'
  - /dev/sda
  - /dev/sdb
  - /dev/sdc
  - /dev/sdd
  - /dev/sde
  - --db-devices
  - /dev/sdf
  - /dev/sdg
  - --report
  - --format=json
  delta: '0:00:02.195371'
  end: '2020-10-16 16:59:32.533778'
  rc: 0
  start: '2020-10-16 16:59:30.338407'
  stderr: ''
  stderr_lines: <omitted>
  stdout: '{"changed": false, "osds": [], "vgs": []}'
  stdout_lines: <omitted>

[dsavineau]

Could you share with us the devices capacity (for both bluestore data and db) ?

Are you sure to have enough space left on the db devices (sdf and sdg) for another 31G db logical volume ?

As per [1], if the db size can't be satified then the OSD isn't deployed.

[1] https://docs.ceph.com/en/latest/ceph-volume/lvm/batch/#explicit-sizing

[tlebars]

Hi,

Disk space does not seem to be the problem here. It is only a matter of adding a new OSD. For example, if at the first run of ceph-ansible, I directly integrate all the disks, it works without any problem.

root@serv2:~ (0) # lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0   7,3T  0 disk
sdb      8:16   0   7,3T  0 disk
sdc      8:32   0   7,3T  0 disk
sdd      8:48   0   7,3T  0 disk
sdf      8:64   0 447,1G  0 disk
sdg      8:80   0 447,1G  0 disk
sde      8:96   0   7,3T  0 disk
sdh      8:112  0 111,3G  0 disk
├─sdh1   8:113  0   512M  0 part /boot/efi
└─sdh2   8:114  0 110,8G  0 part /

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

[aqader]

Is it solved?