Improvement of ansible ceph integration

[thatsdone]

Is your suggestion related to a problem? Please describe.

To improve ceph integration process via opensds-installer in order to avoid troubles (e.g. #255).

Describe the solution you'd like

I would suggest step-by-step improvement like below.

• step1: Add configuration / status checks
  • 1a : pre-flight check for ceph-ansible
    • check if: 'devices:' in ansible/group_vars/ceph/all.yml is empty or not,
      • if not empty, check if the listed devices exist or not,
    • check if 'monitor_interface:' in ansible/group_vars/ceph/all.yml is correct or not,
  • 1b: take 'monitor_interface:' from ansible fact('ansible_default_ipv4.interface')
  • 1c: Add wait logic for ceph stabilization
    • until 'ceph -s' returning HEALTH_OK and all the defined osds are 'up'
  • 1d: Wait for the define ceph pools become visible
    • using 'rados df' that's equivalent of the API osdsdock uses (through ceph-go)
• step2: Make ceph version more configurable
  • 2a: Take ceph-ansible branch name (e.g. stable-3.0) from osdsdock.yml
  • 2b: Update 'install_ansible.sh' to detect appropriate ansible version according to ceph version.
  • Below is a combination matrix of ceph and ansible versions from: https://docs.ceph.com/ceph-ansible/master/, and I would suggest to switch to 'stable-3.1' and keep the ansible version intact at first.

ceph-ansible branch	required ansible	ceph version	notes
stable-3.0	ansible-2.4	jewel/luminous	we use this currently.
stable-3.1	ansible-2.4	luminous/mimic	suggested update
stable-3.2	ansible-2.6	luminous/mimic
stable-4.0	ansible-2.8	nautilus	bionic target?

• step3: Refactoring
  • Enable to use all the ceph rbd features, as is, depending on the target ceph version.
  • Simply removing 'rbd default features = 1' from /etc/ceph.conf
  • (Optionally) Create a loopback device like lvm backend case so that we don't need to care about additional disk devices for osds.
  • Make sure several installation scenarios works as expected.
    1. Build opensds + ceph all-in-one system from scratch
    2. Build opensds + a separate ceph cluster from scratch
    3. Integrate with an existing ceph cluster from scratch
       • As Himanshu pointed out at slack, there is a nit in ceph.conf handling now
    4. Integrate with already/locally configured ceph
    5. etc.

[wisererik]

@thatsdone it's a good suggestion. My friend Joseph will check it. Or are you intereted at implementing this feature?

[thatsdone]

@wisererik I have some working codes for (a part of) step 1 and step 2, and I'm creating PRs. As for step 3, I think it's better discussing details first.

[wisererik]

Sure, we can discuss on slack, it's more efficient.

[thatsdone]

FYI. You can find my working branch below.

https://github.com/thatsdone/opensds-installer/commits/refactor-ceph-ansible

[wisererik]

Sure, it's better you submit a PR so that I can know the code changes well

[thatsdone]

After adding waiting logic for ceph stabilization, it turned out that we still get the same symptom. So, I looked into osdsdock side, and I noticed one potential (?) issue there. The below is the issue I filed.

https://github.com/opensds/opensds/issues/991

[thatsdone]

Sure, it's better you submit a PR so that I can know the code changes well

I submitted a PR (https://github.com/opensds/opensds-installer/pull/271) for this, implementing step1 and step2. Hope this PR makes sense.

Please note that I updated my original suggestion at the top, and this PR covers all the points from 1a to 2b there.

Regarding step3, I'd like to have a consensus on the priority of installation scenarios at first.

[thatsdone]

So far, @himanshuvar gave me very useful feedbacks, and at the moment this PR has a problem when the target Ubuntu box has existing ceph packages and/or /etc/ceph. Please note if you begin from a fresh Ubuntu 16.04 box WITHOUT any ceph packages, it actually worked including ceph-ansible stable-3.1+mimic. But, anyway in general, it's desirable to take care of existing ceph packages (mostly upgraded).

I'm working on the (new additional sub)issues and submitting another PR soon (hopefully?).

[himanshuvar]

@thatsdone Thank you Please update your PR with the changes you find it relevant to fix now

[thatsdone]

@himanshuvar I submitted updated PR. https://github.com/opensds/opensds-installer/pull/273