Configure openstack volumes and openstack client key via ceph-ansible

fultonj commented 7 years ago

ceph-ansible had an option to make it create the openstack volumes in ceph. how do I use this option when I deploy ceph in containers?

https://github.com/RHsyseng/conv-osp-ceph/blob/osp8_ceph1.3/ceph-ansible-diff/roles/ceph-mon/tasks/openstack_config.yml

fultonj commented 7 years ago

Testing simply using this:

https://github.com/ceph/ceph-ansible/blob/fdc78660720e25a0de956c56af5bf1dc5f52408e/roles/ceph-mon/defaults/main.yml#L43

fultonj commented 7 years ago

Adding openstack_config: true to ceph-ansible.yaml seems to have done the trick.

2017-06-18 23:57:51Z [overcloud.AllNodesDeploySteps]: CREATE_COMPLETE  Stack CREATE completed successfully
2017-06-18 23:57:51Z [overcloud.AllNodesDeploySteps]: CREATE_COMPLETE  state changed
2017-06-18 23:57:51Z [overcloud]: CREATE_COMPLETE  Stack CREATE completed successfully

 Stack overcloud CREATE_COMPLETE 

/home/stack/.ssh/known_hosts updated.
Original contents retained as /home/stack/.ssh/known_hosts.old
Overcloud Endpoint: http://192.168.24.8:5000/v2.0
Overcloud Deployed

real    42m26.742s
user    0m4.800s
sys 0m0.513s
(undercloud) [stack@undercloud tripleo-ceph-ansible]$ ./sanity-check.sh 
 --------- ceph -s --------- 
192.168.24.13 | SUCCESS | rc=0 >>
    cluster b007365e-cc87-11e5-a877-525400330ee0
     health HEALTH_OK
     monmap e1: 1 mons at {overcloud-controller-0=192.168.24.13:6789/0}
            election epoch 3, quorum 0 overcloud-controller-0
     osdmap e17: 6 osds: 6 up, 6 in
            flags sortbitwise,require_jewel_osds
      pgmap v35: 96 pgs, 5 pools, 0 bytes data, 0 objects
            199 MB used, 4381 MB / 4581 MB avail
                  96 active+clean

 --------- ceph df --------- 
192.168.24.13 | SUCCESS | rc=0 >>
GLOBAL:
    SIZE      AVAIL     RAW USED     %RAW USED 
    4581M     4381M         199M          4.36 
POOLS:
    NAME        ID     USED     %USED     MAX AVAIL     OBJECTS 
    rbd         0         0         0          729M           0 
    images      1         0         0          729M           0 
    volumes     2         0         0          729M           0 
    vms         3         0         0          729M           0 
    backups     4         0         0          729M           0 

(undercloud) [stack@undercloud tripleo-ceph-ansible]$

fultonj commented 7 years ago

I am re-opening this as we will need it to configured metrics for gnochi too. Also, I don't want to close it until we have verified that key setup worked as well.

fultonj commented 7 years ago

todo: send pr to ceph-ansible to add metrics to pool list

fultonj commented 7 years ago

https://github.com/ceph/ceph-ansible/pull/1612

fultonj commented 7 years ago

metrics are done. key for clients are being worked on in ceph/ceph-ansible#1617 . Once on the mistral side i can pass this param and see the right things on the ceph server i will close this

fultonj commented 7 years ago

ceph-ansible updated to address ceph/ceph-ansible#1617 .
THT updated to use it by overriding the openstack_keys parameter
Deployment fails on create openstack key(s) task
Error message: TASK [ceph-mon : create openstack key(s)] **\ failed: [192.168.24.7] (item={u\'mon_cap\': u"mon \'allow r\'", u\'osd_cap\': u"mon \'allow r\' osd \'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool=metrics\'", u\'name\': u\'client.openstack\', u\'key\': u\'AQD+aFFZAAAAABAAbgiA+cxCz0XWpodtf71e4w==\'}) => {"changed": false, "cmd": "\\"docker exec ceph-mon-overcloud-controller-2 ceph-authtool -C /etc/ceph/ceph.client.openstack.keyring --name client.openstack --add-key AQD+aFFZAAAAABAAbgiA+cxCz0XWpodtf71e4w== --cap mon \'allow r\' --cap mon \'allow r\' osd \'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool=metrics\'\\"", "delta": "0:00:00.004327", "end": "2017-06-26 23:39:42.628249", "failed": true, "item": {"key": "AQD+aFFZAAAAABAAbgiA+cxCz0XWpodtf71e4w==", "mon_cap": "mon \'allow r\'", "name": "client.openstack", "osd_cap": "mon \'allow r\' osd \'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool=metrics\'"}, "rc": 127, "start": "2017-06-26 23:39:42.623922", "stderr": "/bin/sh: docker exec ceph-mon-overcloud-controller-2 ceph-authtool -C /etc/ceph/ceph.client.openstack.keyring --name client.openstack --add-key AQD+aFFZAAAAABAAbgiA+cxCz0XWpodtf71e4w== --cap mon \'allow r\' --cap mon \'allow r\' osd \'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool=metrics\': No such file or directory", "stdout": "", "stdout_lines": [], "warnings": []}

fultonj commented 7 years ago

I fixed one problem in my THT copy and posted the suggestion to the review.
However, I am still getting the same error and I think it has to do with how the passed string is being parsed.

When I run the command that should be run directly things are fine:

[root@overcloud-controller-2 ~]# docker exec ceph-mon-overcloud-controller-2 ceph-authtool -C /etc/ceph/ceph.client.openstack.keyring --name client.openstack --add-key AQD+aFFZAAAAABAAbgiA+cxCz0XWpodtf71e4w== --cap mon 'allow r' --cap osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool=metrics' 
creating /etc/ceph/ceph.client.openstack.keyring
added entity client.openstack auth auth(auid = 18446744073709551615 key=AQD+aFFZAAAAABAAbgiA+cxCz0XWpodtf71e4w== with 0 caps)
[root@overcloud-controller-2 ~]#

However, when run by ceph-ansible (with the correction to remove "mon 'allow r'" from osd_cap) I get

failed: [192.168.24.7] (item={u\\'mon_cap\\': u"mon \\'allow r\\'", u\\'osd_cap\\': u"osd \\'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool=metrics\\'", u\\'name\\': u\\'client.openstack\\', u\\'key\\': u\\'AQD+aFFZAAAAABAAbgiA+cxCz0XWpodtf71e4w==\\'}) => {"changed": false, "cmd": "\\\\"docker exec ceph-mon-overcloud-controller-2 ceph-authtool -C /etc/ceph/ceph.client.openstack.keyring --name client.openstack --add-key AQD+aFFZAAAAABAAbgiA+cxCz0XWpodtf71e4w== --cap mon \\'allow r\\' --cap osd \\'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool=metrics\\'\\\\"", "delta": "0:00:00.002465", "end": "2017-06-27 01:46:07.364106", "failed": true, "item": {"key": "AQD+aFFZAAAAABAAbgiA+cxCz0XWpodtf71e4w==", "mon_cap": "mon \\'allow r\\'", "name": "client.openstack", "osd_cap": "osd \\'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool=metrics\\'"}, "rc": 127, "start": "2017-06-27 01:46:07.361641", "stderr": "/bin/sh: docker exec ceph-mon-overcloud-controller-2 ceph-authtool -C /etc/ceph/ceph.client.openstack.keyring --name client.openstack --add-key AQD+aFFZAAAAABAAbgiA+cxCz0XWpodtf71e4w== --cap mon \\'allow r\\' --cap osd \\'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool=metrics\\': No such file or directory", "stdout": "", "stdout_lines": [], "warnings": []}

fultonj commented 7 years ago

The raw result from the database is at http://sprunge.us/DLTc (see below)
cat this file through | jq . | sed -e 's/\\n/\n/g' -e 's/\\"/"/g' to read it better

(undercloud) [stack@undercloud tripleo-ceph-ansible]$ mistral task-get-result $TASK_ID | curl -F 'sprunge=<-' http://sprunge.us 
http://sprunge.us/DLTc
(undercloud) [stack@undercloud tripleo-ceph-ansible]$

fultonj commented 7 years ago

This PR (https://github.com/ceph/ceph-ansible/pull/1629) allowed the deployment to finish and I got a configured ceph cluster.
However, I think it just obscured the issue as there is no openstack keyring file on the controller node. The report from ceph-ansible on the task implies that the task ran and returned "OK" but we can see the command output as Error ENOENT: failed to find client.openstack in keyring. Full task output below from two overcloud nodes.

ok: [192.168.24.16] => (item=[{u'mon_cap': u"mon 'allow r'", u'osd_cap': u"osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool=metrics'", u'name': u'client.openstack', u'key': u'AQAckVFZAAAAABAAROP9g8AnWwZ6ldJadH8nvg=='}, {u'_ansible_parsed': True, u'cmd': [u'docker', u'exec', u'ceph-mon-overcloud-controller-0', u'ceph', u'--cluster', u'ceph', u'auth', u'get', u'client.openstack'], u'end': u'2017-06-27 12:52:32.027921', u'_ansible_no_log': False, u'stdout': u'', u'_ansible_item_result': True, u'changed': False, u'item': {u'mon_cap': u"mon 'allow r'", u'osd_cap': u"osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool=metrics'", u'name': u'client.openstack', u'key': u'AQAckVFZAAAAABAAROP9g8AnWwZ6ldJadH8nvg=='}, u'delta': u'0:00:00.209828', u'stderr': u'Error ENOENT: failed to find client.openstack in keyring', u'rc': 2, u'invocation': {u'module_name': u'command', u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': False, u'_raw_params': u'docker exec ceph-mon-overcloud-controller-0 ceph --cluster ceph auth get client.openstack', u'removes': None, u'warn': True, u'chdir': None}}, u'stdout_lines': [], u'failed_when_result': False, u'start': u'2017-06-27 12:52:31.818093', u'warnings': [], u'failed': False}])

TASK [ceph-mon : add openstack key(s) to ceph] ***** ok: [192.168.24.7] => (item=[{u'mon_cap': u"mon 'allow r'", u'osd_cap': u"osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool=metrics'", u'name': u'client.openstack', u'key': u'AQD+aFFZAAAAABAAbgiA+cxCz0XWpodtf71e4w=='}, {u'_ansible_parsed': True, u'cmd': [u'docker', u'exec', u'ceph-mon-overcloud-controller-2', u'ceph', u'--cluster', u'ceph', u'auth', u'get', u'client.openstack'], u'end': u'2017-06-27 12:56:04.327272', u'_ansible_no_log': False, u'stdout': u'', u'_ansible_item_result': True, u'changed': False, u'item': {u'mon_cap': u"mon 'allow r'", u'osd_cap': u"osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool=metrics'", u'name': u'client.openstack', u'key': u'AQD+aFFZAAAAABAAbgiA+cxCz0XWpodtf71e4w=='}, u'delta': u'0:00:00.222394', u'stderr': u'Error ENOENT: failed to find client.openstack in keyring', u'rc': 2, u'invocation': {u'module_name': u'command', u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': False, u'_raw_params': u'docker exec ceph-mon-overcloud-controller-2 ceph --cluster ceph auth get client.openstack', u'removes': None, u'warn': True, u'chdir': None}}, u'stdout_lines': [], u'failed_when_result': False, u'start': u'2017-06-27 12:56:04.104878', u'warnings': [], u'failed': False}])

fultonj commented 7 years ago

Fresh deploy not having this issue...

[root@overcloud-controller-2 ~]# ceph-authtool -l /etc/ceph/ceph.client.openstack.keyring
[client.openstack]
    key = AQD+aFFZAAAAABAAbgiA+cxCz0XWpodtf71e4w==
    caps mon = "allow r"
    caps osd = "allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool=metrics"
[root@overcloud-controller-2 ~]#

fultonj commented 7 years ago

False alarm. We're fine with the openstack key on the server side. As per the command below on one of the nodes where I just ran the update:

The file ceph.client.openstack.keyring exists on only one of the controllers
But if you ask any controller to ceph auth list you can see they all know about the key.

So it's not just that it only works on a new deploy, it worked on the update too, you just need to test it correctly. Closing this issue as I think we just need to get it working on the puppet-ceph side now as per https://github.com/fultonj/tripleo-ceph-ansible/issues/17

(undercloud) [stack@undercloud tripleo-ceph-ansible]$ for ip in $(echo 192.168.24.{9,11,16}); do mon=$ip ; ansible all -i $mon, -u heat-admin  -b -m shell -a "ceph auth list | grep client.openstack -A 6 ; ls -l /etc/ceph/"; done
192.168.24.9 | SUCCESS | rc=0 >>
client.openstack
    key: AQAckVFZAAAAABAAROP9g8AnWwZ6ldJadH8nvg==
    caps: [mon] allow r
    caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool=metrics
total 16
-rw-r--r--. 1 root root 137 Jun 27 04:29 ceph.client.admin.keyring
-rw-r--r--. 1 root root 711 Jun 27 04:29 ceph.conf
-rw-r--r--. 1 root root 553 Jun 27 04:29 ceph.mon.keyring
-rw-r--r--. 1 root root  92 Apr 11 01:53 rbdmapinstalled auth entries:

192.168.24.11 | SUCCESS | rc=0 >>
client.openstack
    key: AQAckVFZAAAAABAAROP9g8AnWwZ6ldJadH8nvg==
    caps: [mon] allow r
    caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool=metrics
total 16
-rw-------. 1 root root 137 Jun 27 05:28 ceph.client.admin.keyring
-rw-r--r--. 1 root root 711 Jun 27 05:28 ceph.conf
-rw-------. 1 ceph ceph 553 Jun 27 05:28 ceph.mon.keyring
-rw-r--r--. 1 root root  92 Apr 11 01:53 rbdmapinstalled auth entries:

192.168.24.16 | SUCCESS | rc=0 >>
client.openstack
    key: AQAckVFZAAAAABAAROP9g8AnWwZ6ldJadH8nvg==
    caps: [mon] allow r
    caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=backups, allow rwx pool=vms, allow rwx pool=images, allow rwx pool=metrics
total 20
-rw-r--r--. 1 root root 137 Jun 27 04:31 ceph.client.admin.keyring
-rw-------. 1 root root 262 Jun 27 12:52 ceph.client.openstack.keyring
-rw-r--r--. 1 root root 711 Jun 27 04:30 ceph.conf
-rw-r--r--. 1 root root 553 Jun 27 04:31 ceph.mon.keyring
-rw-r--r--. 1 root root  92 Apr 11 01:53 rbdmapinstalled auth entries:

(undercloud) [stack@undercloud tripleo-ceph-ansible]$

fultonj commented 7 years ago

The question might come up, if the file ceph.client.openstack.keyring exists on only one of the controllers, then how will the OpenStack services which use Ceph (e.g. Glance) connect to the Ceph cluster?

This won't be an issue because those services will have their key created by puppet-ceph on the client side. In that sense a key of the same content will be there for them in their container. Asking the original question but saying instead "exists on only one ceph-monitor container" makes this a little more clear.

fultonj / tripleo-ceph-ansible

Configure openstack volumes and openstack client key via ceph-ansible #14