'Add rpc_support key to nova' and 'Retrieve authorized keys' failures

[evrardjp]

Hello,

Here are my ansible runs.

TASK [rcbops_openstack-ops : Delete rpc_support keypair in nova] ***************
skipping: [a]
skipping: [a_cinder_scheduler_container-45b88d5e]
skipping: [a_swift_proxy_container-c0cb8b09]
skipping: [a_neutron_server_container-236fcf3e]
skipping: [a_neutron_agents_container-05e0fe11]
skipping: [a_nova_api_os_compute_container-8e58a270]
skipping: [a_memcached_container-cea314a3]
skipping: [a_galera_container-9b94acb9]
skipping: [a_galera_container-ab544444]
skipping: [a_galera_container-95ecb67b]
skipping: [a_rabbit_mq_container-f5f67c59]
skipping: [a_rabbit_mq_container-0e585409]
skipping: [a_rabbit_mq_container-fc7a31e7]
skipping: [a_nova_console_container-5c2e3b78]
skipping: [a_cinder_api_container-bba1113c]
skipping: [a_nova_conductor_container-97084c2c]
skipping: [a_elasticsearch_container-d522f32b]
skipping: [a_horizon_container-cb0a1403]
skipping: [a_horizon_container-6ce496bb]
skipping: [a_keystone_container-31053b50]
skipping: [a_keystone_container-d263b755]
skipping: [a_nova_cert_container-bbd3051d]
skipping: [a_logstash_container-6601e50e]
skipping: [a_repo_container-cab0ec68]
skipping: [a_repo_container-40d7db26]
skipping: [a_glance_container-f5db65df]
skipping: [a_nova_api_metadata_container-8136c345]
skipping: [a_kibana_container-be6fdf21]
skipping: [a_heat_apis_container-1170c42f]
skipping: [a_nova_scheduler_container-d1e4ed8c]
skipping: [a_rsyslog_container-89cc329b]
skipping: [a_heat_engine_container-892dadb9]
changed: [a_utility_container-43ffaf4e]

TASK [rcbops_openstack-ops : Add rpc_support key to nova] **********************
fatal: [a_utility_container-43ffaf4e]: FAILED! => {"changed": true, "cmd": ". /root/openrc\n nova keypair-add --pub-key /root/.ssh/rpc_support.pub rpc_support", "delta": "0:00:02.096563", "end": "2017-06-13 15:06:02.968634", "failed": true, "rc": 1, "start": "2017-06-13 15:06:00.872071", "stderr": "ERROR (Conflict): Key pair 'rpc_support' already exists. (HTTP 409) (Request-ID: req-25bbe6a9-7a1d-47f3-866b-4057e0939304)", "stdout": "", "stdout_lines": [], "warnings": []}

...

TASK [rcbops_openstack-ops : Retrieve authorized keys] *************************
skipping: [a_cinder_scheduler_container-45b88d5e] => (item={u'src': u'/root/.ssh/rpc_support', u'name': u'rpc_support', u'file_mode': u'0600', u'dir_mode': u'0755'})
skipping: [a_swift_proxy_container-c0cb8b09] => (item={u'src': u'/root/.ssh/rpc_support', u'name': u'rpc_support', u'file_mode': u'0600', u'dir_mode': u'0755'})
skipping: [a_cinder_scheduler_container-45b88d5e] => (item={u'src': u'/root/.ssh/rpc_support.pub', u'name': u'rpc_support_pub', u'file_mode': u'0640', u'dir_mode': u'0755'})
skipping: [a_swift_proxy_container-c0cb8b09] => (item={u'src': u'/root/.ssh/rpc_support.pub', u'name': u'rpc_support_pub', u'file_mode': u'0640', u'dir_mode': u'0755'})
skipping: [a_nova_api_os_compute_container-8e58a270] => (item={u'src': u'/root/.ssh/rpc_support', u'name': u'rpc_support', u'file_mode': u'0600', u'dir_mode': u'0755'})
skipping: [a_memcached_container-cea314a3] => (item={u'src': u'/root/.ssh/rpc_support', u'name': u'rpc_support', u'file_mode': u'0600', u'dir_mode': u'0755'})
failed: [a] (item={u'src': u'/root/.ssh/rpc_support', u'name': u'rpc_support', u'file_mode': u'0600', u'dir_mode': u'0755'}) => {"err": "Memcache key not found", "failed": true, "item": {"dir_mode": "0755", "file_mode": "0600", "name": "rpc_support", "src": "/root/.ssh/rpc_support"}, "msg": "The key you specified was not found within memcached. If you are load balancing memcached, attempt to connect to a single node.", "rc": 1}
failed: [a_neutron_agents_container-05e0fe11] (item={u'src': u'/root/.ssh/rpc_support', u'name': u'rpc_support', u'file_mode': u'0600', u'dir_mode': u'0755'}) => {"err": "Memcache key not found", "failed": true, "item": {"dir_mode": "0755", "file_mode": "0600", "name": "rpc_support", "src": "/root/.ssh/rpc_support"}, "msg": "The key you specified was not found within memcached. If you are load balancing memcached, attempt to connect to a single node.", "rc": 1}
failed: [a_neutron_server_container-236fcf3e] (item={u'src': u'/root/.ssh/rpc_support', u'name': u'rpc_support', u'file_mode': u'0600', u'dir_mode': u'0755'}) => {"err": "Memcache key not found", "failed": true, "item": {"dir_mode": "0755", "file_mode": "0600", "name": "rpc_support", "src": "/root/.ssh/rpc_support"}, "msg": "The key you specified was not found within memcached. If you are load balancing memcached, attempt to connect to a single node.", "rc": 1}
skipping: [a_nova_api_os_compute_container-8e58a270] => (item={u'src': u'/root/.ssh/rpc_support.pub', u'name': u'rpc_support_pub', u'file_mode': u'0640', u'dir_mode': u'0755'})

or another run:

TASK [rcbops_openstack-ops : Retrieve authorized keys] *************************
skipping: [a_cinder_scheduler_container-45b88d5e] => (item={u'src': u'/root/.ssh/rpc_support', u'name': u'rpc_support', u'file_mode': u'0600', u'dir_mode': u'0755'})
skipping: [a_swift_proxy_container-c0cb8b09] => (item={u'src': u'/root/.ssh/rpc_support', u'name': u'rpc_support', u'file_mode': u'0600', u'dir_mode': u'0755'})
skipping: [a_cinder_scheduler_container-45b88d5e] => (item={u'src': u'/root/.ssh/rpc_support.pub', u'name': u'rpc_support_pub', u'file_mode': u'0640', u'dir_mode': u'0755'})
skipping: [a_swift_proxy_container-c0cb8b09] => (item={u'src': u'/root/.ssh/rpc_support.pub', u'name': u'rpc_support_pub', u'file_mode': u'0640', u'dir_mode': u'0755'})
skipping: [a_nova_api_os_compute_container-8e58a270] => (item={u'src': u'/root/.ssh/rpc_support', u'name': u'rpc_support', u'file_mode': u'0600', u'dir_mode': u'0755'})
skipping: [a_memcached_container-cea314a3] => (item={u'src': u'/root/.ssh/rpc_support', u'name': u'rpc_support', u'file_mode': u'0600', u'dir_mode': u'0755'})
failed: [a] (item={u'src': u'/root/.ssh/rpc_support', u'name': u'rpc_support', u'file_mode': u'0600', u'dir_mode': u'0755'}) => {"err": "Memcache key not found", "failed": true, "item": {"dir_mode": "0755", "file_mode": "0600", "name": "rpc_support", "src": "/root/.ssh/rpc_support"}, "msg": "The key you specified was not found within memcached. If you are load balancing memcached, attempt to connect to a single node.", "rc": 1}
failed: [a_neutron_agents_container-05e0fe11] (item={u'src': u'/root/.ssh/rpc_support', u'name': u'rpc_support', u'file_mode': u'0600', u'dir_mode': u'0755'}) => {"err": "Memcache key not found", "failed": true, "item": {"dir_mode": "0755", "file_mode": "0600", "name": "rpc_support", "src": "/root/.ssh/rpc_support"}, "msg": "The key you specified was not found within memcached. If you are load balancing memcached, attempt to connect to a single node.", "rc": 1}
failed: [a_neutron_server_container-236fcf3e] (item={u'src': u'/root/.ssh/rpc_support', u'name': u'rpc_support', u'file_mode': u'0600', u'dir_mode': u'0755'}) => {"err": "Memcache key not found", "failed": true, "item": {"dir_mode": "0755", "file_mode": "0600", "name": "rpc_support", "src": "/root/.ssh/rpc_support"}, "msg": "The key you specified was not found within memcached. If you are load balancing memcached, attempt to connect to a single node.", "rc": 1}
skipping: [a_nova_api_os_compute_container-8e58a270] => (item={u'src': u'/root/.ssh/rpc_support.pub', u'name': u'rpc_support_pub', u'file_mode': u'0640', u'dir_mode': u'0755'})
skipping: [a_memcached_container-cea314a3] => (item={u'src': u'/root/.ssh/rpc_support.pub', u'name': u'rpc_support_pub', u'file_mode': u'0640', u'dir_mode': u'0755'})
skipping: [a_galera_container-ab544444] => (item={u'src': u'/root/.ssh/rpc_support', u'name': u'rpc_support', u'file_mode': u'0600', u'dir_mode': u'0755'})
skipping: [a_galera_container-9b94acb9] => (item={u'src': u'/root/.ssh/rpc_support', u'name': u'rpc_support', u'file_mode': u'0600', u'dir_mode': u'0755'})
failed: [a] (item={u'src': u'/root/.ssh/rpc_support.pub', u'name': u'rpc_support_pub', u'file_mode': u'0640', u'dir_mode': u'0755'}) => {"err": "Memcache key not found", "failed": true, "item": {"dir_mode": "0755", "file_mode": "0640", "name": "rpc_support_pub", "src": "/root/.ssh/rpc_support.pub"}, "msg": "The key you specified was not found within memcached. If you are load balancing memcached, attempt to connect to a single node.", "rc": 1}
failed: [a_neutron_agents_container-05e0fe11] (item={u'src': u'/root/.ssh/rpc_support.pub', u'name': u'rpc_support_pub', u'file_mode': u'0640', u'dir_mode': u'0755'}) => {"err": "Memcache key not found", "failed": true, "item": {"dir_mode": "0755", "file_mode": "0640", "name": "rpc_support_pub", "src": "/root/.ssh/rpc_support.pub"}, "msg": "The key you specified was not found within memcached. If you are load balancing memcached, attempt to connect to a single node.", "rc": 1}
failed: [a_neutron_server_container-236fcf3e] (item={u'src': u'/root/.ssh/rpc_support.pub', u'name': u'rpc_support_pub', u'file_mode': u'0640', u'dir_mode': u'0755'}) => {"err": "Memcache key not found", "failed": true, "item": {"dir_mode": "0755", "file_mode": "0640", "name": "rpc_support_pub", "src": "/root/.ssh/rpc_support.pub"}, "msg": "The key you specified was not found within memcached. If you are load balancing memcached, attempt to connect to a single node.", "rc": 1}

As you can see, the role doesn't cater for these issues.

I'd suggest:

• for the rpc_support keypairs, to work in a more idempotent way, (register and if need be, retries)
• for 'Retrieve authorized keys', that we do the targetting to the appropriate node we know works, or restructure the play so it always works.

Thank you in advance.

[evrardjp]

Oh, forgot to mention the sha: de006ade2f88790c43ec06a104abc895d98fe9b6

[BjoernT]

Thats based off master? I wanted to get rid of memcache there anyway so let me look at this

[evrardjp]

It's probably newton-14.1. But I think it's worth doing in master and backporting :)

Don't hesitate to use:

- block:
  rescue:
  always:

[evrardjp]

Do you have any update?

[evrardjp]

@BjoernT I will go on holiday today, could you please update either @git-harry @cloudnull or @weezer of any changes happening? We are using this as an indicator of "RPC leapfrog success"

[BjoernT]

@evrardjp I started working on it and I will be back in the office on Monday. I'm not sure what you mean with "We are using this as an indicator of "RPC leapfrog success"" ?

[git-harry]

@BjoernT how is you progress with this bug? How long do you think it will be before it is fixed?

[BjoernT]

I'm mostly done but have trouble even reproducing this problem. Do you know why the nova keypair-add failed ? How was that environment been built, aio with artifacts enabled or a upgrade from an older version to an artifact build ?

[git-harry]

@BjoernT I'm not familiar with the specifics of how @evrardjp has been working. He is in the process of getting Kilo to Newton leapfrog upgrades working but I do not know how his environment was set up.

I've taken a look at the tasks that are failing and my guess would be the following:

• I suspect this is failing when the upgrade is being performed.
• This command should be nova keypair-delete rpc_support.
• Although the task Delete rpc_support keypair in nova is failing, Ansible is reporting it as having changed.
• Add rpc_support key to nova attempts to add the new key because it believes the task Delete rpc_support keypair in nova made a change and then fails because the old one is still there.

[BjoernT]

The issue was simply a typo at https://github.com/rsoprivatecloud/openstack-ops/blob/master/tasks/support_key_pair.yml#L37 but I rewrote the entire thing via https://patch-diff.githubusercontent.com/raw/rsoprivatecloud/openstack-ops/pull/34.patch which you're welcome to checkout. What I did is inside the pull description. It's a lot more idempotent now and does not use memcache anymore

[evrardjp]

I will have a look. It sounds better.

[BjoernT]

OK I merged the changes into the newton 14.1 branch and will test it once more.

[BjoernT]

Closing it out now. Reopen if you find anything else. Thanks