bionic-queens - ceph-mon, ceph-osd, and designate units fail to start

[mgaruccio]

When deploying the bionic-queens bundle ceph-mon and ceph-osd fail to start and show a juju status of hook failed: install. This seems to be due to attempting to pull a xenial source on a bionic OS. I tried changing the source to cloud:bionic-queens but that started generating an error that the source did not exist. Ultimately removing the option entirely fixed the problem.

Designate also failed to start. After further research I found that this is due to designate requiring a nameservers option in queens and above. Adding the option using the example text provided by the designate charm allows the deployment to continue. It's not 100% clear to me if this is asking for addresses of existing nameservers or if it names the servers that are created using those names but it at least allows the rest of the deployment to function.

PR for both of these changes incoming

[glzavert]

I too am getting this error on the latest pull today. The "bundle-bionic-queens.yaml" clearly references a "cloud:xenial-queens" source. Is there a "cloud:bionic-queens" source anywhere? How do we get around this? Do I remove the source option all together? Is anyone going to fix this or respond with a solution or workaround? A prompt acknowledgement of the issue and assignment to someone to work on this would be greatly appreciated.

[mgaruccio]

@glzavert removing the source option entirely worked for me. See PR #46 for the changes.

[glzavert]

@mgaruccio , Still cannot get the stack to run on a clean install of bionic. I removed the "source" option for ceph and added the nameserver option for designate but I used googles dns ip addresses in the "bundle-bionic-queens.yaml"

designate: charm: cs:designate num_units: 1 nameservers: "8.8.8.8 8.8.8.4"

juju status shows:

Model Controller Cloud/Region Version SLA default lxd localhost/localhost 2.3.8 unsupported

App Version Status Scale Charm Store Rev OS Notes ceilometer 10.0.0 waiting 1 ceilometer jujucharms 252 ubuntu ceilometer-agent 10.0.0 active 2 ceilometer-agent jujucharms 243 ubuntu ceph-mon 12.2.4 active 3 ceph-mon jujucharms 24 ubuntu ceph-osd 12.2.4 error 3 ceph-osd jujucharms 261 ubuntu ceph-radosgw 12.2.4 active 1 ceph-radosgw jujucharms 257 ubuntu cinder 12.0.1 active 1 cinder jujucharms 271 ubuntu cinder-ceph 12.0.1 active 1 cinder-ceph jujucharms 232 ubuntu designate 6.0.1 blocked 1 designate jujucharms 18 ubuntu designate-bind 9.11.3+dfsg active 1 designate-bind jujucharms 12 ubuntu glance 16.0.1 active 1 glance jujucharms 264 ubuntu gnocchi 4.2.4 error 1 gnocchi jujucharms 7 ubuntu heat 10.0.0 active 1 heat jujucharms 251 ubuntu keystone 13.0.0 active 1 keystone jujucharms 280 ubuntu memcached unknown 1 memcached jujucharms 21 ubuntu mysql 5.7.20-29.24 active 1 percona-cluster jujucharms 264 ubuntu neutron-api 12.0.1 active 1 neutron-api jujucharms 259 ubuntu neutron-gateway 12.0.1 active 1 neutron-gateway jujucharms 251 ubuntu neutron-openvswitch 12.0.1 active 2 neutron-openvswitch jujucharms 249 ubuntu nova-cloud-controller 17.0.3 active 1 nova-cloud-controller jujucharms 309 ubuntu nova-compute 17.0.3 active 2 nova-compute jujucharms 282 ubuntu openstack-dashboard 13.0.0 active 1 openstack-dashboard jujucharms 258 ubuntu rabbitmq-server 3.6.10 active 1 rabbitmq-server jujucharms 73 ubuntu

Unit Workload Agent Machine Public address Ports Message ceilometer/0 waiting idle 0 10.76.180.193 Incomplete relations: database ceph-mon/0 active idle 1 10.76.180.53 Unit is ready and clustered ceph-mon/1 active idle 2 10.76.180.15 Unit is ready and clustered ceph-mon/2 active idle 3 10.76.180.78 Unit is ready and clustered ceph-osd/0 error idle 4 10.76.180.161 hook failed: "mon-relation-changed" ceph-osd/1 error idle 5 10.76.180.131 hook failed: "mon-relation-changed" ceph-osd/2 error idle 6 10.76.180.94 hook failed: "mon-relation-changed" ceph-radosgw/0 active idle 7 10.76.180.49 80/tcp Unit is ready cinder/0 active idle 8 10.76.180.240 8776/tcp Unit is ready cinder-ceph/0 active idle 10.76.180.240 Unit is ready designate-bind/0* active idle 10 10.76.180.231 Unit is ready designate/0* blocked idle 9 10.76.180.65 9001/tcp nameservers must be set* glance/0 active idle 11 10.76.180.42 9292/tcp Unit is ready gnocchi/0* error idle 12 10.76.180.179 8041/tcp hook failed: "identity-service-relation-changed" heat/0 active idle 13 10.76.180.93 8000/tcp,8004/tcp Unit is ready keystone/0 active idle 14 10.76.180.164 5000/tcp Unit is ready memcached/0* unknown idle 15 10.76.180.87 11211/tcp* mysql/0 active idle 16 10.76.180.74 3306/tcp Unit is ready neutron-api/0 active idle 17 10.76.180.174 9696/tcp Unit is ready neutron-gateway/0 active idle 18 10.76.180.13 Unit is ready nova-cloud-controller/0 active idle 19 10.76.180.181 8774/tcp,8778/tcp Unit is ready nova-compute/0 active idle 20 10.76.180.150 Unit is ready ceilometer-agent/0 active idle 10.76.180.150 Unit is ready neutron-openvswitch/0 active idle 10.76.180.150 Unit is ready nova-compute/1 active idle 21 10.76.180.130 Unit is ready ceilometer-agent/1 active idle 10.76.180.130 Unit is ready neutron-openvswitch/1 active idle 10.76.180.130 Unit is ready openstack-dashboard/0 active idle 22 10.76.180.136 80/tcp,443/tcp Unit is ready rabbitmq-server/0 active idle 23 10.76.180.17 5672/tcp Unit is ready

Machine State DNS Inst id Series AZ Message 0 started 10.76.180.193 juju-ac3087-0 bionic Running 1 started 10.76.180.53 juju-ac3087-1 bionic Running 2 started 10.76.180.15 juju-ac3087-2 bionic Running 3 started 10.76.180.78 juju-ac3087-3 bionic Running 4 started 10.76.180.161 juju-ac3087-4 bionic Running 5 started 10.76.180.131 juju-ac3087-5 bionic Running 6 started 10.76.180.94 juju-ac3087-6 bionic Running 7 started 10.76.180.49 juju-ac3087-7 bionic Running 8 started 10.76.180.240 juju-ac3087-8 bionic Running 9 started 10.76.180.65 juju-ac3087-9 bionic Running 10 started 10.76.180.231 juju-ac3087-10 bionic Running 11 started 10.76.180.42 juju-ac3087-11 bionic Running 12 started 10.76.180.179 juju-ac3087-12 bionic Running 13 started 10.76.180.93 juju-ac3087-13 bionic Running 14 started 10.76.180.164 juju-ac3087-14 bionic Running 15 started 10.76.180.87 juju-ac3087-15 bionic Running 16 started 10.76.180.74 juju-ac3087-16 bionic Running 17 started 10.76.180.174 juju-ac3087-17 bionic Running 18 started 10.76.180.13 juju-ac3087-18 bionic Running 19 started 10.76.180.181 juju-ac3087-19 bionic Running 20 started 10.76.180.150 juju-ac3087-20 bionic Running 21 started 10.76.180.130 juju-ac3087-21 bionic Running 22 started 10.76.180.136 juju-ac3087-22 bionic Running 23 started 10.76.180.17 juju-ac3087-23 bionic Running

At this point I do not know what to do. I figure the first thing is to get ceph working. Then I need to change the ceph config to act more like RAID and eliminate replication via erasure coding. This is all in a VM and I need to maximize storage availability rather than resiliency as I have that via VM snapshots.

After Ceph, I guess I need to figure out how to get designate to work. I need an internal VM DNS for the images running in openstack but I really don't need external DNS resolution other than what I get by setting the nameservers on the host to google.

Also, I have not even begun to figure out what is up with gnocchi or why memcached is reporting unknown.

I followed the instructions here pretty closely and didn't deviate other than what I said above. IT is frustrating try to get an environment working so I can get on to learning more about the admin and use of openstack instead of being in the perpetual install and reinstall phase for months.

I was able to get the "conjure-up" install of openstack to work but it is a bare-bones cluster and is missing some of the things I need most, like heat and ceilometer and the conjure-up install doesn't let you add those in to a base install.

Any help would be appreciated. I also attached the "unit-ceph-osd-0.log" file.

unit-ceph-osd-0.log

[mgaruccio]

@glzavert for ceph did you remove the source option from both ceph-mon and ceph-osd?

as for designate it looks like you put the nameservers option at the top level of the object rather than in options, which would prevent it from being honored. for the record it's not clear to me in the designate docs whether an authoritative public nameserver like 8.8.8.8 will work or if you need to provide a server you control that is delegating control of a zone, I can confirm that using a non-existent DNS name of "ns1.example.com" allows designate to configure itself but the DNS features of the cluster do not work.

if you just pull from my repo and run that install does it work or do you still see designate/ceph errors?

[glzavert]

I did finally get the whole stack working.

I did remove the source from the yaml file and I also used the "testing" PPA for queens. The issue with ceph is that even though BTRFS is supposed to be the preferred storage backend, it is not supported in this build variant. So I rebuilt and went back to ZFS and ceph is good now.

As for designate, from what I gather, the nameserver is an option and is used as the FQDN of the primary and secondary nameserver it created for it's own internal use. I used a different FQDN that coincided with my FQDN for the host server, i.e ns1.{host.fqdn} and ns2.{host.fqdn}. Which explains the behavior seen by me and in your build where you used ns1.example.com and ns1.example.com.

[mgaruccio]

Ok just wanted to make sure you removed both of the source lines, not sure about BTFRS as I was using ZFS as the storage backend. Are things currently working for you or are things still failing?

[glzavert]

All is working. Now to reconfigure ceph to use erasure-coding.

With all of this, I have one heck of a build script so far.