microceph returns only one MON address to clients

[nobuto-m]

ceph-public-address in the relation data only has one IP address. https://bugs.launchpad.net/charm-microceph/+bug/2066167

$ sudo microceph status
MicroCeph deployment summary:
- sunbeam-1 (10.0.123.11)
  Services: mds, mgr, mon, osd
  Disks: 2
- sunbeam-2 (10.0.123.12)
  Services: mds, mgr, mon, osd
  Disks: 2
- sunbeam-3 (10.0.123.13)
  Services: mds, mgr, mon, osd
  Disks: 2

$ juju show-unit -m openstack glance/2

...

  - relation-id: 50
    endpoint: ceph
    cross-model: true
    related-endpoint: ceph
    application-data: {}
    related-units:
      microceph/0:
        in-scope: true
        data:
          auth: cephx
          broker-rsp-glance-0: '{"exit-code": 0, "request-id": "cc6dca75e3585dfba9af7293ce83810bffaa1a97"}'
          broker-rsp-glance-1: '{"exit-code": 0, "request-id": "cc6dca75e3585dfba9af7293ce83810bffaa1a97"}'
          broker-rsp-glance-2: '{"exit-code": 0, "request-id": "cc6dca75e3585dfba9af7293ce83810bffaa1a97"}'
          ceph-public-address: 10.0.123.11
          egress-subnets: 10.0.123.11/32
          ingress-address: 10.0.123.11
          key: AQDhGEtme4xqIBAAUoDECPLPeod728vTLZNvAQ==
          private-address: 10.0.123.11
      microceph/1:
        in-scope: true
        data:
          egress-subnets: 10.0.123.12/32
          ingress-address: 10.0.123.12
          private-address: 10.0.123.12
      microceph/2:
        in-scope: true
        data:
          egress-subnets: 10.0.123.13/32
          ingress-address: 10.0.123.13
          private-address: 10.0.123.13

          broker-rsp-glance-0: '{"exit-code": 0, "request-id": "cc6dca75e3585dfba9af7293ce83810bffaa1a97"}'
          broker-rsp-glance-1: '{"exit-code": 0, "request-id": "cc6dca75e3585dfba9af7293ce83810bffaa1a97"}'
          broker-rsp-glance-2: '{"exit-code": 0, "request-id": "cc6dca75e3585dfba9af7293ce83810bffaa1a97"}'
          ceph-public-address: 10.0.123.11
          egress-subnets: 10.0.123.11/32
          ingress-address: 10.0.123.11
          key: AQDhGEtme4xqIBAAUoDECPLPeod728vTLZNvAQ==
          private-address: 10.0.123.11
      microceph/1:
        in-scope: true
        data:
          egress-subnets: 10.0.123.12/32
          ingress-address: 10.0.123.12
          private-address: 10.0.123.12
      microceph/2:
        in-scope: true
        data:
          egress-subnets: 10.0.123.13/32
          ingress-address: 10.0.123.13
          private-address: 10.0.123.13

[UtkarshBhatthere]

Thanks for reporting this @nobuto-m, taking a look

[lmlg]

This is also the case in ceph-mon: https://opendev.org/openstack/charm-ceph-mon/src/branch/master/src/ceph_client.py#L80 It's not necessarily a problem. Depending on what the client uses to connect to the cluster (i.e: librados), it can get the monmap by simply using a single monitor address, its username and key.

[nobuto-m]

It's important to agree on what to expect both from the consumer side of the interface and the provider. As long as the consumer has an established method to list (at least) 3 MON endpoints for HA, we are good. https://bugs.launchpad.net/snap-openstack/+bug/2066167

[hemanthnakkina]

@lmlg What happens when mon is down for some reason? Does charm-microceph detects and changes the relation data to point to another mon?

[UtkarshBhatthere]

The microceph concern is that changing the behaviour of the config key might make it incompatible with pre-sunbeam clients which only expect a single address to be provided ?

In this regard, may I request you to use a new key ceph-public-addresses (plural) which may be used to consume all the mon addrs at once.

[hemanthnakkina]

Fix for review https://github.com/canonical/charm-microceph/pull/97