Microceph fails to retrieve default gateway when network setup is not straigthforward

[gboutry]

Description

During a ceph-client relation, Microceph will try to retrieve an IP address associated with the default gateway so that clients can access Ceph services.

Function: https://github.com/canonical/charm-microceph/blob/f47b8c69763db42648c7dc87d6134a306ebde144/src/charm.py#L49

In some wonky network setups, this command fails.

Failure example

Here's a reproduction on a setup that fails:

>>> import netifaces
>>> from netifaces import AF_INET, gateways, ifaddresses
>>> def _get_local_ip_by_default_route() -> str:
...     """Get IP address of host associated with default gateway."""
...     interface = "lo"
...     ip = "127.0.0.1"
...     # TOCHK: Gathering only IPv4
...     if "default" in gateways():
...         interface = gateways()["default"][AF_INET][1]
...     ip_list = ifaddresses(interface)[AF_INET]
...     if len(ip_list) > 0 and "addr" in ip_list[0]:
...         ip = ip_list[0]["addr"]
...     return ip
... 
>>> ip = _get_local_ip_by_default_route()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 7, in _get_local_ip_by_default_route
KeyError: 2

Additional notes

If running the gateways(), you can see that default returns an empty dict:

>>> gateways()
{'default': {}, 2: [('10.99.0.1', 'eth2', False), ('10.150.19.1', 'eth1', False)]}

Retrieving a working IP Address is still possible

(this might not be the way Microceph want to handle this, just an example)

Here's an example with NDB, showing that it's still possible to get the default route (and infer interface):

>>> import pyroute2
>>> import socket
>>> with pyroute2.NDB() as ndb:
...     default_route_ifindex = ndb.routes["default"]["oif"]
...     iface = ndb.interfaces[default_route_ifindex]
...     ipaddr = iface.ipaddr[socket.AF_INET]["address"]
... 
>>> print(ipaddr)
10.150.19.180