search

canonical / cos-proxy-operator

https://charmhub.io/cos-proxy
Apache License 2.0
2 stars 12 forks source link

Errors when relating with ceph-dashboard #7

Closed mmanciop closed 2 years ago

mmanciop commented 2 years ago

Rather vanilla setup imaginable with Ceph running on LXD:

root@dell:~# juju status --relations
Model  Controller  Cloud/Region  Version  SLA          Timestamp
ceph   controller  maas1         2.9.14   unsupported  02:54:05-06:00

SAAS        Status  Store  URL
grafana     active  cos    admin/cos.grafana-dashboards
prometheus  active  cos    admin/cos.prometheus-scrape

App             Version  Status   Scale  Charm           Store       Channel  Rev  OS      Message
ceph-dashboard           blocked      3  ceph-dashboard  charmstore  stable     3  ubuntu  Charm config option grafana-api-url not set
ceph-fs         15.2.14  active       1  ceph-fs         charmhub    stable    36  ubuntu  Unit is ready
ceph-mon        15.2.14  active       3  ceph-mon        charmstore  stable    62  ubuntu  Unit is ready and clustered
ceph-osd        15.2.14  active       5  ceph-osd        charmstore  stable   316  ubuntu  Unit is ready (1 OSD)
ceph-radosgw    15.2.14  active       1  ceph-radosgw    charmhub    stable   499  ubuntu  Unit is ready
cos-proxy                error        1  cos-proxy       charmhub    edge       1  ubuntu  hook failed: "dashboards-relation-changed"
easyrsa         3.0.1    active       1  easyrsa         charmstore  stable   441  ubuntu  Certificate Authority connected.

Unit                 Workload  Agent  Machine  Public address  Ports   Message
ceph-fs/0*           active    idle   2/lxd/1  192.168.122.66          Unit is ready
ceph-mon/0*          active    idle   0/lxd/0  192.168.122.15          Unit is ready and clustered
  ceph-dashboard/0*  blocked   idle            192.168.122.15          Charm config option grafana-api-url not set
ceph-mon/1           active    idle   1/lxd/0  192.168.122.12          Unit is ready and clustered
  ceph-dashboard/2   blocked   idle            192.168.122.12          Charm config option grafana-api-url not set
ceph-mon/2           active    idle   2/lxd/0  192.168.122.27          Unit is ready and clustered
  ceph-dashboard/1   blocked   idle            192.168.122.27          Dashboard is not enabled
ceph-osd/0           active    idle   0        192.168.122.40          Unit is ready (1 OSD)
ceph-osd/1*          active    idle   1        192.168.122.39          Unit is ready (1 OSD)
ceph-osd/2           active    idle   2        192.168.122.9           Unit is ready (1 OSD)
ceph-osd/4           active    idle   4        192.168.122.75          Unit is ready (1 OSD)
ceph-osd/5           active    idle   5        192.168.122.70          Unit is ready (1 OSD)
ceph-radosgw/0*      active    idle   1/lxd/1  192.168.122.80  80/tcp  Unit is ready
cos-proxy/0*         error     idle   6        192.168.122.91          hook failed: "dashboards-relation-changed" for ceph-dashboard:grafana-dashboard
easyrsa/1*           active    idle   0/lxd/2  192.168.122.25          Certificate Authority connected.

Machine  State    DNS             Inst id              Series  AZ       Message
0        started  192.168.122.40  o7k1                 focal   default  Deployed
0/lxd/0  started  192.168.122.15  juju-8022af-0-lxd-0  focal   default  Container started
0/lxd/2  started  192.168.122.25  juju-8022af-0-lxd-2  focal   default  Container started
1        started  192.168.122.39  o7k2                 focal   default  Deployed
1/lxd/0  started  192.168.122.12  juju-8022af-1-lxd-0  focal   default  Container started
1/lxd/1  started  192.168.122.80  juju-8022af-1-lxd-1  focal   default  Container started
2        started  192.168.122.9   o7k3                 focal   default  Deployed
2/lxd/0  started  192.168.122.27  juju-8022af-2-lxd-0  focal   default  Container started
2/lxd/1  started  192.168.122.66  juju-8022af-2-lxd-1  focal   default  Container started
4        started  192.168.122.75  o7k4                 focal   default  Deployed
5        started  192.168.122.70  o7k5                 focal   default  Deployed
6        started  192.168.122.91  o7k6                 focal   default  Deployed

Relation provider                       Requirer                     Interface          Type         Message
ceph-dashboard:grafana-dashboard        cos-proxy:dashboards         grafana-dashboard  regular      
ceph-mon:dashboard                      ceph-dashboard:dashboard     ceph-dashboard     subordinate  
ceph-mon:mds                            ceph-fs:ceph-mds             ceph-mds           regular      
ceph-mon:mon                            ceph-mon:mon                 ceph               peer         
ceph-mon:osd                            ceph-osd:mon                 ceph-osd           regular      
ceph-mon:prometheus                     cos-proxy:prometheus-target  http               regular      
ceph-mon:radosgw                        ceph-radosgw:mon             ceph-radosgw       regular      
ceph-radosgw:cluster                    ceph-radosgw:cluster         swift-ha           peer         
ceph-radosgw:gateway                    cos-proxy:prometheus-target  http               regular      
cos-proxy:downstream-grafana-dashboard  grafana:grafana-dashboard    grafana_dashboard  regular      
cos-proxy:downstream-prometheus-scrape  prometheus:metrics-endpoint  prometheus_scrape  regular      
easyrsa:client                          ceph-dashboard:certificates  tls-certificates   regular

Error trace as follows:

unit-cos-proxy-0: 02:51:41 ERROR unit.cos-proxy/0.juju-log dashboards:11: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 260, in <module>
    main(COSProxyCharm)
  File "/var/lib/juju/agents/unit-cos-proxy-0/charm/venv/ops/main.py", line 426, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-cos-proxy-0/charm/venv/ops/main.py", line 142, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-cos-proxy-0/charm/venv/ops/framework.py", line 276, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-cos-proxy-0/charm/venv/ops/framework.py", line 736, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-cos-proxy-0/charm/venv/ops/framework.py", line 783, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-cos-proxy-0/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 1029, in update_dashboards
    self._upset_dashboards_on_event(event)
  File "/var/lib/juju/agents/unit-cos-proxy-0/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 1033, in _upset_dashboards_on_event
    dashboards = self._handle_reactive_dashboards(event)
  File "/var/lib/juju/agents/unit-cos-proxy-0/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 1150, in _handle_reactive_dashboards
    t = self._strip_existing_datasources(t)
  File "/var/lib/juju/agents/unit-cos-proxy-0/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 1096, in _strip_existing_datasources
    dash = template["dashboard"]
KeyError: 'dashboard'
unit-cos-proxy-0: 02:51:41 ERROR juju.worker.uniter.operation hook "dashboards-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1
dperez-sct commented 2 years ago

The same happens here.


Traceback (most recent call last):
  File "./src/charm.py", line 260, in <module>
    main(COSProxyCharm)
  File "/var/lib/juju/agents/unit-cos-proxy-10/charm/venv/ops/main.py", line 426, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-cos-proxy-10/charm/venv/ops/main.py", line 142, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-cos-proxy-10/charm/venv/ops/framework.py", line 276, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-cos-proxy-10/charm/venv/ops/framework.py", line 736, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-cos-proxy-10/charm/venv/ops/framework.py", line 783, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-cos-proxy-10/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 1028, in update_dashboards
    self._upset_dashboards_on_event(event)
  File "/var/lib/juju/agents/unit-cos-proxy-10/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 1032, in _upset_dashboards_on_event
    dashboards = self._handle_reactive_dashboards(event)
  File "/var/lib/juju/agents/unit-cos-proxy-10/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 1149, in _handle_reactive_dashboards
    t = self._strip_existing_datasources(t)
  File "/var/lib/juju/agents/unit-cos-proxy-10/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 1095, in _strip_existing_datasources
    dash = template["dashboard"]
KeyError: 'dashboard'
unit-cos-proxy-10: [13:52:36](https://www.youtube.com/watch?v=iQ99jYPQao0&t=43200s) ERROR juju.worker.uniter.operation hook "dashboards-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1```
simskij commented 2 years ago

We will never be able to solve this as the dashboard is not transmitted over the relation, but rather just a "enable this" command.

The proper solution here would be for the ceph-dashboard charm to transmit their dashboard, just as all other charms do.