Empty metrics from controllers running juju 2.8.x and older

[przemeklal]

The application doesn't export any machines when running against juju 2.8.x and older.

[przemeklal]

Update: not all 2.8.x versions are affected, but I managed to reproduce the issue on juju 2.8.6:

ubuntu@przemeklal-bastion:~$ juju status -m controller
Model       Controller                     Cloud/Region             Version  SLA          Timestamp  Notes
controller  serverstack-serverstack-2.8.6  serverstack/serverstack  2.8.6    unsupported  10:57:20Z  upgrade available: 2.8.13

Machine  State    DNS        Inst id                               Series  AZ    Message
0        started  10.5.3.15  592b6cd5-919a-402e-9ffa-77bc3d9efa57  bionic  nova  ACTIVE

ubuntu@przemeklal-bastion:~$ curl localhost:5000/metrics
# HELP juju_machine_state Running status of juju machines
# TYPE juju_machine_state gauge

The same works fine on 2.8.13:

ubuntu@przemeklal-bastion:~$ juju status -m serverstack-serverstack:admin/controller
Model       Controller               Cloud/Region             Version  SLA          Timestamp
controller  serverstack-serverstack  serverstack/serverstack  2.8.13   unsupported  10:58:22Z

Machine  State    DNS        Inst id                               Series  AZ    Message
0        started  10.5.1.55  2a328634-4591-4b38-8960-86158265646d  bionic  nova  ACTIVE

ubuntu@przemeklal-bastion:~$ curl localhost:5000/metrics                                                                                                                                                                            [18/18]
# HELP juju_machine_state Running status of juju machines                                                            
# TYPE juju_machine_state gauge                       
juju_machine_state{cloud_name="serverstack",customer="przemeklal",hostname="juju-363f44-controller-0",job="prometheus-juju-exporter",juju_model="controller",type="kvm"} 1.0

Running with --debug logs enabled I don't see any issues or errors when running against 2.8.6.

[mert-kirpici]

This is due to the hostname information being only available in the Machine property when working against juju controllers version 2.8.10+. See the pull request and related issue: https://github.com/juju/python-libjuju/pull/475

Verified that when running against 2.8.6 controller, the hostname property returns None as documented.

[mert-kirpici]

Verified the last working version is 2.8.10. Tested both versions and in 2.8.9 and we are returning empty metrics due to the same issue.

[agileshaw]

@mert-kirpici This seems to be the same issue as mentioned in https://github.com/canonical/prometheus-juju-exporter/issues/24. I would suggest to keep checking hostname as default and fall back to checking instance-id when hostname is empty (for better readbility in metrics labels)

[mert-kirpici]

@agileshaw I saw your comment and was not sure if it is the same issue since I couldn't see the juju bundle or the juju status, I did not know what the controller version was in that case. Also some of the machines having it and some not made me think there might be a different underlying issue with that one, although I agree with you it sounds VERY similar.

In all the testing I did, I deployed multiple ubuntu units and none of the machines in the workload model and the controller model contained hostname information returning from libjuju.

[agileshaw]

@mert-kirpici I think they are the same issue because that model was migrated from a old controller to the new one. My assumption is, during the migration, there are some fields that were not populated correctly. I will share the juju status output with you privately.