Improve conversion from D-Bus uids to MQTT uids

[blammit]

We need to improve the approaches for converting D-Bus uids to make it more robust.

For example, one of the current approaches looks at the '/ServiceName' for services that report them. This works for conversions like the following:

dbus/com.victronenergy.dcsystem.ttyS5 -> mqtt/dcsystem/40/Devices/0 dbus/com.victronenergy.solarcharger.ttyO0 -> mqtt/solarcharger/256/Devices/0

But many services don't have a /ServiceName, and also a service may use this to define an associated service (rather than to refer to its own serivce) and so in this case the uid is incorrectly converted. For example, dbus/com.victronenergy.vebus.ttyO1/Dc/0/Power is converted to mqtt/system/0/Ac/In/0/Dc/0/Power which does not exist; instead, it should be mqtt/vebus/257/Dc/0/Power.

For services that do not report a ServiceName, we assume it has a zero device instance, and convert it to the form mqtt/service/0. For example:

dbus/com.victronenergy.settings/Settings -> mqtt/settings/0/Settings dbus/com.victronenergy.system/Serial -> mqtt/system/0/Serial

The works for core system services (e.g. settings, system, platform), but it doesn't work for device-type services, as they have non-zero device instances. For example, dbus/com.victronenergy.temperature.adc_builtin0_5 is converted to mqtt/temperature/0 when it should be mqtt/temperature/22 if the device instance is 22.

[blammit]

To start with, we could look up the /DeviceInstance for all services and use that to get the mqtt/service/<DeviceInstance> part of the uid. But what about com.victronenergy.vecan, which does not have a /DeviceInstance? If there are both com.victronenergy.vecan.can0 and com.victronenergy.vecan.can1, will the MQTT uids be mqtt/vecan/0 and mqtt/vecan/1?

[blammit]

@jepefe do you have any suggestions for a foolproof method of converting a D-Bus uid to its MQTT form?

It would be great if there was some service/tool that provided a way to map between the two.

[jepefe]

@blammit

From D-Bus to MQTT you can use the service type (solar/battery/...). From MQTT / D-Bus I'm afraid that there is no way to convert it as part of the uuid is not known. You then need scan and map the services.

I found a relatively easy to follow example: https://github.com/victronenergy/dbus-flashmq/blob/55fa32a5ae1275682c1cc9e9499fd085b1528c93/src/state.cpp#L506

Also in Python: https://github.com/victronenergy/dbus-mqtt/blob/master/dbus_mqtt.py#L400

[blammit]

@jepefe For the D-Bus to MQTT case, we can easily convert 'dbus/com.victronenergy.system/some_path' to 'mqtt/system/0/some_path', but the problem is for services with non-zero device instances. For example, how should 'dbus/com.victronenergy.evcharger.xzy/some_path' be converted to 'mqtt/evcharger/256/some_path'? If there is only one EV charger then we can just use the first available mqtt/evcharger/, but if there are multiple EV chargers, then how should this be done? Or for multiple solar chargers, DC inputs, etc? Keeping in mind that when loading gui-v2 from the browser, we don't have dbus access so cannot scan for dbus values.

[jepefe]

It can be done the same way as other products by checking the /DeviceInstance. See the attached screenshot, in this case evcharger device instance is 40. dbus/com.victronenergy.evcharger.evc_HQ22317CXWN/some_path is converted to mqtt/evcharger/40/some_path

The combination of service type + device instance is unique per connected product.

[blammit]

@jepefe

What I'm wondering is, if there are multiple EV chargers, how should it be translated, when we all we have is the dbus UID and MQTT lookups available? For example:

• dbus/com.victronenergy.evcharger.evc_HQ22317CXWN/some_path -> mqtt/evcharger/40/some_path
• dbus/com.victronenergy.evcharger.evc_ABCDEFGHIJKL/some_path -> mqtt/evcharger/41/some_path

For evcharger services, the "HQ22317CXWN" is useful because we can look up mqtt/evcharger/40/SerialNumber and see if it matches the string "HQ22317CXWN". But for other services, the device serial number is not in the service name.

For solarcharger, we can look up /Devices/0/ServiceName as that matches the actual service names, e.g. com.victronenergy.solarcharger.ttyO0. Same with battery and dcsystem. But this doesn't work for temperature, tank or other services that do not have /Devices/0/ServiceName values.

Actually, something like the modbustcp service is ideal because it provides these values for every service:

• mqtt/modbustcp/0/Services/0/ServiceName = com.victronenergy.tank.adc_builtin0_3
• mqtt/modbustcp/0/Services/0/UnitId = 30 # this appears to be the DeviceInstance
• mqtt/modbustcp/0/Services/1/ServiceName = com.victronenergy.tank.adc_builtin0_2
• mqtt/modbustcp/0/Services/1/UnitId = 31 etc.

This allows us to iterate through the child values of mqtt/modbustcp/0/Services/* and thus map com.victronenergy.tank.adc_builtin0_2 to mqtt/tank/31. So if the modbustcp service was always available, we could just use that. But it isn't -- is there anyway to replicate that, or something like it?

The direct uid conversion is mainly required in the settings pages, where there are references to services like the following:

com.victronenergy.modem com.victronenergy.temprelay com.victronenergy.generator.startstop0 com.victronenergy.pump.startstop0 com.victronenergy.vecan.can0,can1 etc com.victronenergy.rvc.can0,can1 etc com.victronenergy.fronius com.victronenergy.ble com.victronenergy.logger com.victronenergy.platform com.victronenergy.modbustcp

And at the moment, we just assume that they are accessed with a device instance of zero, e.g. mqtt/modem/0, mqtt/ble/0. That seems like an assumption that will work sometimes and fail at other times.

Especially for services like generator and vecan that have additional suffixes like "startstop0" and "can0", we strip them and assume mqtt/generator/0 and mqtt/vecan/0. But what if there are multiple generator services at generator.startstop0 and generator.startstop1? Do we parse the service name for a digit and convert 'startstop1' -> mqtt/generator/1 and 'startstop0' -> mqtt/generator/0?

Does that make sense? It would be ideal there was common solution that worked for uids across all services -- evchargers, solarchargers, modem, ble, etc etc. Failing that, I guess we can do specialized handling case-by-case, looking up SerialNumber or ServiceName, but I don't see how that can be done for the modem, temprelay etc. cases listed above.

[blammit]

If it's not possible to have a straightforward uid conversion, maybe we can have an intermediate solution to support the cases where there is only one service per service type (e.g. only one vecan service). I'll investigate this so that we can at least fix #377, which is caused by the use of the "ServiceName" for uid conversion.

[blammit]

Looked into this further and discussed with Jesús. I think there is no generic solution to get convert uids where there are multiple services (e.g. com.victronenergy.xx.xx rather than com.victronenergy.xx). So, we will need to do specialized conversions for specific service types that are supported, and for other service types, we should not use DataPoint or SingleUidHelper.

To summarize:

• For uids of the form com.victronenergy.xx, this can be converted to mqtt/xx/0 (same as current behavior)

• For generator, pump, vecan and rvc service types, convert depending on the service name suffix. Examples:

  com.victronenergy.generator.startstop0 -> mqtt/generator/0 com.victronenergy.pump.startstop0 -> mqtt/pump/0 com.victronenergy.vecan.can0 -> mqtt/vecan/0 com.victronenergy.rvc.can0 -> mqtt/rvc/0

• For all other com.victronenergy.xx.xx uids: drop the uid conversion support, because there is no reliable way to do this across different service types. DataPoint should warn if a conversion is attempted for such uids. In most cases this should not be a problem, but there might be cases where we will need to use a VeItemTableModel to fetch all the services of a particular service type, in order to get the MQTT uids.

[blammit]

Actually, com.victronenergy.system/ServiceMapping provides a map like this:

com_victronenergy_settings_0 => com.victronenergy.settings com_victronenergy_generator_0 => com.victronenergy.generator.startstop0 com_victronenergy_battery_288 => com.victronenergy.battery.ttyUSB0

which is exactly what we need. However, it seems the values are not updated when services are added/removed.

Using the /ServiceMapping value would be more reliable than the simple string conversion I mentioned in the previous comment, especially as there are parts of the code that need to convert com.victronenergy.vebus.tty01 to its MQTT equivalent, and there is no reliable way of doing that if there is more than one vebus service.

[blammit]

Note that in many cases, it is not necessary to convert from a D-Bus uid to a MQTT uid. For example. when uids are loaded from a VeQItemTableModel or such when running with a MQTT backend, the uids will already be in MQTT form. However, there are cases where the conversion is required:

• In the settings pages, the uids have been ported verbatim from gui-v1, e.g. "com.victronenergy.settings", "com.victronenergy.generator.startstop0", "com.victronenergy.modbustcp"
• When accessing data related to service names, these names are always specified in the D-Bus format, even on a MQTT backend. E.g. when querying the value of com.victronenergy.system/Ac/In/0/ServiceName, the value is specified as e.g. "com.victronenergy.vebus.tty0" rather than "mqtt/vebus/289".