Support Open Vehicle Monitoring System (OVMS)

[amotl]

Dear @markwj and @dexterbg,

thanks for conceiving and maintaining OVMS. It appeared to me that starting with OVMS 3, the system sends out metrics over MQTT. It would be nice to add an appropriate decoder to Kotori.

So, can I humbly ask you to point out some documentation or otherwise tell me about how the metric topics and payloads actually look like?

Thanks already and with kind regards, Andreas.

[amotl]

I found some documentation on:

• https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/blob/master/docs/source/userguide/metrics.rst
• https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/blob/master/vehicle/OVMS.V3/components/vehicle_smarted/docs/metrics.rst

Is this the right information how telemetry field names look on the MQTT bus?

[dexterbg]

Hi Andreas,

thanks for connecting the dots :)

Btw, the compiled OVMS documentation is here:

• https://docs.openvehicles.com/en/latest/userguide/metrics.html
• https://docs.openvehicles.com/en/latest/components/vehicle_smarted/docs/metrics.html
• https://docs.openvehicles.com/en/latest/components/vehicle_renaulttwizy/docs/metrics.html
• etc.

But the OVMS3 MQTT support really lacks documentation, I'll have a look at that. The implementation has been discussed on the OVMS developer mailing list:

• http://lists.openvehicles.com/pipermail/ovmsdev/2018-July/005293.html + followups

The metrics topic scheme is <prefix>/metric/<name>. <prefix> can be configured by the user, defaults to ovms/<username>/<vehicleid>. <name> is the metric name with dots replaced by slashes, so you can e.g. subscribe to <prefix>/metric/xse/# to receive all Smart ED custom metrics. The value is the string representation of the metric value without the unit. Array and set members are separated by comma.

Do you need more info?

Events and notifications are also sent via MQTT, and commands can be sent to the module. Events and notifications topic schemes may be changed, as they currently don't follow the MQTT naming convention. If you want to support these already, we can sort that out now.

Regards, Michael

[amotl]

Dear @dexterbg,

thanks for your quick response with all that details regarding MQTT.

Thanks for connecting the dots :)

Let me share some thoughts I had about this.

Status quo

I see that you are already conceiving a nice mobile App [1,2] as a client to the backend system

as well as some other nice web-based dashboards within the OVMS web UI like

Regarding metrics

OVMS V3 is based on metrics. Metrics can be single numerical or textual values or complex values like sets and arrays [4].

Regarding gauges & charts on top of these metrics

The OVMS web framework has builtin support for the highly versatile Highcharts library with loads of chart types and options [4].

I also recognized that the system might also integrate with OpenEnergyMonitor already in any way?

Integrate with Kotori and Grafana

So, I am humbly asking if you actually see a fit here that ingesting metrics from MQTT through Kotori [5] into InfluxDB and displaying them within Grafana might fill some niche revolving the "data historian" aspect, where you can easily review metric data of the past and harness the whole power of InfluxDB and Grafana for more thorough data analysis.

I am asking because not everything which can be technically done would actually make sense and bring real value to the users of OVMS. So, do you believe it makes sense to work on a respective integration? If you believe it would, I will be happy to support as best I can.

Besides graphs, Grafana can also do maps reasonably well. We recently have been working on improving the Worldmap panel [6] which might get used to overlay metrics on a map. For getting whole tracks to a map there is also [7].

The collective benefit of this whole operation would be the possibility to make it feasible to mix and match data with information coming from other sources like outlined within [8]. Go figure.

What do you think about this?

With kind regards, Andreas.

[1] https://play.google.com/store/apps/details?id=com.openvehicles.OVMS [2] https://apps.apple.com/us/app/open-vehicles/id490098531 [3] https://docs.openvehicles.com/en/latest/components/ovms_webserver/docs/dashboard.html [4] https://docs.openvehicles.com/en/latest/components/ovms_webserver/docs/metrics.html [5] https://getkotori.org/docs/README.html [6] https://github.com/panodata/grafana-map-panel [7] https://github.com/panodata/grafana-map-panel/issues/44 [8] https://community.hiveeyes.org/t/datenmischwerk/702

[dexterbg]

Andreas,

the OVMS mobile Apps as well as the builtin web UI are meant to provide live displays and remote control for the module. They only support predefined data views and do not support pulling in data from other sources, like meteorological data.

If users currently want to do further analysis, they need to download the telemetry data, load it into a spread sheet and then do the necessary data sanitizing steps and chart creation manually. Telemetry data is also only kept on the server for a limited time, most records expire within 24 hours. To do historical analysis, users need to collect and assemble the data manually. Users without experience in data handling typically have issues with this, and being able to automate this would benefit all users.

If Kotori + Grafana enable non-technical users to easily aquire and archive data streams and combine arbitrary data sources for charts and analysis, that may make typical personal analysis and data visualization tasks much easier, and possibly even enable users to implement their own analysis ideas without needing to ask for help.

Your examples do look promising. I'll try to find some time to have a closer look at your system.

Regards, Michael

[amotl]

Dear Michael,

If users currently want to do further analysis, they need to download the telemetry data, load it into a spread sheet and then do the necessary data sanitizing steps and chart creation manually.

To do historical analysis, users need to collect and assemble the data manually. Users without experience in data handling typically have issues with this, and being able to automate this would benefit all users.

Thanks. This resonates with me and answers my question well.

If Kotori + Grafana enable non-technical users to easily acquire and archive data streams and combine arbitrary data sources for charts and analysis, that may make typical personal analysis and data visualization tasks much easier.

That was actually the whole idea around Kotori. While we chose InfluxDB+Grafana as best-of-breed foundation components, Kotori fills some gaps similar to what others are doing with Node-RED or custom one-shot mqtt_to_influx forwarder programs. However, Kotori in its role as a daemon is more of an infrastructure component than a (single-user) programming environment. Saying that, OVMS might additionally add support for Node-RED for users aiming to make their custom data and/or notification flows.

Back to Kotori: We recently upgraded our packaging system to provide readymade packages for Debian amd64 and armhf. This would permit individual users to run the data collector on BeagleBone or RaspberryPi SBC machines without efforts. It could provide data collection for single or multi-tenant datastreams, so it can also operate as a platform for multiple devices and users. Through its configurable vendoring subsystem, things can be designed to work for the average user almost out of the box.

Your examples do look promising.

Thanks for looking at them. Stumbling on OVMS the other day, my thoughts have been whether your project would benefit as yet another example of this DAQ/data historian infrastructure we made there.

I'll try to find some time to have a closer look at your system.

I will be happy to answer any questions which might come up. If you see how we could best adjust mapping data channels through appropriate specific vendor implementations, just let me know.

With kind regards, Andreas.