search

victronenergy / venus-influx-loader

NodeJS server that takes from MQTT into Influx, and config UI and still more
MIT License
12 stars 4 forks source link

Bug: fix MQTT subscription #95

Closed mman closed 3 days ago

mman commented 1 month ago

Quoting suggestion from @wiebeytec:

There are some issues with the use through VRM and on-line MQTT servers. First, client.subscribe('N/+/+/#') is the same as client.subscribe('N/#') and will be rejected by the on-line MQTT servers soon, because of the load it incurs.

This needs to be fixed appropriately.

mman commented 1 month ago

@wiebeytec Working on addressing/improving/fixing the MQTT subscription logic in Venus Influx Loader: currently the code works in a way that:

  1. Immediately after MQTT onConnect, we subscribe to 'N/+/+/#'. I assume this is to match portalId/system/Serial.
  2. Then in MQTT onMessage we check whether it's a system/0/Serial and extract the portalId
  3. And then once we know the portalId we subscribe to N/${portalId}/+/#)`

When you say in your comment above that N/+/+/# will be rejected, I'd like to understand how to implement the logic of extracting the portalId first and then subscribing to receive everything.

Experimenting with venus-docker I think that:

  1. Detecting the portalId can be replaced by directly subscribing to N/+/system/0/Serial and parsing the reply.
  2. Receiving everything can simply be done by subscribing to N/${portalId}/#.

What would be your approach? Is my assessment correct?

For your reference, the logic of first time connect to unknown portal is here:

https://github.com/victronenergy/venus-influx-loader/blob/d18ce6ebd83f98422aeaf3bd8cc788cc22a2b8f6/src/server/loader.js#L318-L320

Initial message parsing is here (note that instance number 0 was already parsed out):

https://github.com/victronenergy/venus-influx-loader/blob/d18ce6ebd83f98422aeaf3bd8cc788cc22a2b8f6/src/server/loader.js#L108-L119

And repeated connect to a known server is here:

https://github.com/victronenergy/venus-influx-loader/blob/d18ce6ebd83f98422aeaf3bd8cc788cc22a2b8f6/src/server/loader.js#L323-L332

wiebeytec commented 1 month ago

@wiebeytec Working on addressing/improving/fixing the MQTT subscription logic in Venus Influx Loader: currently the code works in a way that:

  1. Immediately after MQTT onConnect, we subscribe to 'N/+/+/#'. I assume this is to match portalId/system/Serial.
  2. Then in MQTT onMessage we check whether it's a system/0/Serial and extract the portalId
  3. And then once we know the portalId we subscribe to N/${portalId}/+/#)`

When you say in your comment above that N/+/+/# will be rejected, I'd like to understand how to implement the logic of extracting the portalId first and then subscribing to receive everything.

Experimenting with venus-docker I think that:

  1. Detecting the portalId can be replaced by directly subscribing to N/+/system/0/Serial and parsing the reply.
  2. Receiving everything can simply be done by subscribing to N/${portalId}/#.

What would be your approach? Is my assessment correct?

That works on LAN, but not WAN. Or at least, partly. The intracies of when it does and does not work are too subtle to fully support this, so that's why for VRM use, you need to use the VRM API to retrieve installation identifers of interest.

You then also have the MQTT server hostname available in the response. Not that calculculating it yourself doesn't work, but having one source of truth in case it ever changes in the future, may be preferable.