search

openhab / openhab-addons

Add-ons for openHAB
https://www.openhab.org/
Eclipse Public License 2.0
1.88k stars 3.59k forks source link

[modbus] generate JSON representation of registerArray and make it available as Data Thing #9576

Closed edolis closed 4 months ago

edolis commented 3 years ago

Hi I'm trying to find the best approach to get the data imported in OH through a Modbus connection AND save them to an external relational database for external analysis. To perform this, I'd like to be able to get data from the poller (whose register array I already use to populate various OH data things)saved into an additional OH Modbus Data Thing in the form of a JSON string (I was thinking of something like "timestamp":systemdate , "startAddr":startAddress, "length":length,"registerArray":ModbusRegisterArray.toString())). That way, I'll be able to export them to an external database and process further there as required. If possible, extract only a subset with startAddress/length parameters specified in the Data Thing definition - that would be nice, but not critical.

Can you please give me some hint what I should look into/where I could try to tweak the core library? I have so far spotted ModbusBitUtilities, being absolutely new I don't know how to proceed to modify the Data thing

Not asking you to implement simply because I understand it would not be a priority - but if you can make it I can save a lot of effort for me, as I don't know OH at all.

Thanks!

ssalonen commented 3 years ago

This issue probably belongs into openhab-addons repo. The modbus data thing is defined there: https://github.com/openhab/openhab-addons/blob/main/bundles/org.openhab.binding.modbus/src/main/java/org/openhab/binding/modbus/internal/handler/ModbusDataThingHandler.java

May I suggest to use hex format for registers as it would be compatible with bin2json transformation https://openhab.org/addons/transformations/bin2json/

edolis commented 3 years ago

thanks ssalonen, I did look there, but I got confused when I tried, as an entry point, to see where ModbusRegisterArray was defined. Didn'find there and that's why I looked in core - I thought for some reason the whole block had been moved to core. Didn't think of looking back - Sure, hex would be OK.

ssalonen commented 3 years ago

You would probably want to introduce new channel to poller thing. The channel definitions are here https://github.com/openhab/openhab-addons/tree/main/bundles/org.openhab.binding.modbus/src/main/resources/OH-INF/thing (you can take example from other things)

Probably you would like to have it as String type? One needs to think if different json would be warranted for errors.

The poller thing java code resides in https://github.com/openhab/openhab-addons/blob/main/bundles/org.openhab.binding.modbus/src/main/java/org/openhab/binding/modbus/handler/ModbusPollerThingHandler.java

edolis commented 3 years ago

thanks, ssalonen. Appreciate your help. I had a quick look so far, having a lot of headaches trying to set up Rasppi and I had to suspend. So far I understood that probably I should modify ModbusDataConfiguration to add a length parameter [so that I can read x many registers from a starting point], but I am trying to understand if that's too intrusive and there are smarter ways to do that. Also because having no experience, I don't know the impact on the UI (will there be a new box? How that box can be enabled/disabled according the selected type?) Also trying to understand where I can get the timestamp out of the modbus bridge readout (i mean ModbusRegisterArray) and if it would be possible and hw hard do the next leap and manage to read all the data items defined in the range, and tally their reading in JSON in address:value series (that way you guarantee consistency between the values persisted in the database and the current transformed values in OH maybe?) Lots of fish to fry. Ah, I'm very new and it's hard... I could modify the code (got some ideas, , but still have to figure out how to debug and test it). I guess I'll have to bang my head a little and I'll come back to bother if I am in a really cul-de-sac desperate situation...

Rossko57 commented 3 years ago

I wonder if this is a bit like a generic "string" representation of modbus registers (assuming we are limiting to holding/input types). i.e. just get a block of hex bytes representing the contents of a block of 16-bit registers. Would need to select a hi/lo byte convention.

We might imagine a Modbus data Thing where we give the start register and byte count, and get a string like "A5B6992AF6" which can be linked to an Item and then reprocessed or transformed in arbitrary ways. Treat as ASCII, "Hello world", or build into JSON (you already know register addresses, you defined them), or assemble a bitmap image, etc.

Like any data Thing, the binding already makes available a last read timestamp channel.

edolis commented 3 years ago

yes exactly, sorry of course I did not manage to make myself understood. Let's start with the final goal: I'd like to persist to a relational database the register readings, keeping the link to the timestamp of the binding. Now, the easiest way I mentioned is getting the plain HEX register read by the bridge - identifying as start,length and shoving to the db as a hex string. There I'd need some stored procedure to unpack and get the correct values to push to a table. Downside being that I am duplicating - hence making potentially inconsistent - what I am already doing\tweaking in openHAB. But this is already a very good result - as I cannot, say, build a service to run in parallel to poll the data from modbus, as the port 503 is the same and would break the openHAB binding. (Unless i throw away the OH modbus binding, have the external service do the OH binding stuff and push the values some way to OH. Horrific to me.) Next level, which would be much more complicated [but smart as it can be], is (don't know if this is possible) reading all the data things derived by that poller and dump them as JSON. example - say the poller reads 20 register from 0 and I have

Rossko57 commented 3 years ago

reading all the data things derived by that poller and dump them as JSON.

Understood, but you're the first to ask for this in five years. It's a bit specialist. I'm suggesting a more generic solution - binding makes available entire polled register block in a single channel/Item in some generally convenient form, a hex string seems reasonable. We can debate forms "A5B6449A" or "A5B6, 449A" or even decimal etc. that's just detail. If you want to reprocess that string Item into JSON or CSV or ASCII or BMP you can, with ordinary openHAB transforms and rules.

edolis commented 3 years ago

Well, that would be already a good start. At least I do not have to choose between OH or external poller only to query data from modbus.

ssalonen commented 3 years ago

I agree with @Rossko57 , makes sense to keep it simple.

This should not be too hard to implement as a new channel in poller thing. It has all the bytes coming from the device and openHAB provides the conversion utilities even to build up the hex string.

edolis commented 3 years ago

Agree, being totally ignorant I thought that ModbusRegisterArray.toString(), reprocessed clipping it with a start and length would be the easiest way... add a timestamp and we should be done - for the data part -. Something like {"pollerID":"OHpollerId",'timestamp':"2021-01-01 00:01:01", "regstartpos":"40001", "rawread":"02a45f0d402a45f0d411"} of course I'm only talking of the data contents, the remaining part I have no clue about

Rossko57 commented 3 years ago

To repeat; every Modbus data Thing already comes with timestamp channels, which you can choose to link to an Item and do whatever with e.g. assemble some message in a format of your choice in a rule, from parts of multiple Items.

I'm thinking a generic "ascii" data Thing.

This could have readStart/writeStart specified as a byte target e.g. 250.0, and a readCount/writeCount in bytes. (a new parameter for data Things) However - assuming we do want this to be a writeable type (and why not), that brings some difficulties with writing part-registers, which Modbus does not natively do.

As (a) writing is going to be even rarer than reading in this mode, and (b) simple is good, and (c) the user is certainly going to be doing some kind of external processing on this data... Let's go simple and just do whole registers.

data Thing type ascii, outputs a string, specify readStart/writeStart, and a readCount/writeCount which is of whole 16-bit registers (consistent with poller length param). Output looks like "A5617890B672"

If the user needs byte boundaries in part registers, this can be done in transforms. Likewise comma separated lists as bytes or words, etc.

If you must output as your chosen JSON form - you can do that in a javascript transformation, adding field names, separators, system timestamp etc. But not every other user is forced to use your chosen format.

You'll understand I'm pushing against your desired JSON format because it's very specific to you, and looking for generic support instead. Nothing personal!

ssalonen commented 3 years ago

The write part in fact already exists https://openhab.org/addons/bindings/modbus/#advanced-write-using-json

Only thing we are missing is getting the raw data as hex from device. I think this would fit perfectly as an additional channel for the poller thing...

Rossko57 commented 3 years ago

I was thinking new data Thing type as that already comes with timestamps, transform, features etc. and would be consistent with other multi-register data Thing type like int32.

ssalonen commented 3 years ago

True... Many of the same functions would be usable here as well

edolis commented 3 years ago

sorry for late reply. Yes, writing is personally of no use to me - I would in any case never write a whole set of registers anyway. As for Json, I just mentioned it as it is quite easy to process in mySQL(/mariaDB), but in the end, if I can have a table populated with timestamp of saving, timestamp of poller, poller ID, Start, length, ASCII string, That's obviously fine with me. I just don't know how to do it at the moment as I am very new with OH (I imagine I can add and push some metadata or similar) - If you tell me it's feasible, perfect.. I'll figure out how :)

lsiepel commented 4 months ago

Much has changed to modbus the last years, but i can't find a specific PR that resolves this. This question was already unique at the time and if not already resolved by now i guess it won't be in the future. Feel free to re-open this issue if there is still a need.