L+G E450 Stromnetz Graz (RJ12)

I'm running into very similar issues to #33 with my L+G e450 from Stromnetz Graz (Austria).

So far it seems that https://github.com/scs/smartmeter-datacollector/blob/d4613f504b7b0e1bb63af14c67f200402a6db744/smartmeter_datacollector/smartmeter/hdlc_dlms_parser.py#L61 returns a list of 33 values in total (1 bytearray of the timestamp data, then 1 bytearray of the OBIS code, then 1 integer of the associated value and those two repeat 15 more times).

After the data got extracted into that list, it then gets thrown into https://github.com/scs/smartmeter-datacollector/blob/d4613f504b7b0e1bb63af14c67f200402a6db744/smartmeter_datacollector/smartmeter/hdlc_dlms_parser.py#L80 where it hits https://github.com/scs/smartmeter-datacollector/blob/d4613f504b7b0e1bb63af14c67f200402a6db744/smartmeter_datacollector/smartmeter/hdlc_dlms_parser.py#L84 which hands the bytearray containing the timestamp to self._client.parsePushObjects() causing it to error in the end as it tries to dissect this even further. If I modify the code to parsed_objects = self._client.parsePushObjects(self._dlms_data.value), it tries to parse the timestamp as OBIS code and complains that this is invalid.

There is a fork of this repository that seems to have implemented parsing this data in a more... crude but successful way (https://github.com/scs/smartmeter-datacollector/compare/master...martink173:smartmeter-datacollector:master --> parse_byte_string()), but the existing data structure already feels like sooo close to what should actually happen.

Somehow it seems like self._client.getData() does a bit too much or too little on this data compared to the responses from most other smart meters out there. The data is however already mostly there already (self._dlms_data.value contains decrypted, correct bytearrays and values). To write a proper fix for this special case though I guess I'd need to know how self._dlms_data.value looks on other smart meters. Maybe this info is already enough (I'm not too familiar with gurux-dlms after all) to help you, otherwise maybe someone could help me with a little dump using a cheeky print() or similar.

If someone else wants to tackle this, I can also share data of course.

I have the same issues with https://github.com/Gurux/Gurux.DLMS.Python/blob/a2faa326094702780685c38b77891dd4f86a7dd2/Gurux.DLMS.Push.Listener.Example.python/main.py by the way, after printing the data successfully in lines 137 and 139 it crashes around line 143.

Hi! Thanks for explaining the problem in such detail, this helps a lot. We already encountered the problem with the pushObject list and related software crashes. There are two possibilities how smart meters we have seen so far push the data out. Either they provide a push-object list as the first element that is sent stating all the following OBIS codes that are following in this message. Or they just send the OBIS codes and values directly. Our software currently only handles the first variant (due to Gurux DLMS) and does not work out of the box with the second variant.

However, please have a look at the issue #21. Especially the comment: https://github.com/scs/smartmeter-datacollector/issues/21#issuecomment-1120206925. There you find a fix for this problem, but you need to modify the Python code. In short, you set the object-push list manually at program initialization. Use the DLMS-Translator from Gurux or a Python debugger to find out which OBIS codes in what order are sent by your meter.

We would like to fix this in the smartmeter-datacollector in general, but it is not easy to find a good solution for every meter we encountered so far. Some meters do not send a push-list but send multiple variants of messages. I doubt that manually setting the object-push-list works with these types of meter.

Yeah, considering that there are hundreds of bytes transmitted just to get a timestamp and 16 4-byte values across the wire one would hope that there's some way to have a firmware/protocol version or other UID in there to match against. Doesn't seem like it though. :-(

If it helps, here's a raw, unencrypted message from this meter:

0221090C07E608130500240BFF80008009060100010800FF06000C8F8009060100010801FF0600096A4709060100010802FF060003253909060100010700FF06000001E109060100020800FF060000000009060100020801FF060000000009060100020802FF060000000009060100020700FF060000000009060100030800FF0600006E6E09060100030801FF060000671009060100030802FF060000075E09060100030700FF060000000009060100040800FF06000497B609060100040801FF060003727609060100040802FF060001254009060100040700FF060000006E

16 bytes header/timestamp, then 16*13 bytes data (tuples of 6 bytes OBIS code and 4 bytes value), so case 2 of https://github.com/scs/smartmeter-datacollector/issues/31#issuecomment-1135542827

I suspect that it might be more correct/flexible to also parse the OBIS Codes in each message instead of hard-coding them in the constructor of the class. I'm still unsure how to distinguish the 2 cases though, but once I find a way to check out a message of case 1 in the comment above, I guess it shouldn't be too hard (maybe look at types in that list or something...).

All in all it looks like there are at least 3 cases:

The currently properly handled case (all codes are sent first, then the values)
The simplified case (timestamp + code/value pairs)
The Vienna special case (timestamps + values, no codes in the message at all)

At least case 3 likely needs a way to define a list of expected codes in the config file, my case (2) should maybe be able to get away with creating a new GXDLMSPushSetup() for every message.

Here is the pdf I got that apparently describes everything coming out of that box :roll_eyes: Kundenschnittstelle_Beschreibung_Kunden_v3.pdf

Apparently there's supposed to be a version number somewhere in there too, though it seems to me like the 0x0221 rather refers to the 33 (not 34) elements coming after, not the protocol version 33.

Here is my code so far that works, sprinkled with a few assertions:

from gurux_dlms.objects import GXDLMSData, GXDLMSObject, GXDLMSRegister, GXDLMSPushSetup, GXDLMSClock, GXDLMSCaptureObject
...
    def parse_to_dlms_objects(self) -> Dict[str, GXDLMSObject]:
        parsed_objects: List[Tuple[GXDLMSObject, int]] = []
        if isinstance(self._dlms_data.value, list):
            #pylint: disable=unsubscriptable-object
            # TODO: detect this properly, hardcoded for now
            msg_format = 2
            if msg_format == 1:
                # push list is sent in the message, parsePushObjects can be used
                parsed_objects = self._client.parsePushObjects(self._dlms_data.value[0])
                for index, (obj, attr_ind) in enumerate(parsed_objects):
                    if index == 0:
                        # Skip first (meta-data) object
                        continue
                    self._client.updateValue(obj, attr_ind, self._dlms_data.value[index])
                    LOGGER.debug("%s %s %s: %s", obj.objectType, obj.logicalName, attr_ind, obj.getValues()[attr_ind - 1])
            elif msg_format == 2:
                # message contains timestamp + OBIS code/value pairs
                p =  GXDLMSPushSetup()
                # fill setup with definitions
                # first element is clock
                p.pushObjectList.append((GXDLMSClock(), GXDLMSCaptureObject(2, 0)))
                # every second element after that is the OBIS code
                for idx in range(1,len(self._dlms_data.value),2):
                    assert isinstance(self._dlms_data.value[idx], bytearray)
                    assert len(self._dlms_data.value[idx]) == 6
                    # convert bytearray to list of ints, map ints to str and add a dot between each element
                    obis_code = ".".join(map(str, list(self._dlms_data.value[idx])))
                    p.pushObjectList.append((GXDLMSRegister(obis_code), GXDLMSCaptureObject(2, 0)))
                parsed_objects = p.pushObjectList
                assert len(parsed_objects) == (len(self._dlms_data.value) + 1) / 2
                # Add values
                object_idx = 0  # index of the pushObjectList
                for idx in range(0,len(self._dlms_data.value),2):  # index of the message contents
                    if idx == 0:
                        # first the timestamp
                        assert isinstance(self._dlms_data.value[idx], bytearray)
                        assert len(self._dlms_data.value[idx]) == 12
                    else:
                        # then all the integer values
                        assert isinstance(self._dlms_data.value[idx], int)
                    obj = parsed_objects[object_idx][0]
                    attr_ind = parsed_objects[object_idx][1]
                    self._client.updateValue(obj, attr_ind.attributeIndex, self._dlms_data.value[idx])
                    LOGGER.debug("%d %s %s %s: %s", object_idx, obj.objectType, obj.logicalName, attr_ind.attributeIndex, obj.getValues()[attr_ind.attributeIndex - 1])
                    object_idx += 1
            elif msg_format == 3:
                # message contains timestamp + values, OBIS codes are only documented
                # TODO: make the codes configurable, this is for Wiener Netze only according to issues 21 and 31
                p =  GXDLMSPushSetup()
                p.pushObjectList.append((GXDLMSClock(), GXDLMSCaptureObject(2, 0)))
                p.pushObjectList.append((GXDLMSRegister("1.1.1.8.0.255"), GXDLMSCaptureObject(2, 0)))
                p.pushObjectList.append((GXDLMSRegister("1.1.2.8.0.255"), GXDLMSCaptureObject(2, 0)))
                p.pushObjectList.append((GXDLMSRegister("1.1.3.8.0.255"), GXDLMSCaptureObject(2, 0)))
                p.pushObjectList.append((GXDLMSRegister("1.1.4.8.0.255"), GXDLMSCaptureObject(2, 0)))
                p.pushObjectList.append((GXDLMSRegister("1.0.1.7.0.255"), GXDLMSCaptureObject(2, 0)))
                p.pushObjectList.append((GXDLMSRegister("1.0.2.7.0.255"), GXDLMSCaptureObject(2, 0)))
                p.pushObjectList.append((GXDLMSRegister("1.0.3.7.0.255"), GXDLMSCaptureObject(2, 0)))
                p.pushObjectList.append((GXDLMSRegister("1.0.4.7.0.255"), GXDLMSCaptureObject(2, 0)))
                parsed_objects = self._p.pushObjectList
                for index, (obj, attr_ind) in enumerate(parsed_objects):
                    self._client.updateValue(obj, attr_ind.attributeIndex, self._dlms_data.value[index])
                    LOGGER.debug("%d %s %s %s: %s", index, obj.objectType, obj.logicalName, attr_ind.attributeIndex, obj.getValues()[attr_ind.attributeIndex - 1])
            else:
                # Something went wrong
                assert False
        self._dlms_data.clear()
        return {obj.getName(): obj for obj, _ in parsed_objects}

I also added the OBIS codes to meter_data.py, so they get properly prettified in the logs, see martink173's fork.

It still complains about a missing ID object (falling back to /dev/ttyUSB0 in my case), but I guess that can be fixed somewhere in the config to set the actual smart meter serial number or something like that in there. At least outside the encrypted part there seems to be something with the ASCII letters "LGZ" in every first message sent that might be the "version number" promised in the documentation... no idea how to get that using the library yet though. There is an 8 byte "system title" containing this data when dumping the whole message (from the first to the last 7E) into https://gurux.fi/GuruxDLMSTranslator (Messages tab), then taking all the "Block Data" segments, joining them and then dumping this into the "PDU" tab (which then shows the "system title" and the encrypted data blob). Maybe something could be done using this data, though I suspect that there's not a public/up-to-date list of serial numbers and the companies that own them available...

The "Our Meter" part in https://github.com/scs/smartmeter-datacollector/issues/21#issuecomment-1020986864 seems to contain the serial number of the device in the message, so probably this "system title" is a dead end (seems more related to encryption by the looks of it) and the system ID just needs to be hard-coded in the config in formats 2 and 3.

@MarkusTeufelberger Thank you for your useful contribution! I will, based on your findings and code proposition, improve this part of our software, add further (fallback) configuration where necessary, add the auto-detection of the message format and write unit tests for all the different cases. At the moment, I just have a lot to work on for other projects so please have a little bit of patience :) . But you found a working solution for your case and can work with that. 👍

Meter ID: Yes our meters always send its ID as DLMS/COSEM object (assigned to an OBIS code). Yours doesn't unfortunately. You can decode the unencrypted message content you provided above in the DLMS-Translator (installed tool, not web edition) using the "Data to XML" tab. There is no ID available. I will check if we could use and somehow extract the "System Title" as another fallback for the ID. Or we just add a configuration option "fallback_id" which is set if no ID can be found in the message. Currently as you have seen for yourself, the fallback is the device path of the serial dongle.

I also suspect that with actual active communication with the meter (authenticating with the respective key and then sending requests to it) it might be possible to get some ID or serial number or something like that, but that doesn't seem to be at all implemented, so far this project just passively listens to whatever is being sent on the wire, decrypts and parses the contents and passes it on. It doesn't interact with the meter and doesn't use the authentication key at all.

My end goal is anyways to use a microcontroller (ideally with Tasmota) to do the data forwarding step so I don't need electricity in my electricity meter box (which ironically does not have a plug available to get electricity for a raspi or similar...), but I first wanted/needed to understand what's actually going on in the data that's being sent before I can start to implement this so I don't create yet another "take bytes 17-21 and assume it is the current counter value" solution that seems to be often the case (and that break down as soon as someone has a different provider).

Thanks for your nice feedback, I hope you soon get around to take a closer look, I'd also be more than willing to help with this on weekends, my first instinct would be to look further on the tests sections and make it possible to add different messages there so various "dialects" can be tested and verified.

Ugh, I started writing tests and apparently there are even more differences between implementations.

"Wirkenergie Bezug" is published via OBIS code "1.0.1.8.0.255" by Stromnetz Graz but via "1.1.1.8.0.255" by EKZ (for whatever reason they seem to send some data on "channel 0" and other data on "channel 1" as per https://www.promotic.eu/en/pmdoc/Subsystems/Comm/PmDrivers/IEC62056_OBIS.htm at least in the test fixtures).

The easy fix is to add the channel 0 versions wherever applicable in LGE450_COSEM_REGISTERS (see #37), though for the future it might be better to extend cosem.py to just ignore the channel number.

I'm still unsure how to differentiate the various DLMS dialects, putting it in the config might be the easiest way (just name your provider there and you'll either be prompted to submit sample data or get the correct config). Unfortunately there are nearly no examples of actual data + keys on the internet and I for example also don't want to immortalize my private key in the unit tests of this software. It might be worth to write a small utility to re-encrypt DLMS data with a dummy key (like 0x01020304...), fix a few checksums and IVs in the serial data and use that as test fixture instead. That way people can post their decrypted DLMS data and encrypted HDLC traffic and don't need to publish their private key. I didn't find a good solution yet to start straight from decrypted DLMS data yet, seems like the gurux library does a bit more magic behind the scenes to set everything up the way they need it.

I went ahead and implemented the config option, see #38

I also suspect that with actual active communication with the meter (authenticating with the respective key and then sending requests to it) it might be possible to get some ID or serial number or something like that, but that doesn't seem to be at all implemented, so far this project just passively listens to whatever is being sent on the wire, decrypts and parses the contents and passes it on. It doesn't interact with the meter and doesn't use the authentication key at all.

This project was initiated by EKZ for smart-meters installed by them, should however be expandable for other smart-meter (primarily for Switzerland due to a new law that states that a end-user must have an opportunity to gather data from his smart-meter). All smart-meters we've encountered so far only push data through the M-Bus or P1 interface. The electricity providers here in Switzerland do not allow any incoming data traffic / requests over this interface. Therefore, bidirectional traffic will probably not be implemented in this project.

The easy fix is to add the channel 0 versions wherever applicable in LGE450_COSEM_REGISTERS (see https://github.com/scs/smartmeter-datacollector/pull/37), though for the future it might be better to extend cosem.py to just ignore the channel number.

Yes, probably we will extract the required information from the OBIS codes and generalize the Cosem registers in the future for all meters.

I'm still unsure how to differentiate the various DLMS dialects, putting it in the config might be the easiest way (just name your provider there and you'll either be prompted to submit sample data or get the correct config).

If possible we want to do this automatically because of two reasons: As stated above the tool will probably be extended for usage in whole Switzerland and there are over 600 providers. Secondly, the tool should not be more complicated as it already is for the end-user, so entering sample data to get the right config is no option in my opinion. Let me find a better (more automatic) solution when I get some time.

Unfortunately there are nearly no examples of actual data + keys on the internet and I for example also don't want to immortalize my private key in the unit tests of this software. It might be worth to write a small utility to re-encrypt DLMS data with a dummy key (like 0x01020304...), fix a few checksums and IVs in the serial data and use that as test fixture instead. That way people can post their decrypted DLMS data and encrypted HDLC traffic and don't need to publish their private key. I didn't find a good solution yet to start straight from decrypted DLMS data yet, seems like the gurux library does a bit more magic behind the scenes to set everything up the way they need it.

Yes, I encountered this problem myself a couple of times. Usually the people send me encrypted data and their private key (by e-mail) and I cannot use the data + key for any unit-tests. Such a tool would be very helpful. For a start it would already suffice to generate valid unencrypted HDLC frames from the decrypted DLMS data to test the parsing. Gurux does a lot of "magic" under the hood but as you probably noted, debugging and understanding how the decoding/parsing works based on the Gurux library is very tedious and difficult. Seems to be auto-generated and later fixed code somehow...

I went ahead and implemented the config option, see https://github.com/scs/smartmeter-datacollector/pull/38

Thanks for this pull-request. I'm currently working on the same but couldn't complete it due to lack of time at the moment. I will use your proposition and combine it with ideas of mine.

scs / smartmeter-datacollector

L+G E450 Stromnetz Graz (RJ12) #34