search

scs / smartmeter-datacollector

Smart Meter Data Collector
Other
63 stars 24 forks source link

Integrate push-setup configuration #31

Closed raymar9 closed 1 year ago

raymar9 commented 2 years ago

Based on issue: #21

Some meters are configured in a way that no push object list is provided at the beginning of the push message. Currently, smartmeter-datacollector is not able to parse these messages. However, Gurux provides the option to set a push setup list of OBIS codes that will be received in every message. This configuration should be integrated into the smartmeter-datacollector and smartmeter-datacollector-configurator projects in a generic way to be able to set this for other push configurations and meters as well.

raymar9 commented 2 years ago

For reference: Gurux forum: https://www.gurux.fi/comment/23821 Push Setup information: https://www.gurux.fi/Gurux.DLMS.Objects.GXDLMSPushSetup

raymar9 commented 2 years ago

@gcey / @LichtiMC Since we do not possess a L+G meter without the push-list configuration I require some raw data for testing and writing unit tests. Can you provide me with some unencrypted raw data and the corresponding push object list? Or send me some encrypted raw data including the decryption key and the corresponding push object list? However, this unit test code, your data and decryption key is going to be published on this repository, so unencrypted data, if possible, would be the easiest solution.

gcey commented 2 years ago

Raw data: 7E A0 67 CE FF 03 13 38 BD E6 E7 00 DB 08 4C 47 5A 67 73 84 27 EE 4F 20 00 10 13 D4 CC B4 B4 D6 C5 58 2C EB 97 4D 58 18 E2 5A 99 9B 76 BF E6 F6 29 19 FA 11 63 7C 92 48 6D 90 03 B7 5B 97 7C 38 2F FB 35 04 91 21 BA 08 48 BA 3D F0 77 9B 47 64 C6 37 59 8B 61 02 B2 F8 ED CE 1E 20 B1 8A F8 58 8F E3 6E 6C 28 51 97 C2 7E

The crypted part of this is: CC B4 B4 D6 C5 58 2C EB 97 4D 58 18 E2 5A 99 9B 76 BF E6 F6 29 19 FA 11 63 7C 92 48 6D 90 03 B7 5B 97 7C 38 2F FB 35 04 91 21 BA 08 48 BA 3D F0 77 9B 47 64 C6 37 59 8B 61 02 B2 F8 ED CE 1E 20 B1 8A F8 58 8F E3 6E 6C 28 51

After decryption, this part is: 0F 00 10 13 D4 0C 07 E6 04 0E 04 0F 31 02 FF 80 00 00 02 09 09 0C 07 E6 04 0E 04 0F 31 02 FF 80 00 80 06 00 04 FE 3D 06 00 00 00 00 06 00 01 C6 CB 06 00 00 01 5C 06 00 00 02 92 06 00 00 00 00 06 00 00 01 40 06 00 00 00 00

I successfully used GXDLMSPushSetup() in this way:


from gurux_dlms.objects import GXDLMSPushSetup, GXDLMSClock, GXDLMSCaptureObject
...
class HdlcDlmsParser:
    ...
    def __init__(self, cosem_config: Cosem, block_cipher_key: Optional[str] = None) -> None:
        ...  
        self._p = GXDLMSPushSetup()
        self._p.pushObjectList.append((GXDLMSClock(), GXDLMSCaptureObject(2, 0)))
        self._p.pushObjectList.append((GXDLMSRegister("1.1.1.8.0.255"), GXDLMSCaptureObject(2, 0)))
        self._p.pushObjectList.append((GXDLMSRegister("1.1.2.8.0.255"), GXDLMSCaptureObject(2, 0)))
        self._p.pushObjectList.append((GXDLMSRegister("1.1.3.8.0.255"), GXDLMSCaptureObject(2, 0)))
        self._p.pushObjectList.append((GXDLMSRegister("1.1.4.8.0.255"), GXDLMSCaptureObject(2, 0)))
        self._p.pushObjectList.append((GXDLMSRegister("1.0.1.7.0.255"), GXDLMSCaptureObject(2, 0)))
        self._p.pushObjectList.append((GXDLMSRegister("1.0.2.7.0.255"), GXDLMSCaptureObject(2, 0)))
        self._p.pushObjectList.append((GXDLMSRegister("1.0.3.7.0.255"), GXDLMSCaptureObject(2, 0)))
        self._p.pushObjectList.append((GXDLMSRegister("1.0.4.7.0.255"), GXDLMSCaptureObject(2, 0)))
    ...
    def parse_to_dlms_objects(self) -> Dict[str, GXDLMSObject]:
        parsed_objects: List[Tuple[GXDLMSObject, int]] = []
        if isinstance(self._dlms_data.value, list):
            #pylint: disable=unsubscriptable-object
            parsed_objects = self._p.pushObjectList
            for index, (obj, attr_ind) in enumerate(parsed_objects):
                self._client.updateValue(obj, attr_ind.attributeIndex, self._dlms_data.value[index])
                LOGGER.debug("%d %s %s %s: %s", index, obj.objectType, obj.logicalName, attr_ind.attributeIndex, obj.getValues()[attr_ind.attributeIndex - 1])
        self._dlms_data.clear()
        return {obj.getName(): obj for obj, _ in parsed_objects}
raymar9 commented 2 years ago

Thank you. I try, but I'm not sure if I find a way to use this raw (encrypted data) without the actual decryption key for writing some unit- / and integration tests.

raymar9 commented 2 years ago

Unfortunately, I can't use your example data since Gurux requires raw data to be pushed into the HDLC buffer and due to the complexity of Gurux I can't directly inject the decrypted part somewhere.

I need either:

gcey commented 2 years ago

I do not have any whole unencrypted message. Can I send my key to you by e-mail?

raymar9 commented 2 years ago

@gcey That would work at least for development & testing on my side. It would be the clean way to write a unit test with your example message and using your key and commit / push this to this Github project for future developments. However, I think this is not acceptable for you, due to data privacy reasons and since your meter is provided by your electricity provider. But that is okay, better having an example during development than having nothing :). Unfortunately, I could not find a way to produce an unencrypted message or an encrypted message with a default-key based on your provided decrypted data with the Gurux tools. That would be another good way. Do you by chance have an idea achieving that?

You can send your key to my temporary email address: fez8kbzefnq@temp.mailbox.org

Some further questions: Is it correct that in your message absolutely no OBIS codes but only the data values are transmitted? If yes, we could determine 3 different variants by now how the data is pushed from a L+G E450 meter:

  1. with the object-push-list in the beginning of the message (like the datacollector works now)
  2. like the example of LichtiMC: obisA | dataA | obisB | dataB | obisC | dataC | … (see https://github.com/scs/smartmeter-datacollector/issues/21#issuecomment-1020986864)
  3. your case: no OBIS information -> must be provided as object-push-list beforehand

We understand how option 1 & 2 happen to be configured but not 3. How do you read out your meter? By IR or M-Bus? Is it correct to assume that you've never had access to the configuration of your smartmeter but your provider set the data -push configuration and provided you with the decryption key?

Thanks for helping me out!

gcey commented 2 years ago

Unfortunately, I could not find a way to produce an unencrypted message or an encrypted message with a default-key based on your provided decrypted data with the Gurux tools.

I also tried to do so, but with no success.

You can send your key to my temporary email address: fez8kbzefnq@temp.mailbox.org

Did you get it?

Is it correct that in your message absolutely no OBIS codes but only the data values are transmitted?

To me it seems so.

We understand how option 1 & 2 happen to be configured but not 3.

Ask Wiener Netze, because Wien ist anders :-)

How do you read out your meter? By IR or M-Bus?

IR.

Is it correct to assume that you've never had access to the configuration of your smartmeter but your provider set the data -push configuration and provided you with the decryption key?

Yes.

raymar9 commented 2 years ago

Did you get it?

Yes, thank you. I could decode the example data from above with your key.

kreutpet commented 2 years ago

hi , many thanks for sharing the code. i just attended the energy hackday and were also facing the issue with the E450 and the push setup here the links to the content we created last year we will further document the output of this year here

i am trying the setup the dev env here at home on my linux server whcih connects with MBUS and USB TTL to the E450. Currently struggling with the pipenv install --dev command which seems to be broken with python 3.10 I do have a L&G E450 from AEW and i use the gurux push listner client to collect the byte log. you can find my log file in which i logfile the bytes coming from the meter

i decoded some of the bytes in the gurux translator and got , at least i think so a push setup example here the bytes send from the meter

7E 
A0 84 CE FF 03 13 12 8B E6 E7 00 E0 40 00 01 00 00 70 0F 00 
86 3A 0F 0C 07 E6 09 13 01 0C 33 14 FF 80 00 00 02 0E 01 0E 
02 04 12 00 28 09 06 00 08 19 09 00 FF 0F 02 12 00 00 02 04 
12 00 28 09 06 00 08 19 09 00 FF 0F 01 12 00 00 02 04 12 00 
01 09 06 00 00 60 01 00 FF 0F 02 12 00 00 02 04 12 00 03 09 
06 01 00 01 07 00 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 
00 02 07 00 FF 0F 02 12 00 00 5C 3C 7E 7E A0 7D CE FF 03 
13 D0 45 E0 40 00 02 00 00 6C 02 04 12 00 03 09 06 01 01 01 
08 00 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 02 08 00 
FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 05 08 00 FF 0F 
02 12 00 00 02 04 12 00 03 09 06 01 01 06 08 00 FF 0F 02 12 
00 00 02 04 12 00 03 09 06 01 01 07 08 00 FF 0F 02 12 00 00 
02 04 12 00 03 09 06 01 01 08 08 00 FF 0F 02 12 00 00 C1 95 7E 
7E A0 8A CE FF 03 13 AA EA E0 C0 00 03 00 00 79 02 04 
12 00 03 09 06 01 00 1F 07 00 FF 0F 02 12 00 00 02 04 12 00 
03 09 06 01 00 33 07 00 FF 0F 02 12 00 00 02 04 12 00 03 09 
06 01 00 47 07 00 FF 0F 02 12 00 00 09 06 00 08 19 09 00 FF 
09 08 35 36 35 34 37 34 31 33 06 00 00 00 00 06 00 00 0C 60 
06 00 8D D6 C4 06 00 F2 70 0C 06 00 2B FA 22 06 00 06 C0 07 06 
00 06 84 64 06 00 0F 29 2E 12 01 5C 12 01 FC 12 01 DC AC 
ED 7E

it comes in 3 frames 

<HDLC len="131" >
<TargetAddress Value="13311" />
<SourceAddress Value="1" />
<!--S frame.-->
<FrameType Value="13" />
<PDU>
<GeneralBlockTransfer>
  <!--Last block: false-->
  <!--Streaming: true-->
  <!--Window size: 0-->
  <BlockControl Value="64" />
  <BlockNumber Value="1" />
  <BlockNumberAck Value="0" />
  <BlockData Value="0F00863A0F0C07E60913010C3314FF800000020E010E020412002809060008190900FF0F02120000020412002809060008190900FF0F01120000020412000109060000600100FF0F02120000020412000309060100010700FF0F02120000020412000309060100020700FF0F02120000" />
</GeneralBlockTransfer>
</PDU>
</HDLC>
<HDLC len="124" >
<TargetAddress Value="13311" />
<SourceAddress Value="1" />
<!--S frame.-->
<FrameType Value="13" />
<PDU>
<GeneralBlockTransfer>
  <!--Last block: false-->
  <!--Streaming: true-->
  <!--Window size: 0-->
  <BlockControl Value="64" />
  <BlockNumber Value="2" />
  <BlockNumberAck Value="0" />
  <BlockData Value="020412000309060101010800FF0F02120000020412000309060101020800FF0F02120000020412000309060101050800FF0F02120000020412000309060101060800FF0F02120000020412000309060101070800FF0F02120000020412000309060101080800FF0F02120000" />
</GeneralBlockTransfer>
</PDU>
</HDLC>
<HDLC len="137" >
<TargetAddress Value="13311" />
<SourceAddress Value="1" />
<!--S frame.-->
<FrameType Value="13" />
<PDU>
<GeneralBlockTransfer>
  <!--Last block: true-->
  <!--Streaming: true-->
  <!--Window size: 0-->
  <BlockControl Value="192" />
  <BlockNumber Value="3" />
  <BlockNumberAck Value="0" />
    <BlockData Value="0204120003090601001F0700FF0F02120000020412000309060100330700FF0F02120000020412000309060100470700FF0F0212000009060008190900FF0908353635343734313306000000000600000C6006008DD6C40600F2700C06002BFA22060006C007060006846406000F292E12015C1201FC1201DC" />
  </GeneralBlockTransfer>
</PDU>
</HDLC>

here the translation of the 3 PDU block 

<DataNotification>
  <LongInvokeIdAndPriority Value="8796687" />
  <!--2022-09-19 12:51:20-->
  <DateTime Value="07E60913010C3314FF800000" />
  <NotificationBody>
    <DataValue>
      <Structure Qty="14" >
        <Array Qty="14" >
          <Structure Qty="4" >
            <UInt16 Value="40" />
            <!--0.8.25.9.0.255-->
            <OctetString Value="0008190900FF" />
            <Int8 Value="2" />
            <UInt16 Value="0" />
          </Structure>
          <Structure Qty="4" >
            <UInt16 Value="40" />
            <!--0.8.25.9.0.255-->
            <OctetString Value="0008190900FF" />
            <Int8 Value="1" />
            <UInt16 Value="0" />
          </Structure>
          <Structure Qty="4" >
            <UInt16 Value="1" />
            <!--0.0.96.1.0.255-->
            <OctetString Value="0000600100FF" />
            <Int8 Value="2" />
            <UInt16 Value="0" />
          </Structure>
          <Structure Qty="4" >
            <UInt16 Value="3" />
            <!--1.0.1.7.0.255-->
            <OctetString Value="0100010700FF" />
            <Int8 Value="2" />
            <UInt16 Value="0" />
          </Structure>
          <Structure Qty="4" >
            <UInt16 Value="3" />
            <!--1.0.2.7.0.255-->
            <OctetString Value="0100020700FF" />
            <Int8 Value="2" />
            <UInt16 Value="0" />
          </Structure>
          <Structure Qty="4" >
            <UInt16 Value="3" />
            <!--1.1.1.8.0.255-->
            <OctetString Value="0101010800FF" />
            <Int8 Value="2" />
            <UInt16 Value="0" />
          </Structure>
          <Structure Qty="4" >
            <UInt16 Value="3" />
            <!--1.1.2.8.0.255-->
            <OctetString Value="0101020800FF" />
            <Int8 Value="2" />
            <UInt16 Value="0" />
          </Structure>
          <Structure Qty="4" >
            <UInt16 Value="3" />
            <!--1.1.5.8.0.255-->
            <OctetString Value="0101050800FF" />
            <Int8 Value="2" />
            <UInt16 Value="0" />
          </Structure>
          <Structure Qty="4" >
            <UInt16 Value="3" />
            <!--1.1.6.8.0.255-->
            <OctetString Value="0101060800FF" />
            <Int8 Value="2" />
            <UInt16 Value="0" />
          </Structure>
          <Structure Qty="4" >
            <UInt16 Value="3" />
            <!--1.1.7.8.0.255-->
            <OctetString Value="0101070800FF" />
            <Int8 Value="2" />
            <UInt16 Value="0" />
          </Structure>
          <Structure Qty="4" >
            <UInt16 Value="3" />
            <!--1.1.8.8.0.255-->
            <OctetString Value="0101080800FF" />
            <Int8 Value="2" />
            <UInt16 Value="0" />
          </Structure>
          <Structure Qty="4" >
            <UInt16 Value="3" />
            <!--1.0.31.7.0.255-->
            <OctetString Value="01001F0700FF" />
            <Int8 Value="2" />
            <UInt16 Value="0" />
          </Structure>
          <Structure Qty="4" >
            <UInt16 Value="3" />
            <!--1.0.51.7.0.255-->
            <OctetString Value="0100330700FF" />
            <Int8 Value="2" />
            <UInt16 Value="0" />
          </Structure>
          <Structure Qty="4" >
            <UInt16 Value="3" />
            <!--1.0.71.7.0.255-->
            <OctetString Value="0100470700FF" />
            <Int8 Value="2" />
            <UInt16 Value="0" />
          </Structure>
        </Array>
        <!--0.8.25.9.0.255-->
        <OctetString Value="0008190900FF" />
        <!--56547413-->
        <OctetString Value="3536353437343133" />
        <UInt32 Value="0" />
        <UInt32 Value="3168" />
        <UInt32 Value="9295556" />
        <UInt32 Value="15888396" />
        <UInt32 Value="2882082" />
        <UInt32 Value="442375" />
        <UInt32 Value="427108" />
        <UInt32 Value="993582" />
        <UInt16 Value="348" />
        <UInt16 Value="508" />
        <UInt16 Value="476" />
      </Structure>
    </DataValue>
  </NotificationBody>
</DataNotification>

i would be very interested to contribute to the push setup topic as a also believe we need to get the configuration of the data from the meter. let me know if above helps.