Belgium DSMR does not parse correctly

rvdbreemen commented 4 years ago

Hi Matthijs, I have seen reports that belgium smartmeters do not parse correctly using the 5.0.2 dsmr parsing. This is due to the extention made by the Belgium netbeheerder.

The document for Belgium can be found here

I will be looking into making changes to the library to enable parsing of the DSMR of Belgium. Robert

matthijskooijman commented 4 years ago

Interesting! Would be nice to also support the Belgian standard, looking forward to see what you come up with :-)

rvdbreemen commented 4 years ago

Found the differences:

At least the OBIS numbers differ, from Belgium to the Dutch standards. Minimal difference, but it will confuse the parsing. Version number P1 and gas_delivered.

How to tackle this best I wonder?

matthijskooijman commented 4 years ago

I haven't looked in detail (bit busy atm), but if it is just the OBIS numbers and the associated formats that are different while the telegram format itself is the same, this might be a matter of defining some extra fields for them? Using a namespace belgium, you could end up with e.g. a field belgium::gas_delivered rather than just gas_delivered in your struct definition?

rvdbreemen commented 4 years ago

Yes, so far it really looks like just some extra OBIS codes and fiels. In belgium they have gas meters that are not temperature corrected, so they legally needed to use different OBIS codes it seems.

rvdbreemen commented 4 years ago

My first attempt at fixing this for Belgium users is to define the OBIS codes. And get some help getting it validated. Before I try to understand how to correctly extend using your suggestion of namespaces in the code.

rvdbreemen commented 4 years ago

@matthijskooijman we have a working field.h for belgium (confirmed by belgium user). I can share the fields.h (for belgium situation). I looked at your library, but I am wondering how best to integrate a second fields.h with an alternative for the OBIS parsing.

I realize you are busy, but it would be great to update your library with the belgium dsmr specification fields. I attached the fields.h in the ZIP in this comment.

fields.zip

rvdbreemen commented 4 years ago

Suggested a PR, added namespace for belgium in fields.h (see pull request #29 )

matthijskooijman commented 4 years ago

Let's keep this issue open until it is actually solved.

I read a bit through the Belgian specification, still playing around with it a bit to figure out the best way to support this. It seems that almost all fields are identical, just that the belgian spec adds some fields, so I think we can support this without duplicating all fields.

Anyway, what I really wanted to ask: @rvdbreemen, @pdemeyer, does either of you have some example telegrams for testing? I looked at the e-MUCS-H spec, but there are none. In #29 there is a voorbeelden.txt, but it is empty?

matthijskooijman commented 4 years ago

I've looked closely at the belgian spec, and wrote a bit to put into the README later (added at the bottom of this command). As for implementing this, I'm just thinking out load here.

Overall, it is just a matter of adding extra fields.

Gas value

The two different gas meter fields have different ids, so that would result in two different fields. However, you would probably want to refer to them under a single name. This could be be implemented by defining both fields with the same name in different namespaces (e.g. use gas_delivered for the default DSMR versoin and emucs::gas_delivered to include the e-MUCS version in your parsed telegram definition).

But then you need to choose which one to use at compiletime, you cannot put both of them in the same struct if they have the same name. We could then decide to define them separately with different names (e.g. gas_delivered_corrected and gas_delivered_not_corrected), but then when you need the gas value, you would need to check both fields to see which is present (which would work, but could be a bit cumbersome).

One alternative (not sure how to implement that yet, though) would be to allow specifying multiple Obis IDs to a single gas_delivered field definition. Then if either of the IDS is found, the associated value is stored in this field. If both are present (should never happen), this would be handled as a duplicate field with an error.

You could then still also offer two distinct fields (so you can choose at compile time) in case you want to parse one but not the other, or both and know which one was found.

I just realized (looking at the DSMRloggerAPI code with DSMR-3 support) that the gas_delivered value in DSMR-3 actually uses a different ID than the wone in DSMR-4. Looking at the comments in my code, I previously only noticed that the description is different (5-minute vs hourly value), but the ID is also different. This means that everything that applies to the eMUCS gas value above, also applies to the DSMR-3 gas value equally.

Limiter threshold

This field was originally specified in DSMR3, then removed in DSMR4 and now re-added in e-MUCS. However, DSMR3 specified a unit of A, while eMUCS specifies a unit of kW. DSMR as it is now strictly checks the unit in the telegram and fails to parse when the unit is different from what was expected. Again, this could be solved by splitting it into two fields with the same ID. You then have to choose either one at compiletime, and cannot support DSMR3 and eMUCS with the same struct. Since this is a bit of a specific field, that might not be relevant, but it would be nicer to support this, but I'm not sure how yet.

matthijskooijman commented 4 years ago

Oh, here's the README addition I mentioned above:

Differences between Dutch and Belgian meters

e-MUCS H spec v1.3, based on DSMR 5.0.2 P1, except:

Telegram version number has a different format, so uses a different
ID.
Adds additional M-bus device ID field, apparently because the first
field is apparently supposed to onlly contain the "fabrication
number" (e.g. I think that means serial number, but not model number
or version numbers). than 96 characters to identify themselves).
Adds gas measurement without temperature correction. ID is different,
values should be encoded the same.
Re-adds gas valve position and breaker/switch position (removed in
DSMR4). IDs are identical to DSMR3, but DSMR3 defines values as
"on/off/released" and e-MUCS defines "Disconnected/Connected/Ready for
reconnection".
Re-adds limiter threshold field (removed in DSMR4). Again, IDs are
default, but DSMR3 defined this in Ampere, while e-MUCS specifies kW.
Adds "Fuse supervision threshold (L1)" field this field is specified
in Ampere.
Re-adds a "consumer message code" field (removed in DSMR5). The ID is
the same, but the max length is increased from 8 numeric digits to
128 characters, and where DSMR4 talks about displaying the message on
the display, e-MUCS only talks about sending messages to home
automation or energy managment (CEMS) systems attached to P1.

Some of these changes (gas reading and mbus device ids) refer to the
"IDIS pack2" specification, implying that the DSMR specification
actually uses some values wrongly (but the differences seem to be minor,
and should not influence the parsing of fields, though maybe the
interpretation of them).

matthijskooijman commented 4 years ago

I also wonder about the updated "version information" field. AFAICS, the eMUCS H spec has no info at all about what valus that field should have (unlike DSMR, which specifies what version number should appear in that field). Anyone see any documentation or examples of this field?

rvdbreemen commented 4 years ago

@matthijskooijman just added some examples in the voorbeelden.txt of telegrams, two meters.

rvdbreemen commented 4 years ago

I read through your comments. And you are coming to most of the conclusions I came too. I made it work using a doubling of the field definitions and made a switch to implement and get some field testing going. Users of Willem's hardware from Belgium now used it twice.

As you see in the code the difference are very limited. Yes, there is some specification difference as you point out. Like the protocol version ID is suddenly 5 digits instead of 2.

Like Willem, I think it would be useful to enable the library to parse more dynamically. When setting up the parser you should able switching the MBUS devices around from default. And also the protocol version.

To limit space usage you could opt for a compile time switch, but I think since the dfference are so very small a more dynamic approach would be better.

rvdbreemen commented 4 years ago

On the comment to parse the fields to either gas_delivered and gas_delivered_not_corrected is a valid discussion. Uncorrected gas measurements are used in Belgium. Normally in the Netherlands, we do correct for temperature, but it seems to be more explicit in Belgium. That's why I did not bother to make a new field in my version. I later did notice the difference, functionally there is no difference, but it would be good to have field alternatives.

However, for real-world implementations like Willem's DSMRlogger, you could say it complicates the situation where it is really not needed. So yes, it would be good to reflect the spec more correctly, but let's keep in mind too that the fields are used in real-world situations too.

matthijskooijman commented 4 years ago

Thanks for the examples. Looking at the version number:

0-0:96.1.4(50213)

That probably means DSMR 5.0.2, e-MUCS H 1.3

Also, the limiter threshold is actually in kW, as documented:

0-0:17.0.0(999.9*kW)

And in DSMR-3, it is actually in A as well (from #22):

0-0:17.0.0(999*A)

rvdbreemen commented 4 years ago

Yes, true, I think you are right. V5.0.2-1.3. That does make sense.

So it differs from version to version... that's simple to cover, using different field specfications per protocol versions.

hemeleerse commented 4 years ago

Hello, here's a Belgian user trying to contribute :-) I found the eMUCs P1 version 1.4 that answers some of the questions raised (e.g. version information)

e-MUCS_P1_Ed_1_4.pdf

hemeleerse commented 4 years ago

The parser gives an 'unknown field' error on this part of the data : 1-0:31.4.0(999*A) I tried to add this to fields.h and fields.cpp but still get the same error. Would appreciate some help as I'm not an expert in these matters.

hemeleerse commented 4 years ago

Here's an example of P1 output: BE-Fluvius-slimme-meter.txt

erhapp commented 4 years ago

Here's an example of P1 output: BE-Fluvius-slimme-meter.txt

Please forgive me my off topic question.

@hemeleerse I started looking into the workings of my own Belgium meter yesterday and I'm wondering about the circuit you are using to read the P1 values. Are you willing to share your circuit?

hemeleerse commented 4 years ago

@erhapp this article will help you : https://opencircuit.be/Blog/Slimme-meter-uitlezer

It’s based one the ESP-01 module with ESP8266 processor

rvdbreemen commented 4 years ago

@erhapp or you could buy the DSMR logger print: https://opencircuit.be/Product/Slimme-meter-uitlezer-V4.5-Geassembleerd

keukenrol commented 1 year ago

I've just tried the example "read.ino" on a teensy 2.0. As output it only seems to parse missing items, no actual data is shown. Is this linked to the standard not being supported? The condition if (reader.parse(&data, &err)) never gets met. I tried this by blinking a LED if it matches this condition. If i directly use serial.print i get version number 50215 (Sagecom T211 Belgium)

karelv commented 9 months ago

Any update for Belgium?

matthijskooijman commented 9 months ago

No progress on my part, I haven't found any time to work on this project yet...

karelv commented 8 months ago

Thanks for your heads up!

FYI: Mr Wheel (Willem Aandewiel) created: https://github.com/mrWheel/arduino-dsmr-be

However it did not compile on Teensy 4.1: https://github.com/mrWheel/arduino-dsmr-be/issues/5

matthijskooijman / arduino-dsmr