DSMR Error Message larger than buffer

[jozsefhabit]

The problem

Hello,

i have just got the SlimmeLezer board, installed using the documentation, but could net get it working. In the debug log i see this error message constantly repeating: [E][dsmr:036]: Error: Message larger than buffer

In the Home Assistant most of the entities have no value (Unknown), some are working, the ones that are not related to P1 port values (sensor.esphome_version_2, sensor.ip_address, sensor.uptime, sensor.wi_fi_bssid, sensor.wi_fi_signal).

Tried to reinstall the board with ESP Home through Home Assistant, no luck, then using the firmware from the zuidwijk.com website. Always got the same results.

Which version of ESPHome has the issue?

v2021.8.2

What type of installation are you using?

Home Assistant Add-on

Which version of Home Assistant has the issue?

2021.9.2

What platform are you using?

ESP8266

Board

d1_mini

Component causing the issue

dsmr

Example YAML snippet

No response

Anything in the logs that might be useful for us?

[E][dsmr:036]: Error: Message larger than buffer

Additional information

No response

[probot-esphome[bot]]

Hey there @glmnet, @zuidwijk, mind taking a look at this issue as it has been labeled with an integration (dsmr) you are listed as a code owner for? Thanks! _{^{(message by CodeOwnersMention)}}

[L44RS]

I have made a reader myself, but have got the same error message. It has worked before with the same hardware setup, just custom MQTT code. Wanted to integrate it into ESP-home but it does not work yet.

[Zolli]

Same happend to me, my meter just got installed and I get same messages.

[L44RS]

Yeah so I worked on this last weekend, together with Maurice from the ESPHome discord. (Thanks again Maurice!). We found out that de debug logs are quite limited for the DSMR component. For me it turned out that I was just getting some nonsense data, and that by some random chance it would find a header (which is just an exclamation mark), after which it could not find the footer and threw an error. The issue for me personally is that the UART is inverted on my specific meter.

As of now, there's no way to invert software (or hardware) serial in ESPHome for the ESP8266. There is however an option for this for the ESP32 platform. Consider using an ESP32, or try to implement an inverted software serial in the UART component. (Which should not be so hard, because in the arduino Software serial library, there is an option to invert the signal.)

[Be-Virtual]

Hi, could it be you're a belgian user ? If yes, integrate this piece of code to use the updated dsmr 0.5 lib that solves this problem.

esphome: platformio_options: lib_deps: glmnet/Dsmr@0.5 lib_ignore: glmnet/Dsmr@0.3

(Don't forget the indent, could not make them visible) Phil

[Zolli]

Iím acutally in Hungary, as a last chance i tried to flas esp-link back to my slimmeLezer, it worked, sort of,.. I actually able to receive data in telnet and the telegram actually arrives but i get some strange data at the end i think. Iím also tried to use homeAssistant dsmr-slimme-reder intgration and i got this error message in my logs when a try to setting it up:

'ascii' codec can't decode byte 0xff in position 1068: ordinal not in range(128)

So it look slike the pythone component that HA is using expecting telegram to be decoded as ascii, but our telegram contains something that is not in the ascii char table.

Referenced issue: https://github.com/ndokter/dsmr_parser/issues/80

Here is an example of the telegram iím receiving:

/AUX59902747755

0-0:1.0.0(210918193600S)
0-0:42.0.0(AUX1030302747755)
0-0:96.1.0(9902747755)
0-0:96.14.0(0002)
0-0:96.50.68(ON)
0-0:17.0.0(90.000*kW)
1-0:1.8.0(000036.705*kWh)
1-0:1.8.1(000018.819*kWh)
1-0:1.8.2(000017.886*kWh)
1-0:1.8.3(000000.000*kWh)
1-0:1.8.4(000000.000*kWh)
1-0:2.8.0(000150.481*kWh)
1-0:2.8.1(000128.819*kWh)
1-0:2.8.2(000021.662*kWh)
1-0:2.8.3(000000.000*kWh)
1-0:2.8.4(000000.000*kWh)
1-0:3.8.0(000000.164*kvarh)
1-0:4.8.0(000042.539*kvarh)
1-0:5.8.0(000000.120*kvarh)
1-0:6.8.0(000000.044*kvarh)
1-0:7.8.0(000015.276*kvarh)
1-0:8.8.0(000027.263*kvarh)
1-0:15.8.0(000187.187*kWh)
1-0:32.7.0(235.3*V)
1-0:52.7.0(235.7*V)
1-0:72.7.0(231.8*V)
1-0:31.7.0(000*A)
1-0:51.7.0(000*A)
1-0:71.7.0(001*A)
1-0:13.7.0(0.762)
1-0:33.7.0(0.714)
1-0:53.7.0(0.675)
1-0:73.7.0(0.807)
1-0:14.7.0(49.97*Hz)
1-0:1.7.0(00.539*kW)
1-0:2.7.0(00.000*kW)
1-0:5.7.0(00.000*kvar)
1-0:6.7.0(00.000*kvar)
1-0:7.7.0(00.000*kvar)
1-0:8.7.0(00.458*kvar)
0-0:98.1.0(210901000000S)(000000.000*kWh)(000000.000*kWh)(000000.000*kWh)(000000.000*kWh)(000000.000*kWh)(000000.000*kWh)(000000.000*kvarh)(000000.000*kvarh)(000000.000*kvarh)(000000.000*kvarh)(000000.000*kvarh)(000000.000*kvarh)(000000.000*kWh)(00.000*kW)(00.000*kW)(00.000*kW)(00.000*kW)(00.000*kW)(00.000*kW)
0-0:96.13.0(▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒)
!C610

[henri98]

I'm facing this issue too, tried both esp8266 and esp32 boards.

[E][dsmr:036]: Error: Message larger than buffer

Smartmeter: ISKRA AM550 (DSMR 5.0) (The Netherlands)

Any ideas?

I was able to solve it by inverting the rx signal.

[Bubi504]

Is there any chance to solve this ? This error makes metering impossible. Smartmeter : Sanxing SX631, the problematic OBIS is 0:96.13.0.

[mmakaay]

Can you put your ESPHome configuration that you used for the meter here, so we can check if that one is correct?

One thing that you also saw is that there some strange data in there for 0:96.13.0.. I don't know if that is actually being sent by the meter, or that this is an issue with the ESP8266 software. It might be the meter, because it looks like it is actively masking data between (......) there. Really peculiar. I'm not sure if it would be a problem though. Maybe the DSMR parser would trip on it, but currently we don't get to the parser at all.

All in all, the telegram that you pasted above looks pretty good to me. So I checked its size and it is almost 3000 bytes long. I think that is the main issue here. The hard-coded max telegram size is 1500 bytes. So about halfway reading the telegram, the reader stops with the error message that you see.

I will implement a way to configure this max telegram size from the configuration. Once we get the reading part fixed, we'll see how the DSMR parser handles the telegram. I'll let you know when it can be tested.

[Zolli]

I think the data is fine, it parses using the DSMR4/5 standard correctly using the DSMRReader project it working fine but need the CRC check disabled. I think this 0:96.13.0 OBIS code contains a message that the provider sends to the meter using SMS, and this property contains the last SMS message body. In the hungarian language we have a few character that not in the ASCII character set.

[mmakaay]

But the telegram doesn't contain a variety of characters in there. In the paste it looks like a repeated character. Maybe that's just a representation issue with the pasted data. It would be interesting to use my UART debugger from within ESPHome. That one would log non-ASCII as escaped hex codes, so then we'd be sure about what's in there and how CRC issues might be fixed. It never feels right to me to disable CRC.

[Bubi504]

I've disabled Esphome integration... the gateway is a Slimmelezer, but reflashed with the 'esplink' firmware (to get the data with telnet, with dsmr component). The esplink also has a built-in webserver which shows raw data: :

[Bubi504]

and with putty client, UTF8 codepage

another putty, ISO8859-1

[mmakaay]

Ah okay, that looks like the terminal is expecting UTF-8 characters, but the data in there is not ASCII, but also not UTF-8. Maybe the output will be better when setting the terminal to ISO-8859-2 or ISO-8859-16 then.

Apart from that, I wrote a change for EPSHome to allow for a variable max telegram size. By including that in ESPHome as an external component for now (for testing), we can try if ESPHome works when setting the max size to for example 3000 bytes. Can you try the device with ESPHome and the following code in it?

Note: this test branch also contains all my other improvements for DSMR so far

external_components:
  - source:
      type: git
      url: https://github.com/mmakaay/esphome
      ref: dsmr-add-max-telegram-length
    components: [ "dsmr" ]
    refresh: 60s

dsmr:
  max_telegram_length: 3000

That should get us a step closer to a working device. CRC would be the next thing to tackle. If you get CRC errors, disabling the CRC check in the dsmr: config might already give some working readings. If that happens, I'd really like to setup the UART debugger, so we can investigate the data an possibly fix the error cause.

[Bubi504]

You helped a lot :-)))

[mmakaay]

Very cool. I will submit this change as a PR. I'll do that as soon as another DSMR-related PR of mine get merged, because this change was built on top of that one.

[mmakaay]

Now there's the CRC issue left. Disabling CRC checks is a work-around, but it would be neat if we could get it working correctly. For that, I need to know the exact data that are being sent. For that, I wrote a UART debugger, which is currently still a PR. But you can integrate it, to make the device log the information.

Can you capture a few datagrams with the following config updates?

First add an additional external component that grabs the updated UART code from my branch:

external_components:
  - source:
      type: git
      url: https://github.com/mmakaay/esphome
      ref: dsmr-add-max-telegram-length
    components: [ "dsmr" ]
    refresh: 60s
  - source:
      type: git
      url: https://github.com/mmakaay/esphome
      ref: uart-debugging-external-component
    components: [ "uart" ]
    refresh: 60s

Additionally, you'll have to extend the UART component config with a few debug options:

uart:
  debug:
    direction: RX
    after:
      delimiter: "\r\n"
    sequence:
      - lambda: UARTDebug::log_string(direction, bytes);

Using these changes, the traffic on the UART bus will be logged. The weird non-ASCII bits will be properly escaped, so we will be able to see what is going on exactly there.

[Bubi504]

Installed firmware with this modified config, but there are crc errors - and no data:

where is the UART log ?

[Bubi504]

I'm sorry, the crc check was on :-) Turned off, here's the log:

[07:28:56][I][app:101]: Project zuidwijk.slimmelezer version 1.0 [07:28:57][E][dsmr:173]: ^ OBIS id Empty

               ^

Trailing characters on data line

^ OBIS id Empty

           ^

Invalid number

^ OBIS id Empty

[mmakaay]

Oh, the UART debug log is logged just like all other messages. But it looks like your log level is currently too low to see them :-) Please set

logger:
  baud_rate: 0
  level: DEBUG

to make them visible.

[Bubi504]

Now it's verbose :-) see attached. dmr_log.txt

[mmakaay]

Wait, what am I looking at here? This is not the debug output from the ESPHome dsmr / uart components. This looks like the dsmr integration in Home Assistant instead.

[Bubi504]

That's why had I asked, where to look for :-) Should I copy from the device's webserver screen, from here ?

[mmakaay]

Yes, that is the output that I was looking for! If you can copy / paste what is now in view into a file, then I have something to play with.

What we do see here already, is that there is literally nothing else than 0xff byte values within that last line. It's not misrepresented characters or so, but actual useless garbage :-p This means that there will be no useful data coming from that line, but I will work on the data to see if I can find out how the checksum is constructed for this telegram style.

[Bubi504]

Here you are dmr_log2.txt .

[mmakaay]

Too bad, I wasn't able to find a working CRC computation. So for now you'll have to work with the CRC check disabled.

One comment that I found elsewhere said: "I also have a CRC-error (on RPI4 too) but It worked just fine for 6 months. I assume a faulty update of the DSMR firmware by the provider." If it worked before, but degraded over time, then it does sound logical that a provider firmware update broke the functionality. Maybe inform with your provider about this. If there's something that can be changed in the code to fix the CRC, then please let me know. Otherwise, we'll have to leave it as-is.

[mmakaay]

Heads up for people that have included my patched versions of dsmr and uart:

• The UART debugger is included in ESPHome 2021.11.0 and up
• The max telegram length option will be in ESPHome 2021.12.0 and up Accordingly, you can remove the related external_components when using those new versions of ESPhome.

[LarsJnsn]

Sorry to open this up again but I tried the external component for "dsmr-add-max-telegram-length" but I got the error that that branch doesnt exist. Got any tips or fixes?

external_components:
   -source:
      type: git
      url: https://github.com/mmakaay/esphome
      ref: dsmr-add-max-telegram-length
    components: [ "dsmr" ]
    refresh: 60s

With error: "Remote branch dsmr-add-max-telegram-length not found in upstream origin."

[mmakaay]

The reason is that the code has been merged into the dev tree. I removed my personal branch with this change, since it is no longer needed. The new code will be included in the next version of ESPHome (2021.12.0).

In the meanwhile, you can either use ESPHome dev (not recommended) instead of stable, or you can specifically grab the new dsmr code from the ESPHome dev branch. Something like this should work:

external_components:
   -source:
      type: git
      url: https://github.com/esphome/esphome
      ref: dev
    components: [ "dsmr" ]
    refresh: 60s

[kukacwap]

Did this ever get resolved? My ELMŰ/E.ON Hungary meter does the same with slimmelezzer+. My meter is Sanxing SX601 (S12U16) running latest firmware on esphome 2021.12

[Bubi504]

@kukacwap Yes, it's working perfectly (Hungary, E-On, SX631 meter). Two modifications needed. The first begins with 'external_components' like mmakay commented on Nov19. The second is to turn off crc check and set max length in dsmr section:

dsmr: id: dsmr_instance crc_check: false max_telegram_length: 3000

[kukacwap]

@Bubi504 Thanks! And just to clarify when we talk about config is that something I upload in slimmelezer interface OTA in its entirety (after removing gas and other irrelevant items) or do I need to flash it?

[Bubi504]

With the built-in OTA update function, you have to upload a .bin file (which is recompiled with the modified yaml config). Instead of this, I recommend you to use esphome addon (in homeassistant): https://esphome.io/guides/getting_started_hassio.html With this, you can update the firmware and edit the config in a single click, there's no need to use the slimmelezer device's built-in web interface at all - and it updates the firmware code from github when it changes.

[mmakaay]

The first begins with 'external_components' like mmakay commented on Nov19.

Note that the external component is no longer needed. The required changes are in ESPHome 2021.12.0 and later.

[kukacwap]

Thanks guys. worked perfectly. Only thing I had to add was: dsmr: id: dsmr_instance crc_check: false max_telegram_length: 3000

suggested by @Bubi504 external components part is really not needed anymore as suggested by @mmakaay.

[ocsele]

Guys, I would appreciate a little guidance, I'm also from Hungary suffering from same telegram is larger than 1500 bytes error. It seems clear to me what I should do (changing buffer to 3000 and turning off CRC), but I can't seem to find a way to do it :) I'm on core-2021.12.10 and supervisor-2021.12.2 which has a tighter integration with ESP Home. Adding my Slimmeelezer+ was a breeze, but I can't find a way to edit the configuration of Slimmelezer+: when I try to add ESPHome Dashboard as an official HA add-on, I can't, simply because it's non-existent :) What do I miss? How can I extend the configuration of my Slimmelezer with the neccessary config you guys recommended? Thanks a lot in advance!!!

[zuidwijk]

@ocsele do you have baud_rate: 0 under the section logger:? If not, chances are high it will run on Software Serial instead of Hardware Serial and that causes (in my case) to trigger those errors

[ocsele]

Dear Marcel, where should I check what you just mentioned? All I did was plugged in the device, enter wifi credentials, and add as an ESP Home integration, via its local IP address, worked like a charm. But I don't know how to edit its own yaml.

[ocsele]

if you've meant HA's own configuration yaml, then there's no such thing for me under logger: logger: default: warning

[zuidwijk]

If you're using my device (SlimmeLezer) than you can download the latest firmware here and upload that via the webinterface (OTA).

I do recommend to use your own code. Install the ESPHome add-on and make there your own config. If you look at the ESPHome config (not Home Assistant, only for the device) you see a line logger:. If that's empty, it uses its default which is 115200, and that gives issues. Look at the example.

[ocsele]

FINALLY, I managed to install ESPhome addon, so I can edit the device yaml, so I'm gonna check all these recommendations to see if it helps, and report back. Marcel, just one question - a very noob, indeed - left: when add my wifi ssid and password, the excalmation mark should stay in front of it, or not? I'm afraid to upload the new firmware remotely from my workplace without this knowledge :)

so is it: wifi: networks: []

• ssid: MyOwnWifiSSID password: MyOwnPassword

or is it: wifi: networks: []

• ssid: !MyOwnWifiSSID password: !MyOwnPassword

EDIT: I also got a parsing error when trying to validate, what am I missing?

`ERROR Error while reading config: Invalid YAML syntax:

while parsing a block mapping in "/config/esphome/slimmelezer.yaml", line 31, column 3: networks: [] ^ expected , but found '' in "/config/esphome/slimmelezer.yaml", line 32, column 5:

• ssid: MyOwnWifiSSID ^`

[mmakaay]

With the exclamation mark, you probably are probably referring to something like:

password: !secret wifi_password

In that case, !secret is a special indicator, that tells ESPHome that the password should be read from the secrets.yaml file. That way you don't have to put the password directly in the YAML file and you can reuse it for other devices.

If you are placing your password directly in the YAML configuration, you should only write down the password, literally. No exclamation mark needed there. If you do, then you'll tell ESPHome that the password actually start with !, which is not the case. So you simply need something like:

password: MyOwnPassword

[ocsele]

Thank you very much for the swift help! In the meantime I ran into a yaml validation problem as well :(

ERROR Error while reading config: Invalid YAML syntax:

while parsing a block mapping
  in "/config/esphome/slimmelezer.yaml", line 31, column 3:
      networks: []
      ^
expected <block end>, but found '<block sequence start>'
  in "/config/esphome/slimmelezer.yaml", line 32, column 5:
        - ssid: MyOwnWiFiSSID
        ^

[michaelarnauts]

Can you post the whole block (remove your passwords)? There is probably some indenting issue.

[ocsele]

Can you post the whole block (remove your passwords)? There is probably some indenting issue.

yes thank you, in the meantime I managed to pass it through validation, hopefully in the correct way.

this is the version which gave validation erro:

wifi:
  networks: []
    - ssid: wifiname
      password: password

and this is the one which passed validation:

wifi:
  networks: 
    - ssid: wifiname
      password: password

[ocsele]

@ocsele do you have baud_rate: 0 under the section logger:? If not, chances are high it will run on Software Serial instead of Hardware Serial and that causes (in my case) to trigger those errors

@zuidwijk: yes, coming from your own yaml example, the baud rate is set to 0 under logger, and improvserial is also there, this gives me an error now:

Failed config

improv_serial: [source /config/esphome/slimmelezer.yaml:39]

  improv_serial requires the logger baud_rate to be not 0.
  {}

Which one should I remove, improvserial, or baud rate 0? If I understand correctly, I can't have both. Many thanks for your input!

[zuidwijk]

@ocsele do you have baud_rate: 0 under the section logger:? If not, chances are high it will run on Software Serial instead of Hardware Serial and that causes (in my case) to trigger those errors

@zuidwijk: yes, coming from your own yaml example, the baud rate is set to 0 under logger, and improvserial is also there, this gives me an error now:

Failed config

improv_serial: [source /config/esphome/slimmelezer.yaml:39]

  improv_serial requires the logger baud_rate to be not 0.
  {}

Which one should I remove, improvserial, or baud rate 0? If I understand correctly, I can't have both. Many thanks for your input!

See the issue i've made: https://github.com/esphome/issues/issues/2873

You can't use Improv together with DSMR (not at this moment)

[ocsele]

okay, finally have some achievement, some data started to flow immediately :)

@zuidwijk : Marcel, if you don't mind, I post a couple of questions on your git page on how to read and interpret the data, if you don't mind.

THANKS FOR EVERYONE FOR HELPING OUT A NOOB!

[zuidwijk]

Looks like your DSMR version is 2.2. That version doesnt have those data.

[ocsele]

you are referring to Unknown state of working sensors, or the not working sensors with entitiy not available marks? Because the not available entities were unfortunately cuased by me by translating their names directly in the ESP yaml, when I went back to your original names, they started to work instantly. Now this is the data set coming through, I assume I can safely delete from the yaml those sensors which give Unknown states in HA: