pulse reader fails and dies if i try to get values every second

[disaster123]

Hi,

if i set the cron to:

TIBBER_PULSE_CRON=* * * * * *

i'm getting the following errors and no updates to mqtt. If i set the cron to:

TIBBER_PULSE_CRON=*/15 * * * * *

everything is fine.

Log output:

2023-06-19 09:14:44.118  WARN 1 --- [   scheduling-1] d.w.t.source.TibberPulseHttpReader       : Unable to parse SML from response

java.io.IOException: Invalid SML payload: 
    at de.wyraz.tibberpulse.sml.SMLDecoder.extractMessage(SMLDecoder.java:279) ~[classes!/:1.0.0-SNAPSHOT]
    at de.wyraz.tibberpulse.sml.SMLDecoder.decode(SMLDecoder.java:51) ~[classes!/:1.0.0-SNAPSHOT]
    at de.wyraz.tibberpulse.source.TibberPulseHttpReader.fetch(TibberPulseHttpReader.java:86) ~[classes!/:1.0.0-SNAPSHOT]
    at de.wyraz.tibberpulse.source.TibberPulseHttpReader.fetch(TibberPulseHttpReader.java:56) ~[classes!/:1.0.0-SNAPSHOT]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78) ~[na:na]
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
    at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na]
    at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) ~[spring-context-5.3.20.jar!/:5.3.20]
    at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.3.20.jar!/:5.3.20]
    at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:95) ~[spring-context-5.3.20.jar!/:5.3.20]
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
    at java.base/java.lang.Thread.run(Thread.java:831) ~[na:na]

2023-06-19 09:14:46.073  WARN 1 --- [   scheduling-1] d.w.t.source.TibberPulseHttpReader       : Unable to parse SML from response

java.io.IOException: Invalid SML payload: 
    at de.wyraz.tibberpulse.sml.SMLDecoder.extractMessage(SMLDecoder.java:279) ~[classes!/:1.0.0-SNAPSHOT]
    at de.wyraz.tibberpulse.sml.SMLDecoder.decode(SMLDecoder.java:51) ~[classes!/:1.0.0-SNAPSHOT]
    at de.wyraz.tibberpulse.source.TibberPulseHttpReader.fetch(TibberPulseHttpReader.java:86) ~[classes!/:1.0.0-SNAPSHOT]
    at de.wyraz.tibberpulse.source.TibberPulseHttpReader.fetch(TibberPulseHttpReader.java:56) ~[classes!/:1.0.0-SNAPSHOT]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78) ~[na:na]
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
    at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na]
    at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) ~[spring-context-5.3.20.jar!/:5.3.20]
    at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.3.20.jar!/:5.3.20]
    at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:95) ~[spring-context-5.3.20.jar!/:5.3.20]
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
    at java.base/java.lang.Thread.run(Thread.java:831) ~[na:na]

2023-06-19 09:14:48.118  WARN 1 --- [   scheduling-1] d.w.t.source.TibberPulseHttpReader       : Unable to parse SML from response

java.io.IOException: Invalid SML payload: 
    at de.wyraz.tibberpulse.sml.SMLDecoder.extractMessage(SMLDecoder.java:279) ~[classes!/:1.0.0-SNAPSHOT]
    at de.wyraz.tibberpulse.sml.SMLDecoder.decode(SMLDecoder.java:51) ~[classes!/:1.0.0-SNAPSHOT]
    at de.wyraz.tibberpulse.source.TibberPulseHttpReader.fetch(TibberPulseHttpReader.java:86) ~[classes!/:1.0.0-SNAPSHOT]
    at de.wyraz.tibberpulse.source.TibberPulseHttpReader.fetch(TibberPulseHttpReader.java:56) ~[classes!/:1.0.0-SNAPSHOT]
    at jdk.internal.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) ~[na:na]
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
    at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na]
    at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) ~[spring-context-5.3.20.jar!/:5.3.20]
    at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.3.20.jar!/:5.3.20]
    at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:95) ~[spring-context-5.3.20.jar!/:5.3.20]
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
    at java.base/java.lang.Thread.run(Thread.java:831) ~[na:na]

[SpiGAndromeda]

Maybe the pulse itself can't handle that. You should turn on the debug mode to compare which data is transmitted if the data is drawn every 15 seconds and if it's drawn every second.

[micw]

That's also my guess. From your log, it looks like every 2nd result is empty. You could try it with curl in a loop to see if result is the same. If so, use */2 as trigger.

[disaster123]

the following works fine:

while true; do time wget --user admin --password 'XXX-XXX' -O /dev/null 'http://tibber-bridge.fritz.box/data.json?node_id=1'; sleep 1; done

i always get some kind of binary data

[disaster123]

Ok it‘s not the interval - I generally have seconds or minutes where data is blank… / empty.

[micw]

Strange. Can you increase the log level to trace to check if the http lib gives any errors?

[disaster123]

mhm ok the http request is sometimes just empty only a blank page with 0 bytes... also with wget.

[micw]

A Tibber tech guy told me that they disable the webserver by default because the limited RAM of the ESP32. That also may apply if you query to frequently. Another cause could be that if you query while new data is received, the old data may not be available.

[micw]

I could simply ignore empty results or log it at debug level only so that you log is not spammed too much.

[micw]

I have addressed the logging with the commit above. Please let me know if this removes the errors from log. Unfortunately it will not solve our issue since it's on side of the pulse, not of this tool. To get more frequent readings, you could:

• intercept the pulse's MQTT traffic
• use a hardware like I created (see https://github.com/evcc-io/evcc/discussions/7070, in german) to dupplicate the physical signal from your meter

[disaster123]

thanks. Intercepting MQTT would be an option but isn't that straight forward... regarding all the certificate stuff... not sure if i would like to got that way.

The hardware device looks interesting - but i would only buy it if it is completely assembled...

Thanks a lot for your help!