search

UtilitechAS / amsreader-firmware

ESP8266 and ESP32 compatible firmware to read, interpret and publish data to MQTT from smart electrical meters, both DLMS and DSMR is supported
Other
390 stars 73 forks source link

Need help with testing of new firmware on multiple meter models #143

Closed gskjold closed 2 years ago

gskjold commented 3 years ago

I am currently writing new code to interpret data from the meter. The new firmware should support any IEC62056-7-5 (Norwegian and Danish meters with HDLC payload) and IEC62056-21 (Swedish and possibly Dutch meters with RJ12 connector). To confirm that it actually works, I need to have it tested on multiple brands in all possible countries.

A list to start with, I will expand this as new meters are tested:

Norway

Denmark

Sweden

Any RJ12 meter, please let me know your setup

NOTES

ams2mqtt-esp32-2.0-rc1.zip ams2mqtt-esp8266-2.0-rc1.zip

ams2mqtt-esp32-2.0-rc2.zip ams2mqtt-esp8266-2.0-rc2.zip

ams2mqtt-esp32-2.0-rc3.zip ams2mqtt-esp8266-2.0-rc3.zip

ams2mqtt-esp32-2.0-rc4.zip ams2mqtt-esp8266-2.0-rc4.zip

ams2mqtt-esp32-2.0-rc5.zip ams2mqtt-esp8266-2.0-rc5.zip

ams2mqtt-esp32-2.0-rc6.zip ams2mqtt-esp8266-2.0-rc6.zip

ams2mqtt-esp32-2.0-rc6.1.zip ams2mqtt-esp8266-2.0-rc6.1.zip

ArnieO commented 3 years ago

Got a treat for you tonight

Pre-Christmas candy! šŸŽ…

madsbg commented 3 years ago

I like the improvements to the firmware upload experience šŸ‘ I also like the usage graphs. Maybe you could also add a measurement unit? usage_graph_unit

Could you please send the package timestamp with every MQTT update like the 1.5.x versions? Thanks šŸ˜ƒ

gskjold commented 3 years ago

A type-o have caused package timestamp to be missing, will fix it :)

Edit: I was wrong, it was actually because I forgot to implement it for encrypted meters.. :D

gskjold commented 3 years ago

A couple of questions:

  1. Looking at the graph for daily usage, what is most intuitive; 00-23 or 01-24 ? Right now you will see that it shows use for previous hour. Right now at 1300 CET the graph ends at 12 which has the value for used kWh between 1200-1300.

  2. In meter configuration I have removed the dropdown for choosing meter model. The main reason for this is that I hope the coming versions will support meters from many countries and the number of meters and associated configuration in some countries are staggering. But the baud/parity/invert configuration is also very alien for most people, so what would be the best way to solve this?

madsbg commented 3 years ago

  1. To me 00-23 is the most logical. Now the time is 10:05 and the graph ends at 9. I would expect it to end at 10. The same with the monthly graph. It should end at 29 and not 28 because today is 29. Or am I misunderstanding?

  2. I'm not sure what you mean? Do you want the hide the settings or maybe separate them in another section? meter_config

EDIT: Changed my mind regarding 1. Several times šŸ˜ƒ

ArnieO commented 3 years ago

EDIT: Changed my mind regarding 1. šŸ˜ƒ

Hehe, this is exactly why the question is raised! At first glance it looks counter intuitive that the data point created at 10:00 is labeled 9. But that data point is for the time period where the clock is 9:xx, so I think it could be seen as the most "correct" way to do it.

madsbg commented 3 years ago

I think a data point should hold the data form the previous time period.

ArnieO commented 3 years ago

The same with the monthly graph. It should end at 29 and not 28 because today is 29. Or am I misunderstanding?

I disagree, as the data point "28" is the consumption yesterday (28th Nov). One data point is created each midnight. What you mean is maybe that there should be a new point 29 already, that increases during the day today?

madsbg commented 3 years ago

If there is only one data point that holds the accumulated value for the entire time period, then I think bar charts would be more appropriate.

gskjold commented 3 years ago

I hear what you are saying about the bar chart. Here is an example of that with relevant data: screenshot

Edit: uploaded wrong file

ArnieO commented 3 years ago

Yeah... I think this is better! šŸ‘ And I would like to see the measurement values as you did for voltage and current ("kWh" not needed, it is already on the axis). And why not repeat the purple color you use in the menus?

madsbg commented 3 years ago

I hear what you are saying about the bar chart. Here is an example of that with relevant data:

I like it. That looks really good.

gskjold commented 3 years ago

Ooh I like the color idea! screenshot

gskjold commented 3 years ago

It doesn't scale that well on mobile, but works perfectly fine for tablet.

ArnieO commented 3 years ago

Ooh I like the color idea!

In my opinion, that turned out really great!!

gskjold commented 3 years ago

Updated first post. New version containing:

gskjold commented 3 years ago
  1. I'm not sure what you mean? Do you want the hide the settings or maybe separate them in another section? meter_config

Regarding this point, I think the configuration is too advanced for most users. Right now I am thinking of taking back the old dropdown, but add "Custom" as an option where baud/parity/invert will show.

Still open for suggestions here

ArnieO commented 3 years ago

Right now I am thinking of taking back the old dropdown, but add "Custom" as an option where baud/parity/invert will show. Still open for suggestions here

That dropdown could become long and not necessarily easier to navigate, as this could become a multi dimensional problem:

The dropdown would need to be updated each time a new combination is defined.

Suggestion / idea: Instead of complexifying the firmware, maybe we could take an other approach: Make a table (or some other construction) on this Github repo where user can find correct bitrate/parity for hers/his installation. Should be much easier to maintain than doing the same in firmware. Place a link to that lookup-page from the device config page (open in new browser tab).

gskjold commented 3 years ago

Good point, that list would get big pretty fast with all these factors... So far your suggestion sounds like the most realistic approach.

madsbg commented 3 years ago

Thank you for all the nice improvements to the firmware.

I'm sorry to keep complaining about the timestamps. The package timestamp is now updating every 10 sec. but it seems to be one hour ahead except for the hourly update where the meter clock is also updated.

gskjold commented 3 years ago

Bring on the complaints! That's the only way I am able to test this on foreign meters :) I will look into this, I think I know what is going on.

ArnieO commented 3 years ago

image

ArnieO commented 3 years ago

The two boxes move apart when the screen with changes. Suggestion: image

gskjold commented 3 years ago

I know... I have some trouble adapting the grid layout when production graph is enabled. I am still thinking about that one..

ArnieO commented 3 years ago

I know... I have some trouble adapting the grid layout when production graph is enabled. I am still thinking about that one..

If it simplifies: Move both to the very bottom, below the graphs. I think reactive power and energy is of interest to less than 1% of the users. For the remaining 99% it is only a potential source of confusion.

gskjold commented 3 years ago

New update:

Isaksson commented 3 years ago

Have you planned to implement a solution for my Swedish Aidon RJ45 meter in this version or will it be a project for later? So I know if I should continue to post issues here :)

ArnieO commented 3 years ago

Have you planned to implement a solution for my Swedish Aidon RJ45 meter in this version or will it be a project for later? So I know if I should continue to post issues here :)

Apparently it worked on your meter with this code: https://github.com/gskjold/AmsToMqttBridge/issues/146#issuecomment-977826704

So I see no reason why it should not work with this version; did you try the latest update in the first post? Settings should be: image

gskjold commented 3 years ago

Sorry, I missed your last message completely.. How often is the content invalid? When it returns -2 the checksum for the header is incorrect. The only reason I can see this happening is if the signal is bad or the parity is wrong for your meter.

Isaksson commented 3 years ago

I would say that maybe 1 or 2 out of 10 is valid, then there is a lot of readings that is not processed at all (just filling up the buffer). And between 30-60 minutes it looses connection to WiFi, and a reboot is needed for it to connect again, esp8266. I guess it has something to do that my meter fills the buffer and that causes some issue in the long run, if iam the only one experience issues with WiFi connection with this version.

Isaksson commented 3 years ago

I connected my Mbus adapter to an Arduino and just wrote a small code that dumped the readings, here is the result from my last 1500 readings, maybe you could run them in debug with your code and see if there is some errors in the output from my meter. HAN Readings HEX Dump.txt

gskjold commented 3 years ago

Thanks! That could be useful

ArnieO commented 3 years ago

I would say that maybe 1 or 2 out of 10 is valid, then there is a lot of readings that is not processed at all (just filling up the buffer). And between 30-60 minutes it looses connection to WiFi, and a reboot is needed for it to connect again, esp8266. I guess it has something to do that my meter fills the buffer and that causes some issue in the long run, if iam the only one experience issues with WiFi connection with this version.

Is this only with the code in this thread or did you have the same issues with the code you got working here? -> https://github.com/gskjold/AmsToMqttBridge/issues/146#issuecomment-977826704

ArnieO commented 3 years ago

Strange observation: After upgrading from 2.0-rc2 to 2.0-rc3, RSSI dropped by around 5 dBm. Was around -73 to -75 (indicated by hand-drawn red line). Now around -77 to -80 and below. Remote installed, so I have not been near the meter or touched the door of the cabinet. image

gskjold commented 3 years ago

I see no reasons why the code changes I have done would impact RSSI, so I think this must be related to core espressif behaviour. Probably a reboot could cause same type of change

Isaksson commented 3 years ago

I would say that maybe 1 or 2 out of 10 is valid, then there is a lot of readings that is not processed at all (just filling up the buffer). And between 30-60 minutes it looses connection to WiFi, and a reboot is needed for it to connect again, esp8266. I guess it has something to do that my meter fills the buffer and that causes some issue in the long run, if iam the only one experience issues with WiFi connection with this version.

Is this only with the code in this thread or did you have the same issues with the code you got working here? -> #146 (comment)

Its the same issue, if I try with the "old" version (1.5.8) then there is to small buffer so it will not get any good readings, then I changed to this beta code and the HAN port turned green for the first time :) so then i commented that it worked, then I started the discussion in this thread with the issue that I had with this code.

madsbg commented 2 years ago

RC3 fixed both the package timestamp and intermediate zero values for me. šŸ‘ I see no noticeable change in wifi signal.

gskjold commented 2 years ago

@Isaksson I have taken the dump you gave me and run it through a ESP8266 with 2.0-rc3 and there have only been a couple of frames that had checksum errors, but very few. Comparing the original frame and the received frame, there is indeed a difference where one byte is not equal. I think this boils down to instability with software serial on some devices. Not sure what version D1 mini you have, but my v3.1.0 is pretty stable with software serial. You could try connecting MBUS to GPIO13 (D7) and choose UART2 in GPIO settings. You will loose all serial logging via USB, but I'm guessing it would eliminate your problem. Use telnet logging to verify, but I suggest sticking to INFO level. I've been running it on UART2 for a while now and it seems to get every frame.

Isaksson commented 2 years ago

Thanks for that, I will test with UART2. My Lolin D1 mini is also v3.1.0. I will come back here with info after my tests.

Isaksson commented 2 years ago

Using UART2 solves the issue for me, I have had over 100 readings now and all of them is valid. Thanks again for the help, I hope this solves the issue that I had with the WiFi connection, I will find that out within one hour.

gskjold commented 2 years ago

Awesome, good to hear!

Isaksson commented 2 years ago

It's still an issue with the WiFi connection, after about one hour the device looses connection with WiFi, anyone else here using Lolin D1 Mini? @gskjold Have you tried this code with your D1 Mini for several hours?

I have a couple of other D1 mini, Iam going to test with one of them just to verify that it's not any hardware issue on my device.

The image is when the device has gone offline. Screenshot_20211203-193132_Photos

gskjold commented 2 years ago

Alright, I'll plug one in right now and check it tomorrow morning

gskjold commented 2 years ago

Its been running 11 hours straight now, no problems. Any settings I should know about? MQTT?

Isaksson commented 2 years ago

Its been running 11 hours straight now, no problems. Any settings I should know about? MQTT?

That sounds good, then let's hope it's an hardware issue on mine. Telnet debugging enabled, MQTT enabled. I could test to disable both debugging and MQTT and see if that makes any difference. I will also test new hardware today.

Isaksson commented 2 years ago

22 minutes up time before it was disconnected from WiFi with debugging and MQTT disabled.

gskjold commented 2 years ago

After enabling MQTT I have been able to reproduce this error, looking into it.

Isaksson commented 2 years ago

After enabling MQTT I have been able to reproduce this error, looking into it.

I see, my first try when i disabled debug and MQTT I just changed the setting, did not reboot after I changed the setting, I have now rebooted the device when the MQTT setting was disabled and now its reached 1 hour in up time. So it seems that its not enough to just disable it you need to boot up the device with the MQTT disabled, just a thought, maybe that information will help you with this issue

gskjold commented 2 years ago

Continuing this problem in #153

gskjold commented 2 years ago

Updated with 2.0-rc4