Testing v2.1.0 - Githubissues

gskjold commented 2 years ago

Attached is a new major version with changes outlined below. If anyone could test this on their setup, it would be highly appreciated.

!Warning! This is a firmware which is still under development. Testing this firmware is on your own risk. Please only test if you have possibility to flash via TTL or USB if anything goes wrong.

Changes:

Improved algorithm to update hour and day graph values
Removed support for loading config from v1.4.x series firmware
Visualize export as separate negative bar in graph

Added:

Month threshold tracking according to Norwegian distribution tariff
Ability to backup and restore from human readable config file
Support for Home Assistant auto discovery
Send uncoded payload to MQTT (HEX encoded bytes)
Possibility to view wattage display in watt always instead of kw (Click the value to change)
Possibility to factory reset during boot
Possibility to adjust WiFi power

Norway

[ ] Aidon 1p
[x] Aidon 3p IT
[x] Aidon 3p TN
[ ] Kaifa 1p
[ ] Kaifa 3p IT
[x] Kaifa 3p TN
[ ] Kamstrup 1p
[ ] Kamstrup 3p IT
[ ] Kamstrup 3p TN

Denmark

[ ] Kamstrup 1p
[x] Kamstrup 3p

Sweden

[ ] Aidon 3p TN

~~esp32.zip~~ ~~esp8266.zip~~

Fixed config download/upload.

~~esp32.zip~~ ~~esp8266.zip~~

Fixed issue with blank GUI (all values 0) for meters reporting power factor

~~esp32.zip~~ ~~esp8266.zip~~

Adjustments for hour change and real time calculation
Added separate HAN buffer (fixes Unknown data format 7B)
Added real time daily use to MQTT

~~esp32.zip~~ ~~esp8266.zip~~

Fixed data corruption of HA MQTT discovery messages

~~esp32.zip~~ ~~esp8266.zip~~

Added support for multipliers on meter data
Removed stat_cla from HA discovery message if empty

~~esp32.zip~~ ~~esp8266.zip~~

Changed config file upload behaviour. Now rebooting to read file

~~esp32.zip~~ ~~esp8266.zip~~

Fixed HA discovery for hostnames containing dash
Defaulting meter multipliers to 0 (deactivated)
Added ESP32-S2 build
Fixed decryption without auth key
Maybe fixed prices not clearing on MQTT after midnight?

esp32.zip esp8266.zip esp32s2.zip

Please reply to this post with your configuration (esp8266/esp32/pow-u etc), meter manufacturer, distribution type (IT/TT/TN) and country

ArnieO commented 2 years ago

Anyhow; my parity is correct set to 8N1. If I change it to 8E1, this is what I get:

gskjold commented 2 years ago

My correct parity is 8E1, if I change to 8N1 it still works. Can't actually see any errors.

gskjold commented 2 years ago

Spoke too soon, got this:

(D) 7B 22 69 6D 22 3A 33 35 33 33 33 2C 22 6F 6D 22 (D) 3A 30 2C 22 6D 66 22 3A 35 31 2C 22 69 22 3A 35 (D) 37 33 2C 22 65 22 3A 30 2C 22 72 69 22 3A 30 2C (D) 22 72 65 22 3A 35 30 32 2C 22 69 63 22 3A 30 2E (D) 30 31 39 2C 22 65 63 22 3A 30 2E 30 30 30 2C 22 (D) 72 69 63 22 3A 30 2E 30 30 30 2C 22 72 65 63 22 (D) 3A 30 2E 30 31 38 2C 22 75 31 22 3A 32 33 31 2E (D) 39 30 2C 22 75 32 22 3A 00 (W) (readHanPort)(C1) Unknown data format 7B

gskjold commented 2 years ago

Never mind, I know what it is now, not related :)

ArnieO commented 2 years ago

Never mind, I know what it is now, not related :)

I'm curious now! :cat:

gskjold commented 2 years ago

I've introduced a shared buffer in the code to reduce memory footprint. I should not have used the shared buffer for serial as well.

Hint: decode payload to ascii

tronde-ams commented 2 years ago

ESP32 / 42a627e / Aidon 3p IT / Norway

Found another discrepancy.

I lost one hourly packet and the graphics showed 0 as expected. When the next hourly packet was read, the graphics updated as expected. It seems like you have improved this because it showed correct value for both the missing hour and the last hour.

Then to the problem:

The total value of the two missing hours was 5.3 kWh (2.7 + 2.6). Because of this the max value changed from 4.1 / 5 kWh to 4.1 / 10 kWh. 2022-03-16_151736

tronde-ams commented 2 years ago

What vent wrong at 21?

I will check the logic here, it probably missed the hourly data.

I think it is best to rely only on the internal real time clock for everything related to the real time values. Real time values are very useful for control of equipment, but only if the values are to be trusted. If we miss a few readings from the meter it will not give a significant error, but if we miss the hourly reading the real time values will be rendered more or less useless for control.

kabu-kabu commented 2 years ago

@ArnieO: I just uploaded new version 53296cc and everything is working as expected. Kaifa M309 meter is recognized as Sagemcom again, GUI is updated, no restarts and plausible data via MQTT. For me so far: All good!

BTW: On main page in first graph it says Sagecom and on meter config it says Sage_m_com.

stbitdk commented 2 years ago

Esp8266 / 53296cc / Kamstrup / Denmark

I just tested ver 53296cc. It worked, but I've got a few errors:

Han port flipping between green and red.
Log file shows a few "(readHanPort) Unknown data format 7B"

Logfile attached.

ams200_53296cc_17mar22.txt

gskjold commented 2 years ago

Thank you for testing! "Unknown data format 7B" is a known problem. I just updated first post with new version beafcd3 that should fix this problem :)

stbitdk commented 2 years ago

Esp8266 / beafcd3 / Kamstrup / Denmark

I just tested ver beafcd3. Problems solved :)

rugaard commented 2 years ago

@gskjold @stbitdk So basically what you're saying is https://github.com/gskjold/AmsToMqttBridge/commit/beafcd300be2123237bda6a6c0dc012cb1ec1a93 fixes my issue #244? 🥳🥳🥳

gskjold commented 2 years ago

Hah, indeed @rugaard , that should be the case. I forgot that I have actually used that same code in 2.0.x

bjornsivertsen commented 2 years ago

We are not out of the woods yet. Though it rarely leaves green status now, it still jumps to red and yellow occasionally.

Unknown data format A1?

(D) (AmsWebServer)Serving /data.json over http...
(D) (AmsWebServer)Serving /data.json over http...
(D) (AmsWebServer)Serving /data.json over http...
(D) (readHanPort) Frame dump (491b):
(D) 7E A1 E9 41  03 13 C6 37  E6 E7 00 DB  08 4B 41 4D
(D) 45 01 C2 37  8C 82 01 D0  30 00 0E 26  F7 02 69 E4
(D) AB C0 35 43  A2 1F B3 03  D0 30 74 80  05 E6 FA 7C
(D) DF 37 3A B1  E8 3C 86 82  A5 19 AB 93  98 C3 2D 4A
(D) 37 DF D1 07  55 89 8B 03  6D 13 07 35  0B 32 31 0F
(D) F7 FE 5A 06  4F 54 13 44  3E 8C 22 B0  D1 2F CF C8
(D) 4C FE 0B 58  67 A0 8B 7D  E1 6A CB 64  E8 9E 8D 56
(D) DC E1 31 55  87 70 97 E9  09 1A D5 C1  E1 F8 45 7E
(D) 33 07 6F 60  8C FB 87 CF  3C 4E 41 52  4E CF C6 8B
(D) B3 68 B0 7E  6E 15 D5 16  2D 15 1D 56  A0 98 7B 2A
(D) AF AD FF 7B  00 83 C6 6C  C3 5A AB E7  49 4E AB C1
(D) 08 F0 6A 10  C2 A6 FE 28  73 69 4C 6B  4F 17 2A A5
(D) 99 AF D6 F6  46 BF 85 2F  B7 6E 26 74  08 92 E0 BD
(D) 2D 05 DC 90  21 AF E8 BA  B4 3F 54 E0  45 FE 4D FF
(D) 04 DF D3 4A  44 BE 29 54  A0 DD 13 F7  3A 12 D1 81
(D) 24 4E D2 DE  7F FF 6D 6D  BE 18 90 77  8B 8E 9D 1E
(D) 91 2E 32 17  58 E2 B5 B5  64 6A A3 55  B2 0D 2D F3
(D) 5F 11 EA EB  2F 17 2B 71  3B 99 69 62  EB 94 8A D3
(D) 28 05 B2 3A  45 39 1C D2  4E A5 4F 59  3E 6A BD 90
(D) C7 B6 F0 EA  AC D9 A0 B8  A4 B3 BB 51  28 90 0C 9A
(D) 41 93 DB F9  01 C1 E2 EE  F3 F3 4D F5  50 19 AA 8F
(D) F8 79 ED B5  E9 2C F2 23  4F F3 B2 29  19 E9 53 11
(D) F0 97 61 15  C8 BE F3 0D  DF F3 83 86  A1 90 B9 35
(D) A6 22 BE AD  20 BB 04 A4  B0 1A 54 B3  7D 78 9C B6
(D) 31 40 B8 AF  36 AD 91 F0  3C 0A 3E C7  ED C6 30 AB
(D) 1F 65 1C ED  CB 59 3C F3  94 3D B7 F8  7A 85 7E A2
(D) C1 8A 6B AE  A7 76 BE 23  3D E6 73 22  E0 F7 96 86
(D) 6A 5E 0B 8B  AB 7B 8D 74  FB 8E 3F C7  0C A6 56 7A
(D) D4 16 BE 79  D7 72 10 C2  CA B3 99 27  65 7F D0 92
(D) BD 9F 44 E5  1B D4 34 7C  3A 8F 0A 59  C9 F2 DD EC
(D) 53 B1 33 B4  2B 37 A0 13  FD 7E 7E
(D) (readHanPort) System title:
(D) 4B 41 4D 45  01 C2 37 8C
(D) (readHanPort) Initialization vector:
(D) 4B 41 4D 45  01 C2 37 8C  00 0E 26 F6
(D) (readHanPort) Additional authenticated data:
(D) 30 09 DF 55  4F A7 4E C8  92 27 B3 7D  32 11 77 1B
(D) 97
(D) (readHanPort) Authentication tag:
(D) D1 C7 EC 7D  E5 83 CC 43  29 4E C9 BE
(W) (readHanPort) Frame checksum error
(D) (readHanPort) Frame dump (1b):
(D) A1
(D) (readHanPort) System title:
(D) 4B 41 4D 45  01 C2 37 8C
(D) (readHanPort) Initialization vector:
(D) 4B 41 4D 45  01 C2 37 8C  00 0E 26 F6
(D) (readHanPort) Additional authenticated data:
(D) 30 09 DF 55  4F A7 4E C8  92 27 B3 7D  32 11 77 1B
(D) 97
(D) (readHanPort) Authentication tag:
(D) D1 C7 EC 7D  E5 83 CC 43  29 4E C9 BE
(W) (readHanPort) Unknown data format A1
(D) (AmsWebServer)Serving /data.json over http...
(D) (AmsWebServer)Serving /data.json over http...
(D) (loop) Used 0 ms to update temperature
(D) (AmsWebServer)Serving /data.json over http...
(D) (AmsWebServer)Serving /data.json over http...
(D) (readHanPort) Frame dump (491b):
(D) 7E A1 E9 41  03 13 C6 37  E6 E7 00 DB  08 4B 41 4D
(D) 45 01 C2 37  8C 82 01 D0  30 00 0E 26  FA 0F 00 00
(D) 00 00 0C 07  E6 03 11 04  13 1F 0A FF  80 00 00 02
(D) 41 0A 0E 4B  61 6D 73 74  72 75 70 5F  56 30 30 30
(D) 31 09 06 01  01 01 08 00  FF 06 00 0C  02 89 09 06
(D) 01 01 02 08  00 FF 06 00  00 00 00 09  06 01 01 03
(D) 08 00 FF 06  00 00 62 F0  09 06 01 01  04 08 00 FF
(D) 06 00 0B E4  82 09 06 01  01 00 00 01  FF 06 01 4C
(D) 4F A9 09 06  01 01 01 07  00 FF 06 00  00 01 9B 09
(D) 06 01 01 02  07 00 FF 06  00 00 00 00  09 06 01 01
(D) 03 07 00 FF  06 00 00 00  00 09 06 01  01 04 07 00
(D) FF 06 00 00  01 AA 09 06  00 01 01 00  00 FF 09 0C
(D) 07 E6 03 11  04 13 1F 0A  FF 80 00 00  09 06 01 01
(D) 20 07 00 FF  12 00 E2 09  06 01 01 34  07 00 FF 12
(D) 00 E0 09 06  01 01 48 07  00 FF 12 00  00 09 06 01
(D) 01 1F 07 00  FF 06 00 00  00 F4 09 06  01 01 33 07
(D) 00 FF 06 00  00 00 20 09  06 01 01 47  07 00 FF 06
(D) 00 00 00 00  09 06 01 01  15 07 00 FF  06 00 00 01
(D) 7B 09 06 01  01 29 07 00  FF 06 00 00  00 20 09 06
(D) 01 01 3D 07  00 FF 06 00  00 00 00 09  06 01 01 21
(D) 07 00 FF 12  00 47 09 06  01 01 35 07  00 FF 12 00
(D) 30 09 06 01  01 49 07 00  FF 12 00 64  09 06 01 01
(D) 0D 07 00 FF  12 00 45 09  06 01 01 16  07 00 FF 06
(D) 00 00 00 00  09 06 01 01  2A 07 00 FF  06 00 00 00
(D) 00 09 06 01  01 3E 07 00  FF 06 00 00  00 00 09 06
(D) 01 01 16 08  00 FF 06 00  00 00 00 09  06 01 01 2A
(D) 08 00 FF 06  00 00 00 00  09 06 01 01  3E 08 00 FF
(D) 06 00 00 00  00 09 06 01  01 15 08 00  FF 06 00 09
(D) C0 E0 09 06  01 01 29 08  00 FF 06 00  02 41 A9 09
(D) 06 01 01 3D  08 00 FF 06  00 00 00 00  2A 01 C3 12
(D) CD 34 65 0F  91 2C 75 45  29 C6 7E
(D) (readHanPort) System title:
(D) 4B 41 4D 45  01 C2 37 8C
(D) (readHanPort) Initialization vector:
(D) 4B 41 4D 45  01 C2 37 8C  00 0E 26 FA
(D) (readHanPort) Additional authenticated data:
(D) 30 09 DF 55  4F A7 4E C8  92 27 B3 7D  32 11 77 1B
(D) 97
(D) (readHanPort) Authentication tag:
(D) 2A 01 C3 12  CD 34 65 0F  91 2C 75 45
(D) (readHanPort) Valid data, start at byte 47
(V) (EnergyAccounting) Adding 0.0046 kWh
(V) (EnergyAccounting)  calculating threshold, currently at 0
(V) (EnergyAccounting)  new threshold 0
(D) (AmsWebServer)Serving /data.json over http...

gskjold commented 2 years ago

A1 is the second byte, so looks like it lost the first byte for some reason. Do I remember correct that you have a Pow-K ? Try re-seating the board in the module bay, I know others have successfully got rid of data corruption that way in the past.

bjornsivertsen commented 2 years ago

It did solve the A1 error. Thank you!

Still getting a bit of yellow status, about the same time Telnet says there is a Frame checksum error.

I mean it works, but the information might be valuable to you. :)

(D) (AmsWebServer)Serving /data.json over http...
(D) (AmsWebServer)Serving /data.json over http...
(D) (readHanPort) Frame dump (491b):
(D) 7E A1 E9 41  03 13 C6 37  E6 E7 00 DB  08 4B 41 4D
(D) 45 01 C2 37  8C 82 01 D0  30 00 0E 27  5E 38 B6 82
(D) 3A 3B 1B 9C  48 83 27 8E  CD EE 45 9C  09 C1 4A BC
(D) EB 03 1A 67  D4 44 46 D3  CF 2A 57 BF  1D 4F 9D 1E
(D) B7 5C A8 F1  60 9D 68 A7  3F 67 48 CD  29 42 BE 3A
(D) 33 92 CA 6E  01 0D 7E 98  28 E2 4B C3  78 6C 45 BA
(D) 27 96 39 74  BF D8 72 95  B3 07 61 63  F4 5A 01 25
(D) E6 73 D9 12  FB 16 29 02  D7 F5 80 75  7C EB 0F C6
(D) 43 A6 09 9D  51 34 66 84  23 FF 4B DC  D5 CC B0 78
(D) 4E BC 5C 9C  2F 84 02 DD  7E F1 F8 0D  E8 21 68 46
(D) 71 3F 85 D7  1D D0 3C 11  18 60 76 D8  9C 28 1C 66
(D) FE 7D 7A 38  C5 9F 09 76  DE C3 C5 72  03 CF DC 9E
(D) B9 73 52 71  F8 49 31 9E  06 8E 4A AB  7D A0 AF 9D
(D) 39 4E 9F 7A  98 B7 86 A4  A9 14 3A 0A  D8 5F 9B E0
(D) A6 CE DD 56  CA F6 64 6B  C2 79 8A 16  67 71 7C 88
(D) 03 A2 64 62  DC 9E D8 96  AF 60 B9 8A  4F C1 90 51
(D) FD B0 3E 40  55 D1 CE 8D  38 C3 46 E7  17 B3 32 14
(D) DC 43 32 50  30 98 18 87  44 8A DD 35  DE 94 87 C4
(D) A0 4C 0B 27  35 40 2E F7  70 90 04 90  D0 B2 89 C0
(D) 41 9C 41 9F  7A 84 0C 62  5E 59 87 89  6C A6 2B 75
(D) C9 94 3F 34  AC 51 13 5E  55 D3 D1 0C  78 5C 37 12
(D) 59 AA 26 C3  78 08 41 39  7B 98 CB C4  AA BF 83 2B
(D) B7 E1 FB 0D  04 F2 1F B9  E1 B8 21 5A  8E 07 04 3C
(D) B7 DC C8 BD  91 15 CB DA  25 35 13 C3  15 2C 1F 8F
(D) D8 E0 04 4F  0B 4B C7 9F  45 42 8A 69  1E 94 CF 9D
(D) D8 CD 9E 2A  B1 CC CF C2  6F CA DC BC  7B 19 86 36
(D) 57 35 BB 4E  7C FF 70 88  78 21 97 8C  47 CB D6 3E
(D) 4F A9 DC D8  1C 0D 4A C9  71 48 74 72  0B A4 CF 0D
(D) 54 7A 40 C6  70 B8 55 39  7A D6 CC DA  5E F3 FC 3C
(D) 91 F3 B4 20  23 2C 80 B4  79 41 D5 04  E6 7A 40 93
(D) A8 D6 03 3E  1F D8 96 3E  F0 19 7E
(D) (readHanPort) System title:
(D) 4B 41 4D 45  01 C2 37 8C
(D) (readHanPort) Initialization vector:
(D) 4B 41 4D 45  01 C2 37 8C  00 0E 27 5D
(D) (readHanPort) Additional authenticated data:
(D) 30 09 DF 55  4F A7 4E C8  92 27 B3 7D  32 11 77 1B
(D) 97
(D) (readHanPort) Authentication tag:
(D) 26 55 FC 0B  A5 ED 6B B7  9E 79 13 E1
(W) (readHanPort) Frame checksum error
(D) (AmsWebServer)Serving /data.json over http...

gskjold commented 2 years ago

Cool! Although it should be possible to receive all data flawlessly (in a perfect world), random checksum errors are not uncommon. There are a number of things that could cause this, but I think we should be happy if it mostly works :)

bjornsivertsen commented 2 years ago

I am very pleased!

Now just to get the auto discovery working for HA, which I am also struggling with. But that is a discussion for another issue. ;)

gskjold commented 2 years ago

Some adjustments made for HA discovery. Also added multiplier config for those who have CT meters or have a format which is interpreted with incorrect scaling. First post updated.

enordli commented 2 years ago

I tested the latest version, Aidon 6540 with current transformer, powerfactor 40 I have hardcoded the powerfaktor before, but not anymore. It still works with 50 meter old telephone cable between the POW-U and energy-meter The HA discovery looks ok now Backup of configuration is nice, backup of history data would be nice too, i lost mine when i tried to go back to the previous firmware.

bilde

gskjold commented 2 years ago

Beautiful!

Graph values should be included in config file download, and should also restore if you upload. File is human readable, so you should find "dayplot" and "monthplot" in the bottom of the file.

enordli commented 2 years ago

I made a backup before i lost the history. so i made a new backup with of the latest version and replaced the 3 last lines from the first backup. I did a restore from the config, but it loaded and mabe restarted, and i restarted it again, but noting changed, still missing history.
configfileltest-cfg.log

bilde

i tried to backup only Thresholdsdata and replased the 3 lines with the backup data. did a upload but no luck

BKFlister commented 2 years ago

POW-U / Aidon / IT / Norway

I am not able to get autodiscover working. Might be misconfiguration on my side?

POW-U MQTT settings Client ID: pow-u Publish topic: Power Payload: JSON

Mosquitto broker Topic to subscript to: power -> Receiving data OK

Payload: Home-Assistant Error in home-assistant.log:

2022-03-19 09:17:07 ERROR (MainThread) [homeassistant.util.logging] Exception in async_discover when dispatching 'mqtt_discovery_new_sensor_mqtt': ({'name': 'Accumulated reactive export', 'state_topic': 'power/energy', 'unique_id': 'ams-4e30_tQO', 'object_id': 'ams-4e30_tQO', 'unit_of_measurement': 'kWh', 'value_template': '{% if value_json.tQO is defined %} {{ value_json.tQO }} {% else %} {{(states.sensor.ams-4e30_tQO|float)}} {% endif %}', 'device_class': 'energy', 'device': {'identifiers': ['ams-4e30'], 'name': 'AMS reader', 'model': 'ESP8266', 'sw_version': '48bd352', 'manufacturer': 'AmsToMqttBridge', 'configuration_url': 'http://ams-4e30.local/'}, 'state_class': 'total_increasing', 'platform': 'mqtt'},) Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/mqtt/mixins.py", line 246, in async_discover config = schema(discovery_payload) File "/usr/local/lib/python3.9/site-packages/voluptuous/validators.py", line 218, in call return self._exec((Schema(val) for val in self.validators), v) File "/usr/local/lib/python3.9/site-packages/voluptuous/validators.py", line 341, in _exec raise e if self.msg is None else AllInvalid(self.msg, path=path) File "/usr/local/lib/python3.9/site-packages/voluptuous/validators.py", line 337, in _exec v = func(v) File "/usr/local/lib/python3.9/site-packages/voluptuous/schema_builder.py", line 272, in call return self._compiled([], data) File "/usr/local/lib/python3.9/site-packages/voluptuous/schema_builder.py", line 817, in validate_callable return schema(data) File "/usr/local/lib/python3.9/site-packages/voluptuous/schema_builder.py", line 272, in call return self._compiled([], data) File "/usr/local/lib/python3.9/site-packages/voluptuous/schema_builder.py", line 594, in validate_dict return base_validate(path, iteritems(data), out) File "/usr/local/lib/python3.9/site-packages/voluptuous/schema_builder.py", line 432, in validate_mapping raise er.MultipleInvalid(errors) voluptuous.error.MultipleInvalid: invalid template (TemplateSyntaxError: expected token ')', got '_tQO') for dictionary value @ data['value_template']

BKFlister commented 2 years ago

Got AutoDiscover in Home Assistant to work by renaming the POW-U Hostname

gskjold commented 2 years ago

Maybe it doesn't like the dash in the hostname?

gskjold commented 2 years ago

@enordli thank you for the test file. It turns out it was rebooting because of error on ESP8266 when uploading config file. After a lot of testing, I decided to force a reboot after uploading config file, but before parsing it. I am now able to successfully load your config file. First post updated with new firmware.

BKFlister commented 2 years ago

Tested adding a dash/hyphen in the hostname, deleted the existing discovered device. HA did not discover with dash/hyphen in hostname. Changed hostname back without dash/hyphen and HA discovered the device

Getting some warnings in the HA log regarding float: 2022-03-20 20:29:58 WARNING (MainThread) [homeassistant.helpers.template] Template warning: 'float' got invalid input '<template TemplateState(<state sensor.amsreader_u3=0; unit_of_measurement=V, device_class=voltage, friendly_name=L3 voltage @ 2022-03-20T20:29:53.350493+01:00>)>' when rendering template '{% if value_json.U3 is defined %} {{ value_json.U3 }} {% else %} {{(states.sensor.amsreader_U3|float)}} {% endif %}' but no default was specified. Currently 'float' will return '0', however this template will fail to render in Home Assistant core 2022.1

enordli commented 2 years ago

With new firmware, the backup and restore of config worked just fine, It looks like it read the config i uploaded yesterday, because the history was already restored after i uploaded the firmware.

The montly graph is not showing the correct value for each day, i use almost the same every day, 50 to 60 Kwh it has nothing to do with the backup/restore, i know that i sometimes don't get the hourly frames from the powermeter, it is very random. If it missing the hourly frame at midnight, is that the answer to the odd graph? mabe it has something to do with the 50 meter cable, is it normal to miss some frames?

bilde

I pass the mqtt to HA and there it look OK

bilde

tronde-ams commented 2 years ago

" mabe it has something to do with the 50 meter cable, is it normal to miss some frames?"

I understand you use old telphone cable? It is usually not very good for pulsed signals. It can sometimes help to use a schmitt trigger close to the ESP board if this makes some sense to you.

tronde-ams commented 2 years ago

Maybe I was a little bit qiuck here. A schmitt trigger will help to shape the pulses, but the voltage level from the HAN.port complicates.

gskjold commented 2 years ago

It looks like it read the config i uploaded yesterday, because the history was already restored after i uploaded the firmware.

Ah yeah, that's right, the code did not delete the file earlier, so everyone who have already uploaded a file will now have that file restored when they install the latest patch... A bit inconvenient, but that's how beta is..

The monthly graph is not showing the correct value for each day

This is because it did not successfully receive the accumulated values after midnight. The strange "two low, one high" symptom is something I have been chasing since I first released the graph. I thought I had found it and fixed for 2.1 beta, but not sure.

is it normal to miss some frames?

In general you will always loose data every now and then, but you should rarely see this on the graph. I guess your cable could possibly increase the checksum error rate, but I am no expert in that area.

alftore commented 2 years ago

POW-K / Kamstrup 1p / IT / Norway

Looks good. I have not testet config download / upload. HA autodiscovery working after renaming hostname

enordli commented 2 years ago

Ok, I have wifi in the barn now where the powerrmeter is located and i will move the POW-U close to the powermeter and se if it will give less frame loss. Now I loose only 2 -3 frames a day. I will let you know if it makes the difference. The "two low, one high" is from the version 2.0.11, i updated to v2.1.0 18 mars, so we vill see if it goes away with the fix have done.

ArnieO commented 2 years ago

I see occasional frame losses also with a Pow-K board mounted inside a Kamstrup meter, with direct transfer of data (not via M-bus). Just to indicate there could be other reasons for those (few) frame losses than transmission issues.

PalmMx commented 2 years ago

HA autodiscovery working after renaming hostname

Can you please specify what is a proper hostname, i.e. what makes a hostname not working for HA autodiscovery?

alftore commented 2 years ago

Removed dash from hostname. Had same issue as BKFlister.

tronde-ams commented 2 years ago

ESP32 / fb60127 / Aidon 3p IT / Norway

Strange behaviour for prices and MQTT.

Sometimes it seems like the prices sent to MQTT (RAW full) does not update. I use broker.emqx.io which has been OK. I see the same in MQTT explorer on my PC and the Android app on my phone.

Before midnight it was stuck for several hours. See for instance price 19 in MQTT and compare to GUI which uptdates. Older prices are not cleared, except 33 and 34. 2022-03-21_235414 2022-03-21_235441

After midnight it did update, but not clear old prices. Some of the older prices has been changed / repeated. 2022-03-22_003535 2022-03-22_003558

kng commented 2 years ago

HA discover: the "dash in the hostname" problem seems to be solved with: "val_tpl" : "{%% if value_json.%s is defined %%} {{ value_json.%s }} {%% else %%} {{(states('sensor.%s_%s')|float)}} {%% endif %%}", But I'm not sure it's a good solution, as it is guesstimating the resulting name of the sensor and this could lead to trouble. I have made some testing with: "val_tpl" : "{{ value_json.%s | is_defined }}", Check out #251

enordli commented 2 years ago

I use POW-U and Aidon meter and i used the default hostname with dash and it was no trouble at all with HA discovery

bilde

alftore commented 2 years ago

Might be because the first char after the dash was a number in my case. Fix from kng seems right, not sure what the concerns are about guesstimating.

gskjold commented 2 years ago

@kng beautiful! Merged

kng commented 2 years ago

not sure what the concerns are about guesstimating

I meant guesstimating entity_id from the object_id, as this algorithm has changed earlier and for example in my setup it is named entirely different as it's from an earlier generation. It is also prone to add '_1' as padding to maintain unique id's etc. Also means it would be copying old values back into the db for no good reason. Please verify that the solution with is_defined works for you. I have tested it on my HA and looked liked it behaved properly, i.e updating only one value and leaving the rest alone.

risto-m commented 2 years ago

Loaded the latest version 2.1.0 on a ESP32.

And it looks very promising! I am in Sweden and grid operator Gävle Energi.

gskjold commented 2 years ago

@tronde-ams Not sure what is happening there, but I found a problem that I have fixed, not sure if it changes that behaviour though, we'll see.

Updated first post with new firmware. Can someone from DK to confirm that the last build works with encryption ?

All zip files now contains bootloader and boot file for esp32. Added esp32-s2 build as well. Added a flashing script to the zip, but I guess most users are on windows, so will probably add a readme file as well later. Or maybe a windows compatible script. Powershell? Open for suggestions.

tronde-ams commented 2 years ago

ESP32 / e7fc504 / Aidon 3p IT / Norway

No prices fetched. Same if I disable MQTT or disable / re-enable the API key for prices.

(16:48:31.939) <ESC>[0m(I) (setupHanPort) Setting up HAN on pin 16 with baud 2400 and parity 11
(16:48:42.743) <ESC>[0m(D) (setupHanPort)(C1) Hardware serial
(16:48:42.743) <ESC>[0m(I) (loop)(C1) Successfully connected to WiFi!
(16:48:45.504) <ESC>[0m(I) (loop)(C1) IP:  192.168.1.40
(16:48:45.520) <ESC>[0m(I) (loop)(C1) GW:  192.168.1.1
(16:48:45.520) <ESC>[0m(I) (loop)(C1) DNS: 8.8.8.8
(16:48:45.520) <ESC>[0m(D) (loop)(C1) mDNS is enabled, using host: ams-7d80
(16:48:45.520) <ESC>[0m(I) (MQTT_connect)(C1) Connecting to MQTT test.mosquitto.org:1883
(16:48:45.598) <ESC>[0m(I) (MQTT_connect)(C1) Successfully connected to MQTT!
(16:48:45.863) <ESC>[0m(I) (EntsoeApi) Setting midnight millis 25880001
(16:48:50.314) <ESC>[0m(I) (EntsoeApi) Fetching prices for today
(16:48:50.330) <ESC>[0m(D) (EntsoeApi)  url: https://transparency.entsoe.eu/api?securityToken="MY TOKEN"&documentType=A44&periodStart=202203252300&p<ESC>[0meriodEnd=202203262300&in_Domain=10YNO-1--------2&out_Domain=10YNO-1--------2
(16:48:50.361) <ESC>[0m(D) (EntsoeApi)Connection established
(16:48:50.361) <ESC>[0m(D) (EntsoeApi)Receiving data
(16:48:52.048) <ESC>[0m(I) (EntsoeApi) Retrieving EUR to NOK conversion
(16:48:52.063) <ESC>[0m(D) (EntsoeApi)  url: https://data.norges-bank.no/api/data/EXR/M.EUR.NOK.SP?lastNObservations=1
(16:48:52.095) <ESC>[0m(D) (EntsoeApi)Connection established
(16:48:52.095) <ESC>[0m(E) (EntsoeApi)Communication error: 
(16:48:52.157) <ESC>[0m(E) (EntsoeApi)connection refused
(16:48:52.173) <ESC>[0m(D) (EntsoeApi)
(16:48:52.173) <ESC>[0m(D) (readHanPort)(C1) Valid data, start at byte 18
(16:48:52.188) <ESC>[0m(I) (EnergyAccounting) Initializing data at 1648309733
(16:48:52.329) <ESC>[0m(I) (EntsoeApi) Retrieving EUR to NOK conversion
(16:48:52.469) <ESC>[0m(D) (EntsoeApi)  url: https://data.norges-bank.no/api/data/EXR/M.EUR.NOK.SP?lastNObservations=1
(16:48:52.469) <ESC>[0m(D) (EntsoeApi)Connection established
(16:48:52.469) <ESC>[0m(E) (EntsoeApi)Communication error: 
(16:48:52.516) <ESC>[0m(E) (EntsoeApi)connection refused
(16:48:52.516) <ESC>[0m(D) (EntsoeApi)
(16:48:52.516) <ESC>[0m(I) (EntsoeApi) Retrieving EUR to NOK conversion
(16:48:52.531) <ESC>[0m(D) (EntsoeApi)  url: https://data.norges-bank.no/api/data/EXR/M.EUR.NOK.SP?lastNObservations=1
(16:48:52.531) <ESC>[0m(D) (EntsoeApi)Connection established
(16:48:52.531) <ESC>[0m(E) (EntsoeApi)Communication error: 
(16:48:52.578) <ESC>[0m(E) (EntsoeApi)connection refused
(16:48:52.578) <ESC>[0m(D) (EntsoeApi)
(16:48:52.578) <ESC>[0m(I) (EntsoeApi) Fetching prices for tomorrow
(16:48:52.609) <ESC>[0m(D) (EntsoeApi)  url: https://transparency.entsoe.eu/api?securityToken="MY TOKEN"&documentType=A44&periodStart=202203262300&p<ESC>[0meriodEnd=202203272300&in_Domain=10YNO-1--------2&out_Domain=10YNO-1--------2
(16:48:52.625) <ESC>[0m(D) (EntsoeApi)Connection established
(16:48:52.625) <ESC>[0m(E) (EntsoeApi)Communication error: 
(16:48:52.687) <ESC>[0m(E) (EntsoeApi)connection refused
(16:48:52.703) <ESC>[0m(D) (EntsoeApi)
(16:48:52.703) <ESC>[0m(D) (readHanPort)(C1) Valid data, start at byte 18
(16:48:52.703) <ESC>[0m(I) (EntsoeApi) Retrieving EUR to NOK conversion
(16:48:52.828) <ESC>[0m(D) (EntsoeApi)  url: https://data.norges-bank.no/api/data/EXR/M.EUR.NOK.SP?lastNObservations=1
(16:48:52.843) <ESC>[0m(D) (EntsoeApi)Connection established
(16:48:52.843) <ESC>[0m(E) (EntsoeApi)Communication error: 
(16:48:52.875) <ESC>[0m(E) (EntsoeApi)connection refused
(16:48:52.890) <ESC>[0m(D) (EntsoeApi)
(16:48:52.890) <ESC>[0m(I) (EntsoeApi) Retrieving EUR to NOK conversion
(16:48:52.890) <ESC>[0m(D) (EntsoeApi)  url: https://data.norges-bank.no/api/data/EXR/M.EUR.NOK.SP?lastNObservations=1
(16:48:52.890) <ESC>[0m(D) (EntsoeApi)Connection established
(16:48:52.906) <ESC>[0m(E) (EntsoeApi)Communication error: 
(16:48:52.953) <ESC>[0m(E) (EntsoeApi)connection refused
(16:48:52.953) <ESC>[0m(D) (EntsoeApi)
(16:48:52.953) <ESC>[0m(D) (readHanPort)(C1) Valid data, start at byte 18
(16:48:54.450) <ESC>[0m(I) (EntsoeApi) Retrieving EUR to NOK conversion
(16:48:54.559) <ESC>[0m(D) (EntsoeApi)  url: https://data.norges-bank.no/api/data/EXR/M.EUR.NOK.SP?lastNObservations=1
(16:48:54.591) <ESC>[0m(D) (EntsoeApi)Connection established
(16:48:54.591) <ESC>[0m(E) (EntsoeApi)Communication error: 
(16:48:54.606) <ESC>[0m(E) (EntsoeApi)connection refused

What is the difference between the normal ESP32 build and ESP32-S2? They show up as the same build, but the -S2 had to be flashed with USB.

tronde-ams commented 2 years ago

ESP32 / Aidon 3p IT / Norway

Same for fb60127 and 220124.2

One of the prices for tomorrow shows up as negative. https://transparency.en says it should be 184.99 EUR

Multipler is 1,00 and area NO1 2022-03-26_170827

gskjold commented 2 years ago

What is the difference between the normal ESP32 build and ESP32-S2?

ESP32-S2 is a different chip.

One of the prices for tomorrow shows up as negative

I have the same issue... Weird, will have a look at it.

gskjold commented 2 years ago

I know why, we are changing to summer time so tomorrow is only 23 hrs :) "no value" in my code is -127, which is what you see here. It should not have shown, will make sure it is hidden in the future

UtilitechAS / amsreader-firmware

Testing v2.1.0 #247