search

UtilitechAS / amsreader-firmware

ESP8266 and ESP32 compatible firmware to read, interpret and publish data to MQTT from smart electrical meters, both DLMS and DSMR is supported
Other
390 stars 73 forks source link

Request: Support for Kamstrup Omnipower meters using GPK11 encryption key #126

Closed heinekmadsen closed 9 months ago

heinekmadsen commented 3 years ago

Hi...

I'm not sure this is even possible, as i have no deeper knowledge into this topic. The case is, that i bought a Pow-K (Kamstrup) module from amsleser.no. I have fittet it into my Kamstrup OmniPower meter and have supplied the GPK60 and GPK61 keys, but i get no data.

According to my power provider they state that it works with the Smart-Me modules using GPK11 key. the provider does not have access to that part of the Kamstrup software where they can enable that feature.

So my request is, can this software use a GPK11 key to get data?

ArnieO commented 3 years ago

I propose we first establish whether the meter provides data at all, as there has been cases of meters not having the HAN port open despite the grid company insistance. Does the LED show a blue light flickering for around 1 second every 10 seconds?

heinekmadsen commented 3 years ago

Theres a red flash every 2-3 seconds, and the HAN status In the web ui is red.

ArnieO commented 3 years ago

If there are no blue flashes there is no incoming data. The red flashes indicate the same as the red HAN status. There is a similar case in one of the threads here, the problem turned out to be that the port was not activated after all.

ArnieO commented 3 years ago

Here: https://github.com/gskjold/AmsToMqttBridge/issues/115

heinekmadsen commented 3 years ago

If using a multimeter to test if my meter is sending data, then what should i expect to read? I assume i need to measure between pin 5 and 6.

ArnieO commented 3 years ago

Correct. Looking into the meter you will measure between pins 5 and 6: image

If the meter is open, the voltage on pin 5 will be 3,3V for around 9 seconds, then a data frame (voltage jumping between 0 and 3,3V; on a multimeter you will just see that the voltage is not stable) that lasts for around 1 second. There is one such data frame each 10 seconds.

heinekmadsen commented 3 years ago

And is that the same data that a Smart-me module would utilize? or is it another type of data? I Assume data is pushed and not pulled from the meter? is that the same for the smart-me modules?

ArnieO commented 3 years ago

Data is pushed from the meter, not pulled. The DATA_IN pin (2) is not used. It is only on pin 5 data is delivered from the meter, so the Smart-me module must use the same pin and same data.

heinekmadsen commented 3 years ago

So... I hooked up some jump wires to the pins and my multimeter. Used my phone to record a video of the multimeter for 35 seconds.

Result: a steady 3,29v for the whole 35 seconds. Not a single deviation at all.

I assume this means no data is being sent from the meter. And if I used a smart-me module it wouldn't work either, correct?

According to the grid company my meter should be ready for a smart-me module, but it doesn't makes sense to me at all.

ArnieO commented 3 years ago

So - no data. You seem to have the same situation as the guy in https://github.com/gskjold/AmsToMqttBridge/issues/115

I would contact the grid company and ask them to double check that the HAN port is acivated.

heinekmadsen commented 3 years ago

well.. decided to buy a smart-me module for testing... then I can tell the grid company, that even though they say a smart-me module will work, it doesn't (Or probably wont work) They state they don't have access to the feature in the Kamstrup software to enable/disable the HAN port, but a smart-me module works........... without data?

ArnieO commented 3 years ago

Sounds like an expensive way to prove there is no data. I think the other guy was in contact with Kamstrup directly in some way. Maybe ask him in the other thread?

heinekmadsen commented 3 years ago

Well yes it’s the expensive way… So the smart me module was on my doorstep this morning…

And the result really bugs me 😔 plugged it in, joined Wi-Fi, added the gpk11 key and right away it reads data…. So there’s something different with the amstomqttbridge and smart-me module on how they get the data and the gpk keys.

I’d still rather use the pow-k module over the smart-me module, so let me know if It may be possible or if I can take part in some debugging

Hent Outlook til iOShttps://aka.ms/o0ukef

Fra: ArnieO @.> Sendt: Wednesday, October 13, 2021 8:27:50 PM Til: gskjold/AmsToMqttBridge @.> Cc: Heine Madsen @.>; Author @.> Emne: Re: [gskjold/AmsToMqttBridge] Request: Support for Danish meters (Omnipower) using GPK11 key (#126)

Sounds like an expensive way to prove there is no data. I think the other guy was in contact with Kamstrup directly in some way. Maybe ask him in the other thread?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/gskjold/AmsToMqttBridge/issues/126#issuecomment-942597472, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALVECIEMUY6J2LXFCTIO473UGXFSNANCNFSM5FOGQJYQ.

ArnieO commented 3 years ago

Well, this was surprising. There are around twenty Pow-K boards running in Denmark now, and I have not seen this problem before.

The only idea I can think of is that the meter needs an "enable" signal on an input pin to ensure the meter outputs data. This is not specified for the Kamstrup meters, but is used in the IEC 62056-21 interface standard used in for instance Sweden and Netherlands (see my overview here). Those interfaces have a "Request" input that must be set high from the device in order to get data from the meter.

The Kamstrup meter has one input pin, see this document, which is pin 2 (figure 5): image

The document states (page 9) that pin 2 is pulled high inside the meter by an 82k resistor. It also states that the pin is used for "serial communication of data to the meter", which is in most countries not used (disabled in the meter firmware, I assume) - to eliminate the possibility of the user being able to do whatever could be done from this port. This is why I chose to have pin 2 on Pow-K not connected.

Now I can only speculate:

However, pin 2 is specified for maximum voltage of 3,3V, so you cannot connect it to pin 1.

This is what you could do if you're comfortable with trying this and have access to a suitable resistor:

Connect a resistor (I would try 10k, but anything smaller than than 82k should work) between connector pin 2 and 3,3V on the Pow-K board. If you're equipped to solder it in that is fine, just be careful not to short circuit the connector pins, and also do not apply too much heat close to the large capacitor! Like this: image If you are not equipped for soldering, try to twist the leads of the resistor in place, and be sure that there is no short circuit to other pins of the connector.

(Note that there is an unfortunate error on the silkscreen of the card: Vcc and GND on the large capacitor are switched.)

heinekmadsen commented 3 years ago

Thanks for that really nice and detailed reply! :)

I'm fairly confident in soldering things. Can you possibly link me to a resistor? Just so I order the correct one :)

ArnieO commented 3 years ago

I would recommend a small kit, like this one: https://elektronik-lavpris.dk/p95985/k-res-e3-modstandssortiment-e3-480-stk/

(I am not familiar with Danish webshops for components, this was just the first one I found on google - there might be others that are better of cheaper, I have no idea. There might also be stores close to you where you can get such stuff.)

heinekmadsen commented 3 years ago

Great.. just ordered...

I might have some resistors laying around bit will have to check their values.

I also assume i need to discharge the super-cap before doing anything... any how-to's on that? or should i just leave it unplugged for some time?

ArnieO commented 3 years ago

Yes, that is the safe and simple way - just leave it unplugged for some time. It will self discharge. If you want to speed up the discharge, connect a (not much smaller than) 100 ohm resistor across the legs of the capacitor. The voltage will in both cases decrease asymptotically, and faster with a resistor.

heinekmadsen commented 3 years ago

Will this be sufficient? image

ArnieO commented 3 years ago

Yes that will work. I recommend measuring it with a multimeter to be sure of the value.

In fact you could connect just a wire (no resistance), but in case you make a mistake with the connection it is safer to use a resistor.

heinekmadsen commented 3 years ago

I found a 10k ohm, but still no cigar 🤥 image

Should I try some higher ohm resistor?

ArnieO commented 3 years ago

That looks good. Still no blue flashing on the LED?

heinekmadsen commented 3 years ago

Still no blue flashing, only red flashes and red status in the ui

ArnieO commented 3 years ago

What makes this strange is that what you reported earlier (multimeter measurement) indicates that there is no data coming from the meter. However, the Smart-me unit works as expected and reads data from the meter.

The blue pin on the LED is directly connected to the incoming data signal, so it will flash providing there is voltage on the board. Which there clearly is - as the ESP8266 is up and running, generates the UI and generates red flashes on the LED.

So let us assume the board does receive data from the meter, and look for potential errors that could prevent the blue LED from flashing. Could you be so kind as to check two points:

  1. The pin distance on the LED (D1) is very small, so I take great care to verify after soldering that there is no short circuit between pins. Can you check with ohmmeter the resistance between pin 3 and 4 on the LED? And if you have a good magnifying glass, check if you see any indication of a short circuit between them? The blue pin is pin 4, green is pin 3. If there is a short circuit between those, the signal to the green LED would interfere with the data signal. See below indications of D1 pins 3 and 4.
  2. This should not prevent the blue LED from flashing, but is still a point to check: I did have a bad batch of voltage regulators (U1) in the period I made that board - but I did not think I had shipped any board with bad U1. I can see you make good photos: Can you send me a photo showing the marking on U1? The "bad apples" are easy to identify from the marking.

image image

heinekmadsen commented 3 years ago

So i tried checking the led... I can't measure any short circuit.... I tried taking some pictures and a video when it's in the meter image image https://user-images.githubusercontent.com/48906528/137384870-450b283f-cbae-4466-995e-a8fc9d39fc8d.MOV

Just thinking out loud and with minimal knowledge about the meters... could I be that the smart me module gets the data from the meter wirelessly?

And why does it only need one gpk key and the amstomqtt needs two gpk keys?

gskjold commented 3 years ago

Regarding the keys: https://github.com/gskjold/AmsToMqttBridge/issues/73#issuecomment-625050664

Just thinking out loud here, could it be that the smart-me module asks the meter for data rather than passively receiving?

heinekmadsen commented 3 years ago

@gskjold not sure how to tie this with #73... But... i do recall that i might have read somewhere on the internet, that smart-me works by pull and amstomqttbridge works with push'ed data.

Would i be possible to implement a pull meter to the amstomqttbridge? At first just for a POC (Need to return my smart-me module within 10 days :D )

gskjold commented 3 years ago

Quote from the linked comment from #73 : "It seems the key they provided me was for the Smart-Me module. They're looking into getting me the other keys now."

If we are to implement data pull from the meter, I would need to know how to handshake the meter to receive data. If you have found any information related to this, let me know.

heinekmadsen commented 3 years ago

Well.. that might be a dead-end :( https://www.ihc-user.dk/forum/forums/topic/7467-kamstrup-elm%C3%A5lere/?do=findComment&comment=58737

A KMP protocol... Will try to reach out to Kamstrup and see if they are willing to hand out some documentation for this protocol.

gskjold commented 3 years ago

I think you are better of asking your grid company for the GPK60/61 keys instead, like @mikfoo did.

Referencing this for the future (in case we ever figure out how this works): https://github.com/MTrab/Kamstrup

@ArnieO is the meters RX pin connected on Pow-K at all?

heinekmadsen commented 3 years ago

well... i already have all the keys needed... GPK11, GPK60 and GPK61. Currently waiting for a guy at kamstrup to call me back.

gskjold commented 3 years ago

Aha, I see. There has to be some kind of configuration in the meter that they have to change. Unfortunately I don't know the details around that, would be nice to know what to ask for specifically.

ArnieO commented 3 years ago

Thank you for the links; very useful! I believe the explanation and key to a solution lies here: https://www.ihc-user.dk/forum/forums/topic/7467-kamstrup-elm%C3%A5lere/?do=findComment&comment=58732 "Den krypteringsnøgle man fik udleveret før den nye firmware kunne kun bruges til smart-me modulet. Her efter firmware updaten til Radius' kunder sender måleren krypteret data hvert 10. sek, hvor du skal bruge de to nøgler for at dekryptere dem. "

Your grid company must request a firmware upgrade. My understanding is that Kamstrup can do this this remotely - but probably needs to be "pushed" by your grid company.

heinekmadsen commented 3 years ago

Aha, I see. There has to be some kind of configuration in the meter that they have to change. Unfortunately I don't know the details around that, would be nice to know what to ask for specifically.

Exactly... and my grid company don't have access to that part of the software, and are not willing to pay for access to that, as i'm the only one requesting this in their area :( And they stated a smart-me module would work, which it "sadly" does. Just not that into all those cloud services :(

heinekmadsen commented 3 years ago

Thank you for the links; very useful! I believe the explanation and key to a solution lies here: https://www.ihc-user.dk/forum/forums/topic/7467-kamstrup-elm%C3%A5lere/?do=findComment&comment=58732 "Den krypteringsnøgle man fik udleveret før den nye firmware kunne kun bruges til smart-me modulet. Her efter firmware updaten til Radius' kunder sender måleren krypteret data hvert 10. sek, hvor du skal bruge de to nøgler for at dekryptere dem. "

Your grid company must request a firmware upgrade. My understanding is that Kamstrup can do this this remotely - but probably needs to be "pushed" by your grid company.

I agree.. that woudl make amstomqttbridge work as it is right now... but my grid company refuses to buy access to that part of their kamstrup software :(

So if we could figure out how the "KMP" protocol works, maybe support for that protocol could be added.

heinekmadsen commented 3 years ago

Maybe some handy info here... even though it's rater old: https://github.com/bsdphk/PyKamstrup

gskjold commented 3 years ago

Thanks, looks similar to the arduino project I referenced ealier. This method only worked before they added encryption.

ArnieO commented 3 years ago

This could actually be the explanation for the issue in https://github.com/gskjold/AmsToMqttBridge/issues/115. That guy had the meter replaced by a new one - which was probably delivered with new firmware. Then he "just" had to get the HAN-port opened before it worked.

@heinekmadsen -> What is the name of your grid company? I would like to warn other potential buyers.

What about the regulatory requirements? I have not managed to find them for Denmark yet, I searched for it on the website of Energistyrelsen without success so far. For cases like this it would be very useful to know what they say.

The purpose of the HAN-port is to give end user access to data using third party equipment. So what is said here https://github.com/gskjold/AmsToMqttBridge/issues/115 should not be acceptable (if it is correct). My guess is that compliance with regulatory requirements is the reason for the firmware upgrade last year where the meters started to push data.

I suspect your grid company has to get the upgrade done in order to comply with the regulatory requirements, so if we could just find those documents...

heinekmadsen commented 3 years ago

@ArnieO It's called "Nord Energi" https://www.nordenergi.dk/

I'll see if i can find any regulatory info.

ArnieO commented 3 years ago

I am collecting information on the meters here: https://github.com/ArnieO/SmartMeterDocumentation You can see there what I have found for Norway and Sweden in this table: https://github.com/ArnieO/SmartMeterDocumentation#national-technical-requirements

If you find regulatory information for Denmark, it would be great if you could report it as an "issue" on that repo: https://github.com/ArnieO/SmartMeterDocumentation/issues

heinekmadsen commented 3 years ago

Found this in a document for another kamstrup meter image https://stockshed.uk/files/kamstrup/Kamstrup%20Multical%20603%20-%20Technical%20Description.pdf

heinekmadsen commented 3 years ago

https://github.com/ArnieO/SmartMeterDocumentation/issues/1

ArnieO commented 3 years ago

There is supposedly some information on Kamstrup Meter Protocol here: https://github.com/bsdphk/PyKamstrup/issues/3 I have requested access.

ArnieO commented 3 years ago

@heinekmadsen : Thank you for https://github.com/ArnieO/SmartMeterDocumentation/issues/1 !

This is exactly what you must refer to when you discuss with your grid company.

They will then have two options:

  1. Upgrade your meter firmware to push data
  2. Provide full documentation of the Kamstrup Meter Protocol
heinekmadsen commented 3 years ago

I reached out to the grid company once again... They are now willing to swap my meter for a new one to see it that makes it work.

But still think it would be nice to find some knowledge on the encryption on the KMP protocol. Will continue my hunt on this.

ArnieO commented 3 years ago

That is good news! As mentioned, this was eventually the solution for another user having the same problem.

The grid company cost of replacing a meter must then be lower than the cost of getting a technician at Kamstrup push a few buttons and push out a firmware upgrade (the meters are upgraded wirelessly). Smart contracting by Kamstrup! :wink:

ArnieO commented 3 years ago

@ArnieO is the meters RX pin connected on Pow-K at all?

No, it is not connected. I recently discovered documentation stating that there is an 82kohm pullup resistor in the meter on this input - but until now I had no information indicating that it can be used for anything. Being able to talk to the meter via an input is usually considered a risk (of course depending on the meter firmware). IMHO it is as a principle a bad idea.

In a future PCB relayout it could be connected to a GPIO - if this issue prevails. I propose we wait and see if there is more feedback on this issue from Danish customers.

heinekmadsen commented 3 years ago

So the meter guy showed up and we had a long chat about the issues.

We actually ended up not changing the meter, since the info from Kamstrup stated that I wouldn't work changing the meter as the grid company would still need to buy access to that module in Vision Air. So to keep (if any left) good will from the grid company we decided not to change it.

heinekmadsen commented 3 years ago

Well... here's dead-end https://www.gurux.fi/comment/13590#comment-13590

ArnieO commented 3 years ago

Well, the grid company is by law obligated to deliver a signal that can be exploited using "åbne standarder". I believe that is the point to put pressure on if you want to pursue the case with them. Your situation sounds unacceptable to me if my understanding of Danish law on this point is correct.

But please feel free to return the Pow-K and get a refund if you cannot use it!