Authentication fails

[runemoennike]

Hi

Suddenly yesterday the integration started complaining that it could no longer authenticate, and as such the readings cannot be updated :( Is anyone else experiencing this?

The log entry is as follows:

Logger: custom_components.novafos.coordinator
Source: custom_components/novafos/pynovafos/novafos.py:137
Integration: Novafos ([documentation](https://github.com/kpoppel/homeassistant-novafos), [issues](https://github.com/kpoppel/homeassistant-novafos/issues))
First occurred: 15:08:11 (250 occurrences)
Last logged: 20:35:36
Unexpected error fetching Novafos data: Code not retrieved correctly. Plugin will not continue login process. Check user/pass/supplierID

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 205, in _async_refresh
    self.data = await self._async_update_data()
  File "/config/custom_components/novafos/coordinator.py", line 41, in _async_update_data
    if not await self.hass.async_add_executor_job(self.api.authenticate):
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/novafos/pynovafos/novafos.py", line 137, in authenticate
    raise LoginFailed('Code not retrieved correctly. Plugin will not continue login process.  Check user/pass/supplierID')
custom_components.novafos.pynovafos.novafos.LoginFailed: Code not retrieved correctly. Plugin will not continue login process.  Check user/pass/supplierID

What I've tried so far:

1. Restarted HA as well as the host device.
2. Uninstall and reinstall the integration.
3. Change the password at Novafos to make sure it was the right one.
4. Checked the supplier ID to make sure it hasn't changed.

All to no avail.

Now that the integration has been removed, it seems that it's no longer putting the failed login attempts in the log. Is there some other log settings I can change or code I can add to log these attempts?

The only log line I have for each attempt is:

2022-09-15 20:41:33.231 DEBUG (SyncWorker_5) [custom_components.novafos.pynovafos.novafos] Got login form request_verification_token = (155 character token here)

[kpoppel]

I have the same. Not blocked from logging in using the website, so I expect some change in the login procedure. Getting to the current one was not straight forward, so I hope any change made is not in the same league.

[kpoppel]

It seems a Google reCAPTCHA token was added to the login process. https://developers.google.com/recaptcha/docs/verify If it is important I don't know, but it is different. If the purpose is to hinder replay attacks it could be important for the login procedure. For now, disable the plugin - it will not work with this.

[runemoennike]

Argh! I looked around but I can't find any mention of an official API, neither from Novafos or the "Watts" app they refer to.

Edit: I can't see a captcha on the website login. But you see it in the call from python?

[kpoppel]

I have left a message on KMD easy-energy for their sales director:

Som privat forbruger ønsker jeg at kunne hente mine data ud af easy-energy via software. Lige nu ser det ud til at det kun er f.eks. vandværker eller andre professionelle aktører der har adgang til KMD APIet. F.eks. har jeg som privat adgang til at generere et application-token til https://eloverblik.dk/welcome og den vej via deres API hente mine egne data ned via software. Har eller vil KMD tilbyde denne mulighed?

For at give lidt baggrund, findes der et voksende community omkring selv-hostede automatiseringsløsninger i hjemmet, herunder HomeAssistant, hvor integrationer samler folks egne data. Sådanne integrationer driver besparelser for de enkelte på tværs af alle forsyningspunkter. Åbne APIer med f.eks. application-tokens gør det muligt at skrive disse integrationer.

Jeg håber der eksisterer noget, eller at KMD er åbne overfor at lade os hvis data I opbevarer tilgå disse på måder der giver mening i et samlet forsyningsoverblik hos den enkelte.

These are my data, and I really find it reasonable I can get access to them in ways I find meaningful. If others could leave similar messages, maybe some exposure to this could happen.

[runemoennike]

Thank you.

I remember reading that initially there was no API access to the electricity consumption data either, but after a long and hard fight they agreed to open it up. Hopefully this won't be as difficult.

[kpoppel]

Edit: I can't see a captcha on the website login. But you see it in the call from python?

As for this comment. I see this in the network traffic: (new) (new) (new) (new)

The (new) parts revolve around this reCAPTCHA. Particularly the 2nd Login package submits formdata including email and password, and another hidden field I suspect is to avoid spoofed homepages (well done KMD), but now also two more fields "Input.CaptchaToken" and "g-recaptcha-response" I haven't seen before.

[runemoennike]

Yes I see that too, what a bother.

[runemoennike]

I wonder if there's a way to use the "Watts" app's backend instead. It would require sniffing the traffic from the phone or via an emulator... I might give it a shot one day.

[kpoppel]

I personally really dislike Google is now part of knowing I login to see my water data.

[kpoppel]

Watts.dk is currently only getting energy data from energinet.dk, and only customers on Aalborg Forsyning can follow their water usage, see https://watts.dk/funktioner/#alle

According to this page: https://www.kmd.dk/loesninger-og-services/loesninger/energi/kmd-easy-energy

SK Forsyning, Forsyning Helsingør, Frederikshavn Forsyning, Aarhus Kommune (AVA), Bornholms Energi og Forsyning, Fors, NK Forsyning, Silkeborg Forsyning, Arwos, Provas, Jysk Energi, Energistyrelsen, Svendborg Vand&Affald, Ørsted, Novafos (this integration), and Vestforsyning, and more use KMD easy-energy platform.

Having to walk through Watts to get to the API endpoint could be an option, just not a very direct one :-)

[runemoennike]

Hm I can follow my water usage via Watts, so I wonder how that works.

[kpoppel]

In any case I tried blocking the gstatic domain in the browser and attempt login. It immediately responds with "Not sure you are human" and shows the login page again. So I am pretty sure this is the core of this issue. Now to learn how to fix this... If it can be done in a browser it can be done in code.

[kpoppel]

I tried Watts app too now, indeed Novafos is the 2nd water supplier available. Could be interesting to know how they integrate.

[kpoppel]

I think the best I can do right now - as the very purpose of reCAPTCHA is to weed out automated access - is to allow pasting the Bearer token into Homeassistant, after which the integration can update data freely. This utilises the fact that logging into the service results in a 3600 second token from which the service can be used. The token can be used by any service on the network. The just pushed update is half-way there I think. Data can still be imported into HA, but only after manually logging into the web-service and snipping the access token.

[PexJensen]

I can't get the integration to work, say's "Token update does not seem to have a valid length. Please check again. (This message is normal the first time the integration starts)"

My access_token is 1977 characters long? and not 1506 as you write in your documentation.

[kpoppel]

Ok

[kpoppel]

Okay, thanks for the update. I will set the limit to be at least 1500 characters then. It is just to keep possible paste errors out.

[kpoppel]

Release v3.0.1 allows longer token.

[PexJensen]

Works for me now.

Thanks.

[Sdahl1234]

I removed the integration and its not possible to add it again.

[runemoennike]

@kpoppel should this issue be kept open so people won't keep reporting it, until we have something from Novafos?

I wrote Novafos and the first reply hints they are unaware a captcha has been installed. I elaborated and now waiting for their reply again.

[kpoppel]

That is a very good idea.

[runemoennike]

I had no luck with Novafos unfortunately. They were very nice to try and contact KMD to try to have it removed, but KMD refused because it "prevents spam logins."

Sad.

[kpoppel]

I have left a note with Novafos kundeservice today as well (KMD sales earlier, but no response). It is not great that a commercial product like the Watts app can gain problem-free API access to data, whereas we cannot. The absolute non-sense is that as soon as the Bearer token is in hand, the integration works just like before. We just need that token.

[runemoennike]

I'm guessing Watts pay for that access, but I agree it's ridiculous that they actively prevent free use of our own data.

A potential next step is to contact Datatilsynet.

[kpoppel]

I left a message with "identiy@kmd.dk" from coming across this: https://github.com/kmd-identity . It mentions OpenID Connect, which is how the login process is handled - disregarding the reCAPTCHA part. If we're lucky they at least know of there is a path leading to authentication which does not require basically replaying the homepage in software.

Let's see.

Also this looks interesting: https://github.com/kmd-identity/documentation/wiki/OpenID-NativeApp-GetStarted

[kpoppel]

I got an answer back on my request. Basically the change from KMD side is to hinder abuse - which is fair enough. Given increased hacking activity on Danish infrastructure it is the good old equation "convenience = 1/security", and securing our infrastructure can only be appreciated.

I was advised to acquire an optical eye to readout data. This is a way of-course, and Kamstrup does have the hardware:

• https://www.kamstrup.com/en-en/water-solutions/meters-devices/accessories/optical-readout-head/documents
• https://www.kamstrup.com/en-en/water-solutions/meters-devices/accessories/pulse-adapter
• https://www.kamstrup.com/da-dk/varmeloesninger/intelligente-varmemaalere/accessories/common-usb-driver
• Even a mbus<->wmbus adapter: https://www.kamstrup.com/en-en/water-solutions/meters-devices/accessories/wireless-mbus-dongle

I am pretty sure there are more 'open source' alternatives except for the likely proprietary changes made, or encryption of data.

I came across this: https://www.ihc-user.dk/forum/forums/topic/6640-flowm%C3%A5ler-til-brugsvand/ and this product: https://www.relay.de/en/products/m-bus-accessories/optical-head/

After the advice to get the optical eye, I have requested some more information on this: any encryption keys needed mbus/wmbus, as the Kamstrup optical eye has a wired output for the pulse which then needs another running ESP-device or something to pickup these pulses and transmit them to Home Assistant (MQTT comes to mind) and connecting the pulse output to in interrupt to wake up the ESP. but having a USB connected eye or simply reading the wireless signal with a wmbus dongle is just... easier.

Maybe this is for getting the encryption key directly from Kamstrup: https://www.kamstrup.com/en-en/-/media/files/my-kamstrup-guides/encryption-key/my-kamstrup-encryption-key-guide_en.pdf

DIY-project: https://wiki.hal9k.dk/projects/kamstrup

[askpet]

I got an answer back on my request. Basically the change from KMD side is to hinder abuse - which is fair enough. Given increased hacking activity on Danish infrastructure it is the good old equation "convenience = 1/security", and securing our infrastructure can only be appreciated.

I was advised to acquire an optical eye to readout data. This is a way of-course, and Kamstrup does have the hardware:

• https://www.kamstrup.com/en-en/water-solutions/meters-devices/accessories/optical-readout-head/documents
• https://www.kamstrup.com/en-en/water-solutions/meters-devices/accessories/pulse-adapter
• https://www.kamstrup.com/da-dk/varmeloesninger/intelligente-varmemaalere/accessories/common-usb-driver
• Even a mbus<->wmbus adapter: https://www.kamstrup.com/en-en/water-solutions/meters-devices/accessories/wireless-mbus-dongle

I am pretty sure there are more 'open source' alternatives except for the likely proprietary changes made, or encryption of data.

I came across this: https://www.ihc-user.dk/forum/forums/topic/6640-flowm%C3%A5ler-til-brugsvand/ and this product: https://www.relay.de/en/products/m-bus-accessories/optical-head/

After the advice to get the optical eye, I have requested some more information on this: any encryption keys needed mbus/wmbus, as the Kamstrup optical eye has a wired output for the pulse which then needs another running ESP-device or something to pickup these pulses and transmit them to Home Assistant (MQTT comes to mind) and connecting the pulse output to in interrupt to wake up the ESP. but having a USB connected eye or simply reading the wireless signal with a wmbus dongle is just... easier.

Maybe this is for getting the encryption key directly from Kamstrup: https://www.kamstrup.com/en-en/-/media/files/my-kamstrup-guides/encryption-key/my-kamstrup-encryption-key-guide_en.pdf

WOW - This is really great work Thanks!

[runemoennike]

Good work :) Am I understanding correctly that the meter sends out information regularly via wmbus already?

[kpoppel]

Yes this is what I have concluded. In the answer from Novafos it is hinted that while I can attach an optical eye they reserve the right to take up that space for attaching an antenna if they cannot 'get the signal out of the building', as they write. From this I conclude wmbus is used. I have tried to register an account with Kamstrup to register my water meter, but the device is flagged as not open for retrieving the key for it that way. I am awaiting answer from Novafos on this as they certainly have the access (it is their device after all) with reference to the fact that Radius will give out the key for the electricity meter on request.

[runemoennike]

Alright, very interesting.

I've heard a rumor that the meter uses the electricity meter (also from Kamstrup) to communicate long-distance. I guess that makes sense since I'm assuming the wmbus is short range only. I have a device in my electricity meter that reads out electricity usage -- it'd be interesting to see if there was a way to read out water usage as well that way, if the data is already going through the electricity meter.

Edit: I've sent an email to the guy who supplied me with the keys to read the electricity usage, to ask if he knows anything about reading water usage.

Update: Got a reply. Apparently the electricity meter does not forward communication for the water meter, at least not in the setup we have here.

[Sdahl1234]

Im using wmbusmeters (https://github.com/weetmuts/wmbusmeters) add-on to read my multical21 meter. not much data but i get it live. Now i can blame who's using all the water You need the encryption key for your meter

[kpoppel]

@Sdahl1234 Very nice indeed. Mine is an flowIQ 2200 Novafos branded. What hardware did you buy and how did you get the keys?

[Sdahl1234]

I have a IMST 871a (im871a) and i just asked my watersupplier (Frederikshavn forsyning A/S) for the encryption key. https://wireless-solutions.de/products/m-bus/im871a-usb/

My next goal is to get the key for my heating supplier, because I get readings from my Mulical602 to, but i need the key

[kpoppel]

Great. 50€ at shop.imst.de which is probably much less than Kamstrup will charge for their device.

€76,39 shipped.

I'll look into easier ways to use this plugin regardless, or at least wait it out for KMD to realize their end-users should be treated like eloverblik and eforsyning does (== well)

[Sdahl1234]

The add-on supports other dongles. I think you can flash old tv-tuner dongles, but i did'nt have the time to investigate a cheap solution. (too many IoT projects) I just picked IMST 871a because I was sure it supported reading both my water and heating meter AND i could get it in 3 days.

[runemoennike]

That'd be great. €76 is too much for me to justify spending on having this number on the dashboard, since I don't have any use for it other than curiosity.

Some day I'm going to look into how the Watts app fetches data as well. So far I managed to install it on an emulator, but I haven't had time to sniff data yet.

[kpoppel]

I tried installing Watts on the emulator. It seems to use SSL pinning so decryption using MiTM (like fiddler) is kind of difficult. Sources say decompile the APK and remove some safety settings - but that is also a little work. but do let us know if you get around to do this :-)

[runemoennike]

I managed to sniff the traffic by setting up the android emulator to use Fiddler as a proxy, as described here: https://docs.telerik.com/fiddler-everywhere/traffic/configure-android

I can see the login traffic, but the app throws an error on the reply from the server to the login request. Maybe it's due to the custom certificate. I'm trying to rebuild the apk to accept user certificates as described here https://book.hacktricks.xyz/mobile-pentesting/android-app-pentesting/make-apk-accept-ca-certificate .

[kpoppel]

It is possible using selenium test tool to automate the login. It needs either Firefox or Chrom(e/ium) to be installed on the system, but I have been retrieving tokens all day long, and verified the retrieved token does fetch data from the datahub by inserting it using the configuration method I made recently..

[runemoennike]

Oh that's interesting. Does it require a windowing environment or do you think it could run headless, so we can put it in an add-on perhaps?

[kpoppel]

Well getting it into Home Assistant in a HASSIO setting is less than easy because selenium-wire requires gcc to install, and that is not present on HASSIO. So I am looking into this:

• pull docker container selenium/standalone-firefox <- this one uses xrfb and exposes the screen through VNC, so that's cool.
• make another container with the necessary python-stuff in it and a small webserver which in turn dos all the selenium stuff on the other container. This service would then return the token back to the integration and the rest is as always.

I don't know how to make an add-on, but people running this on RPis would be out of the question, both because the selenium container sucks up 1 GB space and a lot of RAM. For now I am just looking at making a docker-compose file to prepare this on another VM somewhere. Caveats are that I have no idea of programming en´ven a simple Python web service, so need to learn that too. I just need one single endpoint http://:8123/tokenapi which returns a token

Btw. the captcha did come back with a "we cannot figure out if you are human" once, so this is also not a sure fire method. I hope though that polling once per 24 hours with some differences in polling time of day and time from filling out the form and "pressing Enter" would be enough to not trigger the robot-mode. Otherwise things will fail, I can dump a screenshot of what went wrong, and maybe a human login once in a while is enough. Let's see.

[runemoennike]

I think an add-on is just a docker container, but if it needs multiple docker containers idk if it's possible. The 1GB space wouldn't be an issue for people with an SSD on the RPi, but the ram might indeed be.

I've used Tornado to expose endpoints as a webserver with python and it was pretty simple iirc. I don't know if it's still the flavour of the day or if something better has come out since.

I was wondering if the captcha "learns" to recognise a bot if it does the same movement every time, so some variation might be smart as you say, maybe if "jiggle" the mouse a bit or something.

[kpoppel]

I have settled for building on top of the selenium/standalone-chrome:4.5.0 container. It turned out that was the easiest thing to do - basically adding the Flask Python code and getting supervisord to start it. So this is a single container with everything in it now. I still need to learn how to pass username and password in a GET, but it reacts nicely to hardcoded values now using curl on the endpoint. Basically forwarding the oidc token like we were used to.

So next step is to stitch this into HomeAssistant and pass the credentials properly.

[runemoennike]

Awesome!

Not sure if it's possible in this case, but normally username/passwords are passed via POST requests to avoid logging them to request logs, for obvious security reasons. Might not be so relevant here, if it all runs on a local network, but just thought I'd mention it since I have some experience in that field hehe.

[kpoppel]

Good point. I'll make it a Post. Too much to learn due to this 😁

[kpoppel]

I have committed what I have in this branch: v4-selenium It is a little rough around the edges, with required changes in a few files, but now it is here. Refer to the README.md for details.

[runemoennike]

Great work!

If it's not too much trouble, could you check how much RAM the docker container uses? I'm wondering if my NAS is able to run it at all.

[kpoppel]

docker stats --no-stream says:

CONTAINER ID   NAME                              CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
f569b9e5c4c7   selenium-pyapi                    1.27%     696.6MiB / 7.737GiB   8.79%     122MB / 363MB     211MB / 5.77GB    46

So it is quite hungry this one. I suppose the effect of running a full X-server and a Chrome browser makes it so unfortunately. But it works and has fetched data for me daily since I deployed the container.

[runemoennike]

Cool, thanks :)

I briefly read about running Chrome headless via Selenium, maybe it can even run without an X-server then. I haven't had time to experiment with it yet.

On Thu, Oct 27, 2022 at 12:51 PM kpoppel @.***> wrote:

docker stats --no-stream says:

CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS f569b9e5c4c7 selenium-pyapi 1.27% 696.6MiB / 7.737GiB 8.79% 122MB / 363MB 211MB / 5.77GB 46

So it is quite hungry this one. I suppose the effect of running a full X-server and a Chrome browser makes it so unfortunately. But it works and has fetched data for me daily since I deployed the container.

— Reply to this email directly, view it on GitHub https://github.com/kpoppel/homeassistant-novafos/issues/10#issuecomment-1293344689, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALTNAZDDGEPKVUI26HRKXLWFJNEBANCNFSM6AAAAAAQNVROJQ . You are receiving this because you authored the thread.Message ID: @.***>