Authentication fails

[runemoennike]

Hi

Suddenly yesterday the integration started complaining that it could no longer authenticate, and as such the readings cannot be updated :( Is anyone else experiencing this?

The log entry is as follows:

Logger: custom_components.novafos.coordinator
Source: custom_components/novafos/pynovafos/novafos.py:137
Integration: Novafos ([documentation](https://github.com/kpoppel/homeassistant-novafos), [issues](https://github.com/kpoppel/homeassistant-novafos/issues))
First occurred: 15:08:11 (250 occurrences)
Last logged: 20:35:36
Unexpected error fetching Novafos data: Code not retrieved correctly. Plugin will not continue login process. Check user/pass/supplierID

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 205, in _async_refresh
    self.data = await self._async_update_data()
  File "/config/custom_components/novafos/coordinator.py", line 41, in _async_update_data
    if not await self.hass.async_add_executor_job(self.api.authenticate):
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/novafos/pynovafos/novafos.py", line 137, in authenticate
    raise LoginFailed('Code not retrieved correctly. Plugin will not continue login process.  Check user/pass/supplierID')
custom_components.novafos.pynovafos.novafos.LoginFailed: Code not retrieved correctly. Plugin will not continue login process.  Check user/pass/supplierID

What I've tried so far:

1. Restarted HA as well as the host device.
2. Uninstall and reinstall the integration.
3. Change the password at Novafos to make sure it was the right one.
4. Checked the supplier ID to make sure it hasn't changed.

All to no avail.

Now that the integration has been removed, it seems that it's no longer putting the failed login attempts in the log. Is there some other log settings I can change or code I can add to log these attempts?

The only log line I have for each attempt is:

2022-09-15 20:41:33.231 DEBUG (SyncWorker_5) [custom_components.novafos.pynovafos.novafos] Got login form request_verification_token = (155 character token here)

[kpoppel]

A quick search says screenshot may be possible: https://code-maven.com/slides/python/selenium-headless-screenshot, which is a requirement in case this feature is caught up in the machine. This tutorial might be something to try: https://linuxhint.com/chrome_selenium_headless_running/

It could be a pretty cool optimisation for resource use to let go of the UI if possible. Maybe also the dependency on the selenium-docker could be scaled down to just installing Chrome, chromedriver, python3 and the appropriate selenium packages.

[achton]

Are any of you familiar with Cypress? I believe it solves some of the same problems as Selenium, and also runs in a (headless) browser, but being more modern it may have a more performant approach. Just a thought.

Cypress has readily made Docker images and can also be set up to measure its own resource usage from inside the container.

(Also: excellent work and findings in this issue 👏🏻 )

[kpoppel]

I haven't seen Cypress before. I am unsure if it is 'just' another way to drive Chrome, and that it is Chrome which make up the bulk of the memory footprint.

[kpoppel]

One could only hope.... I got a mail from the team leader for customer support with a notice that he sent on my request for API access to Novafos metering data to their project department. For now, the most reliable path it seems is to manually copy in the token in Home Assistant.

Keep complaining to Novafos (and any other KMD Easy-energy customers) - only in numbers will this get attention.

[runemoennike]

Just a small update: I was playing around with getting the docker container running directly on the RPi4, which seems to work... to a degree. I've enabled the headless mode, but the integration itself doesn't seem able to authenticate through the add-on, throwing the error "Failed to retrieve bearer token (no JSON returned)".

I don't have a way to connect to the docker image since it's running on the HA RPi, and going to the "web ui" for the add-on just shows "internal server error". I couldn't see anything in the add-on log for either error.

I'll play around more another day :) My progress so far is in this branch if anyone wants to play: https://github.com/runemoennike/homeassistant-novafos/tree/arm64-headless

[kpoppel]

I have a few issues with this at the moment. One is that a new header is required now. This is being fixed right now.

As for the docker venture here it is something which is trying to fight Google's AI and clever developers figuring out if a browser is machine or human controlled. Let's just say this is a battle I cannot win by any length. Last time I checked on my docker image status the browser would consistently display blank pages and hence never get anything to login with. some suggests randomly choosing a browser meaning more stuff in the container, and various things with full screen UI, mouse movement simulators and what not. In the end KMD are the ones to be poked from as many people as possible to allow customers to generate an access token to allow API accesses without requiring a human at the keyboard.

If you find a way to do this login, please share it. I have spent a lot of hours and nights on this just to get to a solution which worked a month or two :-) Also do make a merge request to this repository so that more can get the benefits.

rant on It is simply ridiculous that this is not possible when we can get heating data, electricity data, car data (for EVs anyway), and so on using 3rd party software like this one through API access tokens. For all I care they can revoke the tokens every month, but let us have the darned token. rant off

[kblin]

I guess I'll be sending an email to KMD as well. They clearly have a setup for Watts, it's silly they don't offer that for other tools.