Unauthorised errors

[Meatballs1]

Was working fine, appear to get an unauthorised error since this morning

Logger: custom_components.hypervolt_charger.hypervolt_api_client
Source: custom_components/hypervolt_charger/hypervolt_api_client.py:164
Integration: Hypervolt Charger (documentation, issues)
First occurred: 06:43:29 (1 occurrences)
Last logged: 06:43:29

Update_state_from_schedule, unauthorised

I've not reset my creds and my app is working fine. Received the same after a Ha restart.

Wonder if HV have changed something (maybe intentionally blocked if it was churning through a lot of data), or something related to tariffs update or recent downtime of the energy usage.

[exalyte]

can confirm mine also seems to have stopped responding i CAN send lock command but nothing else worked

Here is the HV log for the past 32 hours;

working 2022-12-24 00:22:11.817 WARNING (MainThread) [custom_components.hypervolt_charger.hypervolt_api_client] notify_on_hypervolt_sync_push unknown message structure: {"jsonrpc":"2.0","id":"1671841331.675049","error":{"code":409,"error":"Concurrent modifications invalidated this request","data":null}} Stopped working 2022-12-25 22:28:46.827 ERROR (MainThread) [custom_components.hypervolt_charger.hypervolt_api_client] Send_message_to_sync cannot send because websocket_sync is not set 2022-12-25 22:29:37.835 WARNING (MainThread) [custom_components.hypervolt_charger.hypervolt_api_client] Update_state_from_schedule, unauthorised 2022-12-25 22:31:02.787 ERROR (MainThread) [custom_components.hypervolt_charger.hypervolt_api_client] Send_message_to_sync cannot send because websocket_sync is not set 2022-12-25 22:31:11.191 ERROR (MainThread) [custom_components.hypervolt_charger.hypervolt_api_client] Send_message_to_sync cannot send because websocket_sync is not set 2022-12-25 22:31:18.627 ERROR (MainThread) [custom_components.hypervolt_charger.hypervolt_api_client] Send_message_to_sync cannot send because websocket_sync is not set 2022-12-25 22:31:35.072 ERROR (MainThread) [custom_components.hypervolt_charger.hypervolt_api_client] Send_message_to_sync cannot send because websocket_sync is not set 2022-12-25 22:32:22.935 WARNING (MainThread) [custom_components.hypervolt_charger.hypervolt_api_client] Update_state_from_schedule, unauthorised 2022-12-25 22:32:44.045 ERROR (MainThread) [custom_components.hypervolt_charger.hypervolt_api_client] Send_message_to_sync cannot send because websocket_sync is not set 2022-12-25 22:32:49.447 ERROR (MainThread) [custom_components.hypervolt_charger.hypervolt_api_client] Send_message_to_sync cannot send because websocket_sync is not set

[exalyte]

Manually probing the API i can get it to work again by relogging in, going to delete the integration and try again....

//

same result i can lock the charger but changing charge mode or forcing charge to start wont work, LED changes dont reflect in the app, its freaking cold outside so not gone to check manually

[Kahn]

I'm running into the same thing now with unauthorized errors even after resetting my creds and uninstalling and reinstalling the integration. I suspect Hypervolt have changed something, and if they have I'll be extremely unhappy.

[gndean]

Thanks for the info. I see the same with my main HA but the integration runs fine when run on a fresh developer HA install. So it's almost certainly a bug in the integration and not anything Hypervolt have done. Probably something expiring after a period of time that needs to be refreshed. I am surprised a reboot of HA or reinstall of the extension doesn't resolve it though.

Investigating 👀

[gndean]

OK, so we are being throttled. The particular API endpoints used for most of the functionality (but not setting the lock state, for example) are refusing connections from the integration at the moment. This would explain why we've all seen connection issues around the same time. I can confirm that Hypervolt are aware of this Home Assistant integration and have reached out to me personally, in a very constructive manner. So I will contact Hypervolt and see what can be done.

In the meantime, I would request everyone be respectful and patient, and assume best intentions all round. ❤️

I'll update this issue when I have any further information. It's also possible, that if the throttling is automated, that it will be lifted and start working again at some stage but until we've had word from Hypervolt themselves, we shouldn't assume a permanent fix, we should be respectful of their infrastructure, and not try and work around the throttling.

[gndean]

Hypervolt have been super responsive to my contact with them, even on boxing day, so huge kudos to them 👍 The integration is causing a large number of errors seen on their APIs. It must be generating much more error traffic than valid traffic and they've had to block this, but the good news is that they would like to support it. We'll be working to resolve this, so there will be a solution. We have to be mindful that we're the uninvited guest here and I'm sure the HV staff have a bunch of other stuff they ought to be working on. Thanks for your patience and support.

[emmet02]

Fair play. Any benefit in us disabling the add on for now? Or are they comfortable that they've "handled" us for the time being.

[BadgerLoaf]

Great work @gndean, and good to hear that Hypervolt are being positive and engaged. My interactions with them have always been good from a technical side (have had a couple of support tickets where I have tried to be 'clever' with what I can do with them).

Your approach is 100% correct too, we should respect their infrastructure (and throttling) and work with them, not against them.

Kudos to you all, and if you need anything from us, please let us know.

[boab21mac]

Thanks for the update, kudos to you and the Hypervolt team in working together

[BadgerLoaf]

Have seen the beta code @gndean ... will test out and let you know how I get on

[BadgerLoaf]

.... have updated to v1.0.5 Beta, and it seems to be connecting and retrieving info ok at the moment. Will continue to test it later today when I have some more time.

Great work!

[gndean]

Well spotted! Yes, there is a beta which hopefully brings the integration back to life and is kinder on the Hypervolt APIs. It's just beta for now until we have some confidence it's good. You can install via HACS but will need to manually re-download the integration via the HACS UI and choose to show Beta versions. If you're not comfortable with that, don't worry, there should be a full release coming soon.

[BadgerLoaf]

Yeah seems to be working well. Have created some automations to change the Charge Mode and LED brightness, and they appear to be working without issue. Will see how it behaves over the next 24-48hrs. Thanks again for all your work on this, have been hoping someone might be able to get an integration working for this. Cheers!

[Kahn]

1.0.5 beta working ok here initially at least.

I put my earlier grumpiness down to a six week old, I've clearly forgotten about infra problems months ago 😆

[BadgerLoaf]

All working ok here still. Got some automations set up (changing between Super Eco and Boost, changes in LED Brightness and using the charging status as a trigger), and it all seems to be working and reporting well.

[gndean]

OK, thanks for the feedback @BadgerLoaf , @Kahn. I've promoted the v1.0.5 Beta to a full release now. The code remains the same as the beta. I'll keep this issue open for now but hopefully it can be closed soon 🤞

[gndean]

Sorry everyone, the issue has recurred and the extension has been blocked again. Stand by for updates.

[BadgerLoaf]

Thanks for the heads up @gndean…. I noticed it earlier but hadn’t had a chance to take a look

[gndean]

I think I have a fix for the hammering of the APIs issue (although I have believed that before!). It's being validated with help from Hypervolt now. So I'm hopeful there will be a new release to share before too long.

[BadgerLoaf]

Thanks for the update and all your work!

[crobbins215]

Fingers crossed

[maegibbons]

Any positive news? Are Hypervolt being helpful?

[gndean]

Hmm, a bit disappointing. I've been running, what I hope is, a fixed version since last weekend and HV volunteered to check the logs for my integration and account. But haven't managed to get round to anything this week. And they've asked for me not to release it.

[Puntoboy]

I’m starting to get the impression that all the dev is going HV3.0. Smart home integration is included there apparently. On 13 Jan 2023 at 21:54 +0000, gndean @.***>, wrote:

Hmm, a bit disappointing. I've been running, what I hope is, a fixed version since last weekend and HV volunteered to check the logs for my integration and account. But haven't managed to get round to anything this week. And they've asked for me not to release it. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.***>

[BadgerLoaf]

Thanks for the update @gndean, hopefully they can get back to you next week. Thanks for all your work on this. Won’t lie I’m already missing having this working (even if it’s only new), it makes my integration with Octopus Intelligent so much more useful. So fingers crossed for next week

[maegibbons]

Couldn't agree more! It is a vital missing link!

[Kahn]

Thanks for the update @gndean

I get the internal development priority issues, but at this point as well, you really have to wonder what is going on with the SRE's at Hypervolt if they're running this close to maximum and the real throttle is just that no one is opening their app to change things frequently rather than proper traffic management controls. Scaling for usage shouldn't mean turning of your clients. If anything it probably underscores the importance of local API's to just avoid this entirely.

[itsjxck]

I have been watching this to see what happens, and have reached out to Hypervolt myself with regards to API access in order to automate the schedule of my charger to align with Octopus Agile pricing. Ultimately it has been a disappointing exchange, after asking for documentation I was initially told "Unfortunately, we do not have any documentation regarding API integration with the Hypervolt just yet." When I asked for just specific information on the API endpoints to facilitate what I needed, I was then told the development team "have decided to go a different direction."

[gndean]

I've still not had any feedback from HV so I'm reverting to seeking forgiveness rather than permission. I've released 1.0.6 as a Beta for now to limit the number of users who'll pick it up. Let's see how it goes 🤞

You can install via HACS but will need to manually re-download the integration via the HACS UI and choose to show Beta versions.

Thanks for all the feedback and your patience.

[mpredfearn]

I have been watching this to see what happens, and have reached out to Hypervolt myself with regards to API access in order to automate the schedule of my charger to align with Octopus Agile pricing. Ultimately it has been a disappointing exchange, after asking for documentation I was initially told "Unfortunately, we do not have any documentation regarding API integration with the Hypervolt just yet." When I asked for just specific information on the API endpoints to facilitate what I needed, I was then told the development team "have decided to go a different direction."

Hopefully the different direction is allowing local control via MQTT, as has been long promised, rather than via the cloud API. One can always dream ;-).

[maegibbons]

Many thanks.

Downloaded, installed and working.

Disappointing that they are not engaging. The promise originally of MQTT was one of the main reasons that I bought HV. Your integration through the API is a vital lifesaver so thank you very much.

Krs

Mark

[Puntoboy]

I've still not had any feedback from HV so I'm reverting to seeking forgiveness rather than permission. I've released 1.0.6 as a Beta for now to limit the number of users who'll pick it up. Let's see how it goes 🤞

You can install via HACS but will need to manually re-download the integration via the HACS UI and choose to show Beta versions.

Thanks for all the feedback and your patience.

Thank you. I'll install this shortly.

Really disappointed with HV. I moved to them from Ohme due to issues like a lack of development. They promised they were different and Flavian often slates the like of MyEnergi. But after all this, they are really no different.

[BadgerLoaf]

Thanks @gndean

Have downloaded this and tested, and it appears to be working for the time being.

Whilst i am as frustrated with Hypervolt as anyone else (both with support of this integration, and with the wider development of both their product and API's); the only way we can make progress is to work with them, not against them. So thank you @gndean for all your effort, time and work on this.

However, with the above in mind, I'm just wondering whether it would be worth not having the Beta available via HACS, and instead only available from Github as per the previous beta. Allows us to help test it, but show some goodwill to HV in giving them time to support you? Granted, if they don't get back to you with anything this week, then I think you have given them a good enough chance, and releasing it as a Beta in HACS makes sense........ Anyhow, just any idea, maybe I'm even being a Devils Advocate, as I feel pretty conflicted in my thoughts on HV!

If there is anything we can do to support you in getting HV to help you please let us know. I'm happy to raise a support case or similar to add weight to your ask.

I guess we all need to be aware that this isn't an official integration, and so we are working on their goodwill... though I would feel more likely to give them that benefit if I felt they were putting their effort into their own developments (which they might be, but we constantly are let down and don't see anything come through).

... sorry, that turned into a bit of a rant from the frustration!

[gndean]

Ramble alert! Just some notes relating to the approach in developing this integration and my philosophy, if that's not too grand a term.

Having recently discovered the world of Home Assistant, I wanted to automate some features of my Hypervolt charger to optimise its use for my solar + home battery combo. So I researched a little and discovered that no suitable solution already existed. As a personal project and challenge for myself, I learned how to create a Home Assistant integration and figured out how the Hypervolt APIs worked. Once I had an integration that was working for me, I debated on whether making this public and how Hypervolt might react. A danger would have been that Hypervolt wouldn't approve any unofficial use of their APIs, prevent unauthorised access thus shutting down the option of a HA integration. However, my hope was that they would be receptive and the potential benefits to many seemed to outweigh the risk to me losing my personal project. At the very least, I would have learned something I thought. So, I posted in the Hypervolt Facebook forums, received some positive feedback from users and to my surprise, received a like from the CEO of Hypervolt. Hurah!

A couple of weeks after the first release, it stops working and this issue on GitHub is raised. On investigation, I found the APIs were returning a HTTP 429 error which means "Too Many Requests" and is the response servers give if a client is being throttled, blocked, or banned. I contacted Hypervolt and received a response from the CTO showing some charts of their API usage spiking, which immediately dropped back to normal after this integration was banned.

The integration was spamming the APIs. My fault for dodgy coding.

I thought I understood the issue, released a fix (1.0.5), along with some other hints from the CTO, but it recurred and the integration, quite rightly, was banned again. I reviewed the code again and found another scenario where the connections to the APIs could build up over time and I hope that 1.0.6 will now fix this properly.

Rather than just publicly release 1.0.6, I contacted Hypervolt's CTO and asked how they would prefer me to manage the release, so that they can monitor any impact it may have. Hypervolt then offered to check the data from my use of the integration (because I was using 1.0.6). They offered to check within the week but ultimately have been too busy. The next week has rolled around and today I've taken the decision to release it as a beta, so that you guys can benefit, we can all generate some more data to be monitored, but most Home Assistant users will not be notified of the release so this should keep the user count low for now. (By the way, I'm not aware of a way to release it to GitHub without also releasing to HACS. My understanding is that HACS tracks any releases made in GitHub.) I have informed Hypervolt I've done this today. It was a difficult decision to release anyway, without explicit approval, but I feel it's necessary to make progress. Without feedback from Hypervolt themselves, and even then, it would only be from one user (and I don't think I've personally seen what I think is the issue with the API spamming) I don't feel it's progressing. Without the release, nobody can benefit from the integration. With the release, there's a chance people will. I’ve taken a punt and I hope it works out.

I'm hugely grateful for Hypervolt for being open to this third-party integration. I'm the uninvited guest to their party and they didn't just kick me out or call the police. It would have been easier for them to do so. They've just banned me from the beer supply until I can learn to moderate my use!

They are busy, already have a packed roadmap and have had to deal with legislation changes over the last year. They don't need people like me making their life more difficult and potentially impacting their core product. That said, if the integration can be made to behave cooperatively, I'm hoping it can show that third-party support enhances the product and helps address some of the feature requests that they have not been able to implement themselves yet. And no doubt people will find creative uses for it. I believe Hypervolt see this too and that's why they've been open to helping with its development.

Hmm, that's probably enough waffle. I hope this helps clarify why we're here. Fingers crossed for the future and stay positive. Thanks for your support.

[ashleycbaker]

Thanks for the overview and thanks for taking the harder route to make it available for us all. I have found it crucial to my setup!

I hope too the HV Team see the positives it brings to the user base and help support it going forward.

I can also confirm the beta is working this end.

[BadgerLoaf]

Ramble alert! Just some notes relating to the approach in developing this integration and my philosophy, if that's not too grand a term.

Having recently discovered the world...............

Well said @gndean, and for what it's worth I believe you have the right approach on this.

As said, their developers will already be tied up in their roadmap, and so support/advice on this integration is on goodwill - especially with the potential for it to affect their core product, and could have easily just have closed it down.

Separate from your work - I believe/hope that Flavian and the Hypervolt team are working hard on the product; but as users we are getting a little frustrated, as there has been alot of promises, but little publicly visible progress (Fingers crossed 2023 will put that to bed). But we (and I include myself) shouldn't let that frustration affect our view on their help/support on projects like this.

Thanks again @gndean for all your work on this, and sharing it with the community, and also to the Hypervolt team for supporting your efforts.

I believe in the Hypervolt product and team, and hope that we can support them by getting things like this working for the more automation mind folks.

PS: v1.0.6 is still working ok for me! 😀

[ashleycbaker]

@gndean Do I dare say, its stopped working again this end? The dreaded Invalid Login error again.

Edit: it's all fine again, must of been something this end 🙈

[Puntoboy]

Still working here :)

Thanks @gndean for the update and I do appreciate your points.

I have little sympathy for Hypervolt though,

[crobbins215]

Will pull the beta now and give it a spin

[gndean]

In the absence of any log information from Hypervolt on how 1.0.6 is performing, what would be helpful would be if some of you could share your log files and I can check for any potential hammering of the APIs. You'd need debug logging enabled for the Hypervolt integration and ideally at least 24 hours of logs, say, with at least one charging session.

Debug logging can be enabled like this or temporarily via the HA UI in Settings / Integrations / Hypervolt Charger ... Enable Debug Logging. When Disabling Debug Logging via this same method, a log file is downloaded.

If you don't want to post the logs here, you can email me direct at graham@gndean.com

Thanks!

[Puntoboy]

I've just sent you a link to my log. I couldn't attach it to the email as it's 24MB.

[BadgerLoaf]

In the absence of any log information from Hypervolt on how 1.0.6 is performing, what would be helpful would be if some of you could share your log files.....

Sorry been flat out with work for a few days, will set up the logging today and then get it over to you.

[gndean]

Thanks to @Puntoboy and @maegibbons for sending the logs - very useful.

From what I can see, the integration is behaving and handling being disconnected from HV APIs as intended. One thing I noticed is that all three of us were disconnected from the HV websocket API at the same time, then the integration attempted to reconnect, as designed, with an increasing delay. However, since all instances of the integration would attempt to reconnect at the same time, this could, at scale, cause a spike on the HV APIs. So to ameliorate this, I've released a 1.0.7 beta with an improvement in this area by adding a random delay. It also adds some more detail to the logging in the case where the websockets are disconnected.

Feel free to update to 1.0.7 beta when you like. You shouldn't notice any difference and I think 1.0.6 is actually fine. I'd also be happy to receive any further logs for further checks, from 1.0.6 or from 1.0.7. Thanks!

[BadgerLoaf]

Thanks @gndean, have just sent you 1.0.6 logs and will install 1.0.7 now and set up logging again.

Thanks for all your work on this!

[maegibbons]

Have just updated to 1.0.7 and re-enabled debug logging.

Will send to you after 24 hours or so.

Krs

Mark

[gndean]

Good news: the Hypervolt CTO has confirmed that no additional traffic has been seen so far from their end, so I've released 1.0.7 now. This release is exactly the same code as the 1.0.7 beta.

For those collecting logs, I would still appreciate any you have to share to double check things are still behaving and pre-empt any further issues. Thanks!

[gndean]

Hopefully not tempting fate, but I'm closing this issue now 🤞 I'm still happy to check any logs one final time.