Pentair integration becoming unavailable frequently

bwoodworth commented 1 year ago

The problem

Since upgrading to 2023.7.1 the Pentair integration become unavailable rather frequently. It is very intermittent, but happens many times throughout the day. here is an example of one of the switches:

What version of Home Assistant Core has the issue?

core-2023.7.1

What was the last working version of Home Assistant Core?

core-2023.6.1

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Pentair

Link to integration documentation on our website

https://www.home-assistant.io/integrations/screenlogic

Diagnostics information

config_entry-screenlogic-18dd665d1ac51f5d447aa6b574f01bb5.json.txt

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

home-assistant[bot] commented 1 year ago

Hey there @dieselrabbit, @bdraco, mind taking a look at this issue as it has been labeled with an integration (screenlogic) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `screenlogic` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign screenlogic` Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

screenlogic documentation screenlogic source _{^{(message by IssueLinks)}}

dieselrabbit commented 1 year ago

Hi @bwoodworth, can you please enable debug logging and attach the log here after at least one 'unavailable' event?

bwoodworth commented 1 year ago

Here you go config_entry-screenlogic-18dd665d1ac51f5d447aa6b574f01bb5.json (2).txt

dieselrabbit commented 1 year ago

Hi @bwoodworth, Thank you for the additional diagnostic data. I am looking for the logs from Settings -> System -> Logs. You can use the Download Full Log button at the bottom of the page.

bwoodworth commented 1 year ago

Ah, thanks for the clarification. My log is many GBs so I can't attach it, but this is probably the relevant bits:

dieselrabbit commented 1 year ago

That indicates that the ScreenLogic integration did not get a response from the Pentair ScreenLogic protocol adapter at the first point in the login process that it needs to hear a response, and that the integration had immediately tried again, but that also failed.

Sometimes, especially after network changes (but not only after network changes), the protocol adapted can get into a bad state and accept a connection, but not respond to requests. Have you tried unplugging and replugging the protocol adapter?

bwoodworth commented 1 year ago

Yes, I have unplugged and replugged the adapter, but that didn't seem to fix the issue. It sounds like this may be more of a Screenlogic issue and not necessarily a HA integration issue

dieselrabbit commented 1 year ago

Do you happen to know what firmware build your protocol adapter is running?

bwoodworth commented 1 year ago

dieselrabbit commented 1 year ago

What virtualization software and host OS are you running your Home Assistant VM in?

bwoodworth commented 1 year ago

I am running Proxmox

ranrushing commented 10 months ago

Was there any resolution here? I feel I've been having the same problem. I started using the integration either late 2022 or early 2023 and the connection was never reliable so I incorporated a smart power plug that would cycle power to the ScreenLogic Protocol Adapter when status of entities were offline or unknown. This was fine for many months but somewhere around Sept it started going offline much more frequently, many times as frequent as every 15min. It would be really good to know if others have had this experience.

dieselrabbit commented 10 months ago

Please try Home Assistant 2023.10.2 (or later). There was some connection handling cleanup in screenlogicpy v0.9.1 that should help with reconnecting after loss of connection.

As it stands right now, I can't reproduce the issue but there are a few factors that we know are different from what has been reported:

Reporting user's protocol adapter is running a later firmware (738.0) version than mine (736.0).
- I intend to flash mine to the latest but I need a block of time to do that as I'll have to reconfigure my network temporarily.
Reporting user's HA is running in a VM, where mine is running Hass OS on x86.
- If the adapter upgrade doesn't yield any useful information, then I would look to try setting up a VM. I don't run any VMs nor have anything ready to spin one up on currently though.

Both updated firmware and running in a VM are common to this and the other connection-related issue currently open (#96803)

@ranrushing if your configuration is different than above, could you attach a diagnostics capture?

ranrushing commented 10 months ago

I had this problem with HA 2023.10.2 and even now on .3, as for the Protocol Adapter, I've had it on version 738 from the beginning. I'm in the same camp as you, I'm not in a position to run a VM either. I suppose I could be, just have to make it happen. Such a bummer as this was a primary reason I moved from Homeseer to HA. Something I've noticed from the start is while the HA integration is active, i cannot use the Screen Logic iOS App and reliably connect using a Local Connection. I can however use Remote connection though. I had chalked this up to Pentair only wanting to have 1 session at a time. I'm also finding that right now if my HA Integration connection breaks and if I don't reset the Protocol adapter, I will not be able to make a Local connection using the ScreenLogic App, however I will be able to using the Remote connection. I honestly thing the weakness/fault is on the Pentair product, but if you know how to stabilize this, that would be awesome.

Please see attached diagnostic log. config_entry-screenlogic-338f70a365ba25352b68c041e8e0e476.json.txt

dieselrabbit commented 10 months ago

Thank you for the diagnostics. It has actually exposed a bug in the data storage aspect of the underlying API, but that shouldn't affect the connection. Two questions:

Is your HA instance on the same subnet/vlan as your protocol adapter?
Do you have a password set on your protocol adapter?

Additionally, if you can enable debug logging on the integration and post logs from when you have connection issues, that would be helpful.

ranrushing commented 10 months ago

Flat network, so ya same subnet and no password. I wonder if I should set a password

ranrushing commented 10 months ago

Attached is a debug log. The entities become unavailable on 10/19/2023 at approx 1303 after which my automation which triggers a smart plug to power cycle the Protocol Adapter. From skimming the logs for "success: False" after the "fetching screenlogic data event" will show right before each reboot. I've got my polling set to 120sec and my power cycle automation is set to check every minute and if unavailable for 20sec then power cycle.

Do you know if there is a way to set the integration to use the "remote" screenlogic connection? When the integration is down, my local connection will be down too, but my remote connection will work.

home-assistant_screenlogic_2023-10-19T18-22-19.758Z.log

dieselrabbit commented 10 months ago

Apologies. I wrote out a whole reply several days ago and could've swore I hit Comment, but It's not here so, take 2:

Thanks for the debug log! It's interesting that your protocol adapter appears to be closing the connection to the integration right around every 30min. There's nothing on the API end that happens at 30min intervals. Would be curious if you changed your polling interval if the interval between connection losses would change too. This would indicate that after a certain number of requests the protocol adapter stops functioning, whereas if the connection loss interval stayed roughly the same, it may indicate a problem specific to the protocol adapter itself, or something else entirely.

As for the remote connection, I don't have any current plans to support it. To do so would require users to NOT set a password for their ScreenLogic. If a password is configured on the protocol adapter, a remote connection will need to use it but a local connection will still work without one. ScreenLogic passwords are sent encrypted (this is good) using a custom encryption algorithm (that's bad) and I don't have any desire to try and replicate it.

ranrushing commented 10 months ago

I have changed the polling time from 120sec to 240. I forget when I made this change but approx 24hrs ago. I'm not sure it's changed the timing of the offline events, hopefully you'll be a better judge of that. home-assistant_screenlogic_2023-10-27T12-04-58.498Z.log

ranrushing commented 10 months ago

@dieselrabbit Hi Diesel, have you had a chance to review my previous log? I've got a newer log as the Screenlogic app is connected less than not now. It started doing this with the polling time set to 240 and it seemed to get worse after a few days where it is disconnecting much more frequently. home-assistant_screenlogic_2023-11-07T14-57-12.681Z.log

dieselrabbit commented 10 months ago

Hi @ranrushing. It looks like the protocol adapter is disconnecting after not receiving a request from the integration for 1min. Can you try setting the polling to 60 and see if that helps?

ranrushing commented 10 months ago

I have changed polling to 60 and issue appears the same.

As for your previous reply talking about using Home Assistant 2023.10.2 or later. Here's the versions I have going and no updates available: Core 2023.11.1 Supervisor 2023.10.1 Operating System 11.1 Frontend 20231030.1

dieselrabbit commented 10 months ago

Thanks. I can see in your last log the updated debug logging in the latest version of the integration.

Would you happen to have a debug log with the polling at 60sec?

ranrushing commented 10 months ago

well it looks like I'm going to need a proper smack in the head. I'm about 30min in with no disconnects, but feeling extremely confident that the source of my problem was an IP conflict with one of my wireless access points. I'm questioning many things about myself at this point and how I let this happen much less how I did not detect it sooner.

Thanks so much for your time and I'm sorry for wasting it.

dieselrabbit commented 10 months ago

All good. This experience was still valuable. It brought on a lot more debug output to hopefully be able to diagnose future issues like this better, and in the process of testing I was able to learn even more about how the protocol adapter expects things to work.

I'm glad (hopeful?) that it seems to be working for you now.

issue-triage-workflows[bot] commented 7 months ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

home-assistant / core