home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
73.59k stars 30.75k forks source link

SharkIQ becomes "unavailabe" every few minutes #94206

Closed nwish closed 8 months ago

nwish commented 1 year ago

The problem

SharkIQ device becomes "unavailable" every few minutes. I know similar issues were previously addressed, but this seems different because it's intermittent every few minutes, rather than once a day as reported before. I just updated to core-2023.6.0 because I thought some of the connectivity issues were addressed, but nothing has changed. I have reloaded the integration several times, and also completely removed and re-added, and no change. Thanks.

shark1 shark2

What version of Home Assistant Core has the issue?

core-2023.5.0 and also core-2023.6.0

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Container

Integration causing the issue

SharkIQ

Link to integration documentation on our website

https://www.home-assistant.io/integrations/sharkiq

Diagnostics information

Logger: homeassistant.components.sharkiq Source: components/sharkiq/update_coordinator.py:71 Integration: Shark IQ (documentation, issues) First occurred: 7:17:38 PM (4 occurrences) Last logged: 7:26:07 PM

Unexpected error updating SharkIQ. Attempting re-auth Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 1152, in _create_direct_connection hosts = await asyncio.shield(host_resolved) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 874, in _resolve_host addrs = await self._resolver.resolve(host, port, family=self._family) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/aiohttp/resolver.py", line 33, in resolve infos = await self._loop.getaddrinfo( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/asyncio/base_events.py", line 867, in getaddrinfo return await self.run_in_executor( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/socket.py", line 962, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ socket.gaierror: [Errno -3] Try again

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/sharkiq/update_coordinator.py", line 71, in _async_update_data all_vacuums = await self.ayla_api.async_list_devices() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/sharkiq/ayla_api.py", line 207, in async_list_devices async with await self.async_request("get", f"{EU_DEVICE_URL if self.europe else DEVICE_URL:s}/apiv1/devices.json") as resp: File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 1141, in aenter self._resp = await self._coro ^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 536, in _request conn = await self._connector.connect( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 540, in connect proto = await self._create_connection(req, traces, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 901, in _createconnection , proto = await self._create_direct_connection(req, traces, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 1166, in _create_direct_connection raise ClientConnectorError(req.connection_key, exc) from exc aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host ads-field-39a9391a.aylanetworks.com:443 ssl:default [Try again]

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

home-assistant[bot] commented 1 year ago

Hey there @jeffresc, @funkybunch, mind taking a look at this issue as it has been labeled with an integration (sharkiq) you are listed as a code owner for? Thanks!

Code owner commands Code owners of `sharkiq` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign sharkiq` Removes the current integration label and assignees on the issue, add the integration domain after the command.

(message by CodeOwnersMention)


sharkiq documentation sharkiq source (message by IssueLinks)

funkybunch commented 1 year ago

Hi there @nwish, this is not something I'm able to replicate on my end. We also did not make any changes to the integration for the either the 2023.5 or 2023.6 release. 2023.4.5 was our last update. Which release did this first start happening for you?

It appears your HA instance is only intermittently able to connect to the API endpoint, so I'm inclined to think there might be some networking things at play. Do you happen to have PiHole or AdGuard or anything that may be blocking the DNS requests on occasion? Sometimes DNS-based ad blockers interfere with the SharkIQ endpoint because it has ads in the domain. You may need to whitelist it if you do have a setup like that.

nwish commented 1 year ago

@funkybunch thanks for the quick reply. I misread parts of the discussion here and thought that the reported authentication issue fix was yet upcoming - seems it was implemented already.

Anyway, to answer your question - I actually did think of that. I am not utilizing anything that I can think of that would intermittently block any outgoing requests, and I searched the router logs for any indication, to no avail. Since you can't replicate the issue, let me do a bit digging. Sorry to waste your time! I'll report back if I do find something.

funkybunch commented 1 year ago

@nwish No worries! Yeah that 2023.4.5 update was a long time coming but we finally merged it in. No reload automations are required anymore.

Are there other cloud-based integrations that you have installed? Are they experiencing similar behavior? Good luck and let me know if you find anything.

funkybunch commented 1 year ago

@nwish Just thought of this - I have a theory... Does your HA instance have an IPv6 address? The sporadic connect/disconnect every few minutes could be jumping between connection attempts made via IPv4 and IPv6. I believe the SharkIQ api only supports IPv4 because the devices themselves only support IPv4.

You may need to figure out a way to force outbound traffic to the API over IPv4 if your installation does support IPv6.

nwish commented 1 year ago

@funkybunch it seems like you're on to something here with the IPv6 connection theory. I disabled all IPv6 protocols on the host and the connection has been solid for the past 14 hours.

I do have another thing that I noticed though: I physically took my robot vac off the dock, carried to the downstairs living room and pressed the "clean" button, and the integration is reporting a status of "Returning to dock" rather than "Cleaning" despite the facts that A) the Shark app shows "Clean in progress" and B) the vacuum is actually cleaning, not looking for the dock. Any insight on that?

Kagey-cmd commented 1 year ago

Just thought I'd drop in here and mention that I believe I'm having the same issue. image

Kagey-cmd commented 1 year ago

Can confirm my HA instance has an IPv4 address.

funkybunch commented 1 year ago

I do have another thing that I noticed though: I physically took my robot vac off the dock, carried to the downstairs living room and pressed the "clean" button, and the integration is reporting a status of "Returning to dock" rather than "Cleaning" despite the facts that A) the Shark app shows "Clean in progress" and B) the vacuum is actually cleaning, not looking for the dock. Any insight on that?

@nwish Not sure what might be causing this off the top of my head. Can you create another issue with any relevant Shark logs from this occurrence?

funkybunch commented 1 year ago

Can confirm my HA instance has an IPv4 address.

@Kagey-cmd do you have a public IPv6 address? Even if you're instance has a private IPv4 address but the router is using IPv6 to transmit over the internet, this will still occur.

Alternatively if your host has an IPv4 and IPv6 address, it is likely defaulting to IPv6. It's not the lack of IPv4 that is the problem, it is the presence of IPv6.

If either one of those is the case, you will need to make some network configuration changes to force that traffic through IPv4. It isn't an HA issue.

nwish commented 1 year ago

@funkybunch Yep, I will do that, sorry! I know how cluttered these things can get. I sort of forgot about it.

@Kagey-cmd I now have my disconnects down to about every few hours, for around 10-15 second each time, and that happened after I disabled all IPv6 connections on all host machine interfaces. I am running Ubuntu 22.04.2 with HA running in a Docker container. I did not make any changes to Docker's network configuration, nor any changes at my router. Hope this helps!

JOHLC commented 1 year ago

Just to chime in here, I am running latest version of core and also believe I have the issue. image

My network has no IPV6 for WAN or LAN.

Here are some log entries:

Logger: homeassistant.components.sharkiq
Source: components/sharkiq/update_coordinator.py:71
Integration: Shark IQ ([documentation](https://www.home-assistant.io/integrations/sharkiq), [issues](https://github.com/home-assistant/core/issues?q=is%3Aissue+is%3Aopen+label%3A%22integration%3A+sharkiq%22))
First occurred: July 24, 2023 at 11:24:14 AM (7 occurrences)
Last logged: July 24, 2023 at 11:42:36 AM

Unexpected error updating SharkIQ. Attempting re-auth
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 1155, in _create_direct_connection
    hosts = await asyncio.shield(host_resolved)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 874, in _resolve_host
    addrs = await self._resolver.resolve(host, port, family=self._family)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/resolver.py", line 33, in resolve
    infos = await self._loop.getaddrinfo(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 867, in getaddrinfo
    return await self.run_in_executor(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/socket.py", line 962, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno -3] Try again

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/sharkiq/update_coordinator.py", line 71, in _async_update_data
    all_vacuums = await self.ayla_api.async_list_devices()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sharkiq/ayla_api.py", line 207, in async_list_devices
    async with await self.async_request("get", f"{EU_DEVICE_URL if self.europe else DEVICE_URL:s}/apiv1/devices.json") as resp:
  File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 1141, in __aenter__
    self._resp = await self._coro
                 ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 536, in _request
    conn = await self._connector.connect(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 540, in connect
    proto = await self._create_connection(req, traces, timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 901, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 1169, in _create_direct_connection
    raise ClientConnectorError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host ads-field-39a9391a.aylanetworks.com:443 ssl:default [Try again]
Logger: homeassistant.components.sharkiq
Source: helpers/update_coordinator.py:262
Integration: Shark IQ ([documentation](https://www.home-assistant.io/integrations/sharkiq), [issues](https://github.com/home-assistant/core/issues?q=is%3Aissue+is%3Aopen+label%3A%22integration%3A+sharkiq%22))
First occurred: July 24, 2023 at 1:23:18 PM (1 occurrences)
Last logged: July 24, 2023 at 1:23:18 PM

Error fetching sharkiq data: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://ads-field-39a9391a.aylanetworks.com/apiv1/dsns/AC000W016110415/properties.json')
Logger: homeassistant.components.sharkiq
Source: helpers/aiohttp_client.py:73
Integration: Shark IQ ([documentation](https://www.home-assistant.io/integrations/sharkiq), [issues](https://github.com/home-assistant/core/issues?q=is%3Aissue+is%3Aopen+label%3A%22integration%3A+sharkiq%22))
First occurred: July 23, 2023 at 8:10:34 PM (22 occurrences)
Last logged: 1:03:10 PM

Unexpected error updating SharkIQ. Attempting re-auth
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/sharkiq/update_coordinator.py", line 80, in _async_update_data
    await asyncio.gather(*(self._async_update_vacuum(v) for v in online_vacs))
  File "/usr/src/homeassistant/homeassistant/components/sharkiq/update_coordinator.py", line 59, in _async_update_vacuum
    await sharkiq.async_update()
  File "/usr/local/lib/python3.11/site-packages/sharkiq/sharkiq.py", line 231, in async_update
    properties = await resp.json()
                 ^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/aiohttp_client.py", line 73, in json
    return await super().json(*args, loads=loads, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1104, in json
    raise ContentTypeError(
aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://ads-field-39a9391a.aylanetworks.com/apiv1/dsns/AC000W016110415/properties.json')
Logger: homeassistant.components.sharkiq
Source: helpers/update_coordinator.py:229
Integration: Shark IQ ([documentation](https://www.home-assistant.io/integrations/sharkiq), [issues](https://github.com/home-assistant/core/issues?q=is%3Aissue+is%3Aopen+label%3A%22integration%3A+sharkiq%22))
First occurred: July 23, 2023 at 8:10:34 PM (27 occurrences)
Last logged: 1:03:10 PM

Error fetching sharkiq data: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://ads-field-39a9391a.aylanetworks.com/apiv1/dsns/AC000W016110415/properties.json')
Error fetching sharkiq data: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://ads-field-39a9391a.aylanetworks.com/apiv1/devices.json')
Error fetching sharkiq data: Cannot connect to host ads-field-39a9391a.aylanetworks.com:443 ssl:default [Try again]

Another note: This has been going on for some time. I actually created an automation to reload the configuration if it is detected as unavailable.

funkybunch commented 1 year ago

A couple thoughts here. The need to reload the integration via automation should not be necessary anymore, as we've been able to bring in new maintainers and make a code change to handle token expiration errors. Your automation may be causing some of these errors if it is reloading upon encountering an error and then immediately encountering an error. So I'd recommend disabling any automations that reload the integration automatically.

Second thing, is one of these errors is a DNS resolution error. So if you're using an ad blocker like pihole or adguard then make sure the domains are whitelisted. If you aren't, or the domains are already whitelisted, make sure your upstream DNS provider is stable.

The Attempt to decode JSON with unexpected mimetype: text/html error typically means authentication wasn't successful. So if you're getting these repeatedly, then the migration when we merged in these token fixes may not have run properly. Try completely removing the integration and setting it back up from scratch and let me know if these continue. the integration should be automatically handling re-authentication, so that makes me think there may be some residual token or cookie in your installation. Re-installing the integration should fix it.

funkybunch commented 1 year ago

There may be a timing bug with re-authentication that causes one of these errors at maximum once an hour - Most commonly every 24 hours. Currently that is the benchmark we are using as normal. We'll continue looking into it, but will need to gather more data.

MarcMTLDK commented 1 year ago

I’m having the same issue and I just performed some testing, as below:

I deactivated the iPv6 on my router and on my hassio and the problem went out but not having ipv6 enable for other integration was creating issues so I had to reactivated,

at least, I can confirm somethings seams to indicate problems with the ip6 transport layer for the Shark integration.

Ziris85 commented 1 year ago

Unsure if my post will be helpful, but just noting that I was getting the same error as OP (not the same behavior though - the connection was never successful in my case). My issue did eventually end up being due to DNS. I just switched ISP providers, which required me to rejigger my home network and DNS setup, and I'd forgotten to update my HomeAssistant docker container that was now sending DNS queries to a non-existent server. Quick fix, corrected the container config to point to my new DNS server and it sprung back to life. (Perhaps ironically, I pointed my container TO my adguard server, but the URL in question for me doesn't have the ads keyword, at least for the moment).

This bug report and discussion proved helpful in tracking the issue down, so thank you!

7ooL commented 1 year ago

I see the same issue and have for a long time. IMG_0245

DiGiSoUL84 commented 12 months ago

A couple thoughts here. The need to reload the integration via automation should not be necessary anymore, as we've been able to bring in new maintainers and make a code change to handle token expiration errors. Your automation may be causing some of these errors if it is reloading upon encountering an error and then immediately encountering an error. So I'd recommend disabling any automations that reload the integration automatically.

Second thing, is one of these errors is a DNS resolution error. So if you're using an ad blocker like pihole or adguard then make sure the domains are whitelisted. If you aren't, or the domains are already whitelisted, make sure your upstream DNS provider is stable.

The Attempt to decode JSON with unexpected mimetype: text/html error typically means authentication wasn't successful. So if you're getting these repeatedly, then the migration when we merged in these token fixes may not have run properly. Try completely removing the integration and setting it back up from scratch and let me know if these continue. the integration should be automatically handling re-authentication, so that makes me think there may be some residual token or cookie in your installation. Re-installing the integration should fix it.

Just wanted to put this out that I have re-installed the integration multiple times and still get this error. I don't have to reload integration because re-auth usually works but I do get these error messages in my logs

JOHLC commented 12 months ago

I am also still seeing this on occasion, even after whitelisting all domains and IPs in my DNS filter.

zadigre commented 12 months ago

I'm also seeing this now... I was wondering why my robots were not able to start randomly... checked my logs... and I see this "not available" message for both of them a few times per day.

DiGiSoUL84 commented 12 months ago

I am also still seeing this on occasion, even after whitelisting all domains and IPs in my DNS filter.

Same at least twice a day

AkaMrBill commented 12 months ago

Ditto to the above. Mine drops to "Unavailable" randomly from every few minutes to a couple times a day. Even my automation to Reload it can't keep up. Honestly, I would drop the integration if I didn't need it to turn on the lights in my basement so the vacuum can see where it's going.

JOHLC commented 12 months ago

Ditto to the above. Mine drops to "Unavailable" randomly from every few minutes to a couple times a day. Even my automation to Reload it can't keep up. Honestly, I would drop the integration if I didn't need it to turn on the lights in my basement so the vacuum can see where it's going.

I taped over my camera with black tape and it didnt seem to affect anything.

AkaMrBill commented 12 months ago

That is quite literally putting a bandaid on the problem and not fixing it. The integration is still severely broken.

funkybunch commented 12 months ago

Folks, we cannot fix anything without logs and ensuring your local network is configured in a way that will make the integration function.

There is a known issue where the API itself does not support IPv6. This is out of our control. If these unavailable status changes are a deal breaker for you, please confirm your HA instance is only using IPv4, even locally. If you're not sure, I recommend testing it by placing your instance on an IPv4 only network with absolutely no IPv6 support for outbound traffic.

The other two things that can cause this are DNS issues and rate limiting. It has not been necessary to have a reload automation for months. If you are still using one, you run the risk of rate limiting yourself, which will again cause this same issue.

If all of that is configured correctly, please post your logs. As I mentioned earlier, right now the only known reason this happens is due to misconfigured DNS, rate limiting, or use of IPv6 - none of which are an issue with the integration. Unless you are experiencing a brand new issue, it is not something that will be fixed because it is an upstream problem or a configuration problem.

If there IS an issue we don't know about, which is possible, chiming in just to say "this is happening to me too" is not helpful. Check the 3 things above, and if it's still happening please post your logs.

We are unable to replicate the issue with the information we currently have, so if you truly have checked everything we need the logs to figure out what is going on.

7ooL commented 12 months ago

Can you tell me if there are specific ways to get logs rather than the error messages that show up?

funkybunch commented 12 months ago

Can you tell me if there are specific ways to get logs rather than the error messages that show up?

First paragraph here: https://www.home-assistant.io/integrations/system_log/

funkybunch commented 12 months ago

I am also still seeing this on occasion, even after whitelisting all domains and IPs in my DNS filter.

Is it as often as you first reported? What kind of frequency are you seeing?

Did the whitelisting at least reduce the error?

zadigre commented 12 months ago

fyi. I stopped restarting the integration for a long time. I'm not rate limited. I was using 1.1.1.1 dns... just switched to Google's DNS. I have this issue... and I do not have any IPv6 (my ISP refuses to provide IPv6... and IPv6 is not enabled on my HA instance).

I've enabled debug on the integration. attached my home assistant log

home-assistant_2023-11-17T00-55-20.213Z.log

AkaMrBill commented 12 months ago

Folks, we cannot fix anything without logs and ensuring your local network is configured in a way that will make the integration function.

There is a known issue where the API itself does not support IPv6. This is out of our control. If these unavailable status changes are a deal breaker for you, please confirm your HA instance is only using IPv4, even locally. If you're not sure, I recommend testing it by placing your instance on an IPv4 only network with absolutely no IPv6 support for outbound traffic.

The other two things that can cause this are DNS issues and rate limiting. It has not been necessary to have a reload automation for months. If you are still using one, you run the risk of rate limiting yourself, which will again cause this same issue.

If all of that is configured correctly, please post your logs. As I mentioned earlier, right now the only known reason this happens is due to misconfigured DNS, rate limiting, or use of IPv6 - none of which are an issue with the integration. Unless you are experiencing a brand new issue, it is not something that will be fixed because it is an upstream problem or a configuration problem.

If there IS an issue we don't know about, which is possible, chiming in just to say "this is happening to me too" is not helpful. Check the 3 things above, and if it's still happening please post your logs.

We are unable to replicate the issue with the information we currently have, so if you truly have checked everything we need the logs to figure out what is going on.

Thanks for the pointers. So, after disabling IPv6 from both my router, thus the network, and locally from my HA server, then completely removing all Automations to reload, I see improvement, but not resolution. I still get random "Unavailable", but it re-loads by it's self. Here is the only part of the LOG that covers the time period after I made the changes. Error fetching sharkiq data: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://ads-field-39a9391a.aylanetworks.com/apiv1/dsns/AC000W021786108/properties.json') 5:13:20 PM – (ERROR) Shark IQ - message first occurred at 2:28:00 PM and shows up 6 times Unexpected error updating SharkIQ. Attempting re-auth 5:13:20 PM – (ERROR) Shark IQ Local fulfillment failed to setup, falling back to cloud fulfillment 5:12:04 PM – (WARNING) Google Assistant Webhook handler 120548bccdd1385e2d8489e1c13a9a68e63a38df69ef9714fa18a2ba3771bc33 for agent user id c5222adc-bed0-4716-b824-041ebced61ce is already defined! 5:12:04 PM – (WARNING) Google Assistant Unexpected error updating SharkIQ. Attempting re-auth 4:49:03 PM – (ERROR) Shark IQ - message first occurred at 2:28:00 PM and shows up 6 times Unexpected error updating SharkIQ. Attempting re-auth 4:47:59 PM – (ERROR) Shark IQ - message first occurred at 4:02:11 PM and shows up 3 times

funkybunch commented 12 months ago

Thanks @zadigre that is super helpful - can you and @AkaMrBill tell me what time zone and region you're in? And @AkaMrBill could you also enable debug logging on the integration and send me the log output after it sits for a couple hours?

On first glance the failures are rarely consecutive failures. Generally it reconnects properly on the next attempt 30 seconds later. I'm wondering if this is an API stability issue. I'd be able to confirm with some more debug log files and seeing if the timestamps match up in like-regions.

If that is the case, given so far we've been seeing the integration reconnect properly once the API is stable again, we could put in a patch to simply debounce the failure time. Basically, not actually change the status to "unavailable" until the last_updated attribute is more than 90 seconds ago or something - just ignore the first failure or two unless it becomes constant.

AkaMrBill commented 12 months ago

Debug logging set at 6:52 PM local time. I live in the US in Mountain Time. I will post the logs again in a few hours.

zadigre commented 12 months ago

Thanks @zadigre that is super helpful - can you and @AkaMrBill tell me what time zone and region you're in?

I'm in EST -5 (Eastern Standard Time, I'm in Montreal, Quebec, Canada)

AkaMrBill commented 12 months ago

Here are the debug enabled logs for the last couple hours. I only saw 2 instances of Unavailable. Again, it did remedy its self. _``` Error fetching sharkiq data: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://ads-field-39a9391a.aylanetworks.com/apiv1/dsns/AC000W021786108/properties.json') 9:35:14 PM – (ERROR) Shark IQ - message first occurred at 2:28:00 PM and shows up 13 times Unexpected error updating SharkIQ. Attempting re-auth 9:35:14 PM – (ERROR) Shark IQ Error fetching WiZ RGBW Garage West data: Failed to update device at 192.168.1.108: 8:27:52 PM – (ERROR) WiZ Time out fetching Ring device data 8:13:39 PM – (WARNING) Ring Error from stream worker: Error demuxing stream: [Errno 110] Operation timed out: 'rtsp://AkaMrBill:bk07229527@192.168.1.151/live' 8:12:45 PM – (ERROR) Stream - message first occurred at 2:12:44 PM and shows up 7 times Error from stream worker: Error demuxing stream: [Errno 110] Operation timed out: 'rtsp://AkaMrBill:bk07229527@192.168.1.141/live' 8:09:54 PM – (ERROR) Stream - message first occurred at 2:11:45 PM and shows up 5 times Error getting new camera image from Backyard West Wyze: Client error '404 Not Found' for url 'http://192.168.1.105:8123/local/Backyard%20West%20Wyze.jpg' For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404 8:09:01 PM – (ERROR) Generic Camera - message first occurred at 2:10:59 PM and shows up 52 times Unexpected error updating SharkIQ. Attempting re-auth 7:57:08 PM – (ERROR) Shark IQ - message first occurred at 2:28:00 PM and shows up 12 times Unexpected error updating SharkIQ. Attempting re-auth 7:56:38 PM – (ERROR) Shark IQ - message first occurred at 4:02:11 PM and shows up 9 times Unexpected error updating SharkIQ. Attempting re-auth 7:16:22 PM – (ERROR) Shark IQ - message first occurred at 5:13:20 PM and shows up 5 times

funkybunch commented 12 months ago

Could you open up your full log and share the timestamps for the 2 failures? The debug log entry should resemble this: DEBUG (MainThread) [homeassistant.components.sharkiq] Finished fetching sharkiq data in [...] seconds (success: False)

You won't be able to simply Ctrl+F that message because[...] will have the length of time the request took. The failures will have success: False though.

AkaMrBill commented 11 months ago

Could you open up your full log and share the timestamps for the 2 failures? The debug log entry should resemble this: DEBUG (MainThread) [homeassistant.components.sharkiq] Finished fetching sharkiq data in [...] seconds (success: False)

You won't be able to simply Ctrl+F that message because[...] will have the length of time the request took. The failures will have success: False though.

Here is one with a lot more instances of failures. From 7:00 am on there are quite a few. home-assistant_2023-11-17T16-22-24.400Z.log

7ooL commented 11 months ago

Eastern Time Zone, I do have PiHole on the network and *.aylanetworks.com is whitelisted, no indication of any URLs related being blocked. IPV6 is disabled at the firewall/router.

`Logger: homeassistant.components.sharkiq Source: components/sharkiq/update_coordinator.py:70 Integration: Shark IQ (documentation, issues) First occurred: November 11, 2023 at 9:48:34 PM (156 occurrences) Last logged: 1:05:42 PM

Unexpected error updating SharkIQ. Attempting re-auth Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/sharkiq/update_coordinator.py", line 70, in _async_update_data all_vacuums = await self.ayla_api.async_list_devices() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/sharkiq/ayla_api.py", line 208, in async_list_devices devices = await resp.json() ^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/helpers/aiohttp_client.py", line 84, in json return await super().json(*args, loads=loads, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1104, in json raise ContentTypeError( aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://ads-field-39a9391a.aylanetworks.com/apiv1/devices.json') `

`Logger: homeassistant.components.sharkiq Source: helpers/update_coordinator.py:322 Integration: Shark IQ (documentation, issues) First occurred: November 11, 2023 at 9:03:08 PM (189 occurrences) Last logged: 1:05:12 PM

Error fetching sharkiq data: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://ads-field-39a9391a.aylanetworks.com/apiv1/dsns/AC000W008134041/properties.json') Error fetching sharkiq data: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://ads-field-39a9391a.aylanetworks.com/apiv1/devices.json') Error fetching sharkiq data: 0, message='Attempt to decode JSON with unexpected mimetype: ', url=URL('https://ads-field-39a9391a.aylanetworks.com/apiv1/dsns/AC000W008134041/properties.json') Error fetching sharkiq data:`

`Logger: homeassistant.components.sharkiq Source: components/sharkiq/update_coordinator.py:79 Integration: Shark IQ (documentation, issues) First occurred: November 11, 2023 at 9:03:08 PM (125 occurrences) Last logged: 1:05:12 PM

Unexpected error updating SharkIQ. Attempting re-auth Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/sharkiq/update_coordinator.py", line 79, in _async_update_data await asyncio.gather((self._async_update_vacuum(v) for v in online_vacs)) File "/usr/src/homeassistant/homeassistant/components/sharkiq/update_coordinator.py", line 58, in _async_update_vacuum await sharkiq.async_update() File "/usr/local/lib/python3.11/site-packages/sharkiq/sharkiq.py", line 231, in async_update properties = await resp.json() ^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/helpers/aiohttp_client.py", line 84, in json return await super().json(args, loads=loads, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1104, in json raise ContentTypeError( aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://ads-field-39a9391a.aylanetworks.com/apiv1/dsns/AC000W008134041/properties.json') `

`Logger: homeassistant.components.sharkiq Source: components/sharkiq/update_coordinator.py:79 Integration: Shark IQ (documentation, issues) First occurred: November 12, 2023 at 1:06:55 AM (8 occurrences) Last logged: 1:05:52 AM

Unexpected error updating SharkIQ. Attempting re-auth Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/sharkiq/update_coordinator.py", line 58, in _async_update_vacuum await sharkiq.async_update() File "/usr/local/lib/python3.11/site-packages/sharkiq/sharkiq.py", line 230, in async_update async with await self.ayla_api.async_request('get', self.update_url, params=params) as resp: File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 1141, in aenter self._resp = await self._coro ^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 560, in _request await resp.start(conn) File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 899, in start message, payload = await protocol.read() # type: ignore[union-attr] ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/aiohttp/streams.py", line 616, in read await self._waiter asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/sharkiq/update_coordinator.py", line 79, in _async_update_data await asyncio.gather(*(self._async_update_vacuum(v) for v in online_vacs)) File "/usr/src/homeassistant/homeassistant/components/sharkiq/update_coordinator.py", line 57, in _async_update_vacuum async with asyncio.timeout(API_TIMEOUT): File "/usr/local/lib/python3.11/asyncio/timeouts.py", line 111, in aexit raise TimeoutError from exc_val TimeoutError `

JOHLC commented 11 months ago

I am eastern time (EST) -- IPV6 is disabled, I use PFBlockerNG for filtering, but have whitelisted the URLs. For external DNS, I am using Cloudflare. I still have the automation to reload, but that is now set to only trigger if the entity is unavailable for 30 minutes (it hasn't triggered any time recently.)

For some additional troubleshooting, I am adding the URL (ads-field-39a9391a.aylanetworks.com) to my Uptime Kuma instance and will report if that shows any issues with the endpoints. (HTTP and DNS)

Here are some more logs. I am working on getting debug logging set.

image
aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://ads-field-39a9391a.aylanetworks.com/apiv1/dsns/AC************MASKED*/properties/SET_Power_Mode/datapoints.json')
2023-11-19 12:19:24.418 ERROR (MainThread) [homeassistant.components.automation.shark_set_to_max_clean] While executing automation automation.shark_set_to_max_clean
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 655, in async_trigger
    await self.action_script.async_run(
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 1578, in async_run
    return await asyncio.shield(run.async_run())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 420, in async_run
    await self._async_step(log_exceptions=False)
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 470, in _async_step
    self._handle_exception(
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 493, in _handle_exception
    raise exception
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 468, in _async_step
    await getattr(self, handler)()
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 704, in _async_call_service_step
    response_data = await self._async_run_long_action(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 666, in _async_run_long_action
    return long_task.result()
           ^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2035, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2072, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 235, in handle_service
    return await service.entity_service_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 876, in entity_service_call
    response_data = await _handle_entity_call(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 948, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/sharkiq/vacuum.py", line 206, in async_set_fan_speed
    await self.sharkiq.async_set_property_value(
  File "/usr/local/lib/python3.11/site-packages/sharkiq/sharkiq.py", line 202, in async_set_property_value
    resp_data = await resp.json()
                ^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/aiohttp_client.py", line 84, in json
    return await super().json(*args, loads=loads, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1104, in json
    raise ContentTypeError(
aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://ads-field-39a9391a.aylanetworks.com/apiv1/dsns/AC************MASKED*/properties/SET_Power_Mode/datapoints.json')
2023-11-19 12:19:54.864 ERROR (MainThread) [homeassistant.components.sharkiq] Unexpected error updating SharkIQ.  Attempting re-auth
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/sharkiq/update_coordinator.py", line 79, in _async_update_data
    await asyncio.gather(*(self._async_update_vacuum(v) for v in online_vacs))
  File "/usr/src/homeassistant/homeassistant/components/sharkiq/update_coordinator.py", line 58, in _async_update_vacuum
    await sharkiq.async_update()
  File "/usr/local/lib/python3.11/site-packages/sharkiq/sharkiq.py", line 231, in async_update
    properties = await resp.json()
                 ^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/aiohttp_client.py", line 84, in json
    return await super().json(*args, loads=loads, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1104, in json
    raise ContentTypeError(
aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://ads-field-39a9391a.aylanetworks.com/apiv1/dsns/AC************MASKED*/properties.json')
2023-11-19 12:47:18.128 ERROR (MainThread) [homeassistant.components.sharkiq] Unexpected error updating SharkIQ.  Attempting re-auth
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/sharkiq/update_coordinator.py", line 79, in _async_update_data
    await asyncio.gather(*(self._async_update_vacuum(v) for v in online_vacs))
  File "/usr/src/homeassistant/homeassistant/components/sharkiq/update_coordinator.py", line 58, in _async_update_vacuum
    await sharkiq.async_update()
  File "/usr/local/lib/python3.11/site-packages/sharkiq/sharkiq.py", line 231, in async_update
    properties = await resp.json()
                 ^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/aiohttp_client.py", line 84, in json
    return await super().json(*args, loads=loads, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1104, in json
    raise ContentTypeError(
aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://ads-field-39a9391a.aylanetworks.com/apiv1/dsns/AC************MASKED*/properties.json')
2023-11-19 12:47:18.129 ERROR (MainThread) [homeassistant.components.sharkiq] Error fetching sharkiq data: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://ads-field-39a9391a.aylanetworks.com/apiv1/dsns/AC************MASKED*/properties.json')
2023-11-19 13:04:09.692 ERROR (MainThread) [homeassistant.components.sharkiq] Unexpected error updating SharkIQ.  Attempting re-auth
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/sharkiq/update_coordinator.py", line 79, in _async_update_data
    await asyncio.gather(*(self._async_update_vacuum(v) for v in online_vacs))
  File "/usr/src/homeassistant/homeassistant/components/sharkiq/update_coordinator.py", line 58, in _async_update_vacuum
    await sharkiq.async_update()
  File "/usr/local/lib/python3.11/site-packages/sharkiq/sharkiq.py", line 231, in async_update
    properties = await resp.json()
                 ^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/aiohttp_client.py", line 84, in json
    return await super().json(*args, loads=loads, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1104, in json
    raise ContentTypeError(
aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://ads-field-39a9391a.aylanetworks.com/apiv1/dsns/AC************MASKED*/properties.json')
2023-11-19 13:04:09.695 ERROR (MainThread) [homeassistant.components.sharkiq] Error fetching sharkiq data: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://ads-field-39a9391a.aylanetworks.com/apiv1/dsns/AC************MASKED*/properties.json')
2023-11-19 13:04:40.859 ERROR (MainThread) [homeassistant.components.sharkiq] Unexpected error updating SharkIQ.  Attempting re-auth
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/sharkiq/update_coordinator.py", line 70, in _async_update_data
    all_vacuums = await self.ayla_api.async_list_devices()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sharkiq/ayla_api.py", line 208, in async_list_devices
    devices = await resp.json()
              ^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/aiohttp_client.py", line 84, in json
    return await super().json(*args, loads=loads, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1104, in json
    raise ContentTypeError(
aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://ads-field-39a9391a.aylanetworks.com/apiv1/devices.json')
2023-11-19 13:04:40.861 ERROR (MainThread) [homeassistant.components.sharkiq] Error fetching sharkiq data: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://ads-field-39a9391a.aylanetworks.com/apiv1/devices.json')
2023-11-19 13:05:12.213 ERROR (MainThread) [homeassistant.components.sharkiq] Unexpected error updating SharkIQ.  Attempting re-auth
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/sharkiq/update_coordinator.py", line 70, in _async_update_data
    all_vacuums = await self.ayla_api.async_list_devices()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sharkiq/ayla_api.py", line 208, in async_list_devices
    devices = await resp.json()
              ^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/aiohttp_client.py", line 84, in json
    return await super().json(*args, loads=loads, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1104, in json
    raise ContentTypeError(
aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://ads-field-39a9391a.aylanetworks.com/apiv1/devices.json')
2023-11-19 13:05:12.214 ERROR (MainThread) [homeassistant.components.sharkiq] Error fetching sharkiq data: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://ads-field-39a9391a.aylanetworks.com/apiv1/devices.json')
2023-11-19 13:05:43.106 ERROR (MainThread) [homeassistant.components.sharkiq] Unexpected error updating SharkIQ.  Attempting re-auth
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/sharkiq/update_coordinator.py", line 79, in _async_update_data
    await asyncio.gather(*(self._async_update_vacuum(v) for v in online_vacs))
  File "/usr/src/homeassistant/homeassistant/components/sharkiq/update_coordinator.py", line 58, in _async_update_vacuum
    await sharkiq.async_update()
  File "/usr/local/lib/python3.11/site-packages/sharkiq/sharkiq.py", line 231, in async_update
    properties = await resp.json()
                 ^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/aiohttp_client.py", line 84, in json
    return await super().json(*args, loads=loads, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1104, in json
    raise ContentTypeError(
aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://ads-field-39a9391a.aylanetworks.com/apiv1/dsns/AC************MASKED*/properties.json')
2023-11-19 13:06:13.644 ERROR (MainThread) [homeassistant.components.sharkiq] Unexpected error updating SharkIQ.  Attempting re-auth
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/sharkiq/update_coordinator.py", line 70, in _async_update_data
    all_vacuums = await self.ayla_api.async_list_devices()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sharkiq/ayla_api.py", line 208, in async_list_devices
    devices = await resp.json()
              ^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/aiohttp_client.py", line 84, in json
    return await super().json(*args, loads=loads, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1104, in json
    raise ContentTypeError(
aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://ads-field-39a9391a.aylanetworks.com/apiv1/devices.json')
JOHLC commented 11 months ago

Here are some debug logs Around the time of 2023-11-19 15:03:39-15:05:11 EST The response times of this server seem to be acceptable (tested every 30 seconds)

image
2023-11-19 15:03:09.029 DEBUG (MainThread) [homeassistant.components.sharkiq] Finished fetching sharkiq data in 1.974 seconds (success: True)
2023-11-19 15:03:39.694 ERROR (MainThread) [homeassistant.components.sharkiq] Unexpected error updating SharkIQ.  Attempting re-auth
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/sharkiq/update_coordinator.py", line 70, in _async_update_data
    all_vacuums = await self.ayla_api.async_list_devices()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sharkiq/ayla_api.py", line 208, in async_list_devices
    devices = await resp.json()
              ^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/aiohttp_client.py", line 84, in json
    return await super().json(*args, loads=loads, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1104, in json
    raise ContentTypeError(
aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://ads-field-39a9391a.aylanetworks.com/apiv1/devices.json')
2023-11-19 15:03:39.699 ERROR (MainThread) [homeassistant.components.sharkiq] Error fetching sharkiq data: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://ads-field-39a9391a.aylanetworks.com/apiv1/devices.json')
2023-11-19 15:03:39.699 DEBUG (MainThread) [homeassistant.components.sharkiq] Finished fetching sharkiq data in 0.645 seconds (success: False)
2023-11-19 15:04:10.264 DEBUG (MainThread) [homeassistant.components.sharkiq] Updating sharkiq data
2023-11-19 15:04:10.264 DEBUG (MainThread) [homeassistant.components.sharkiq] Updating sharkiq data for device DSN AC00**MASKED**
2023-11-19 15:04:11.150 INFO (MainThread) [homeassistant.components.sharkiq] Fetching sharkiq data recovered
2023-11-19 15:04:11.150 DEBUG (MainThread) [homeassistant.components.sharkiq] Finished fetching sharkiq data in 2.096 seconds (success: True)
2023-11-19 15:04:41.351 ERROR (MainThread) [homeassistant.components.sharkiq] Unexpected error updating SharkIQ.  Attempting re-auth
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/sharkiq/update_coordinator.py", line 70, in _async_update_data
    all_vacuums = await self.ayla_api.async_list_devices()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sharkiq/ayla_api.py", line 208, in async_list_devices
    devices = await resp.json()
              ^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/aiohttp_client.py", line 84, in json
    return await super().json(*args, loads=loads, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1104, in json
    raise ContentTypeError(
aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://ads-field-39a9391a.aylanetworks.com/apiv1/devices.json')
2023-11-19 15:04:41.354 ERROR (MainThread) [homeassistant.components.sharkiq] Error fetching sharkiq data: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://ads-field-39a9391a.aylanetworks.com/apiv1/devices.json')
2023-11-19 15:04:41.355 DEBUG (MainThread) [homeassistant.components.sharkiq] Finished fetching sharkiq data in 0.299 seconds (success: False)
2023-11-19 15:05:11.378 DEBUG (MainThread) [homeassistant.components.sharkiq] Updating sharkiq data
2023-11-19 15:05:11.378 DEBUG (MainThread) [homeassistant.components.sharkiq] Updating sharkiq data for device DSN AC00**MASKED**
2023-11-19 15:05:11.436 ERROR (MainThread) [homeassistant.components.sharkiq] Unexpected error updating SharkIQ.  Attempting re-auth
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/sharkiq/update_coordinator.py", line 79, in _async_update_data
    await asyncio.gather(*(self._async_update_vacuum(v) for v in online_vacs))
  File "/usr/src/homeassistant/homeassistant/components/sharkiq/update_coordinator.py", line 58, in _async_update_vacuum
    await sharkiq.async_update()
  File "/usr/local/lib/python3.11/site-packages/sharkiq/sharkiq.py", line 231, in async_update
    properties = await resp.json()
                 ^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/aiohttp_client.py", line 84, in json
    return await super().json(*args, loads=loads, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1104, in json
    raise ContentTypeError(
aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://ads-field-39a9391a.aylanetworks.com/apiv1/dsns/AC00**MASKED**/properties.json')
2023-11-19 15:05:11.440 DEBUG (MainThread) [homeassistant.components.sharkiq] Finished fetching sharkiq data in 0.386 seconds (success: False)
2023-11-19 15:06:01.111 DEBUG (MainThread) [homeassistant.components.sharkiq] Updating sharkiq data
2023-11-19 15:06:01.112 DEBUG (MainThread) [homeassistant.components.sharkiq] Updating sharkiq data for device DSN AC00**MASKED**
2023-11-19 15:06:02.656 INFO (MainThread) [homeassistant.components.sharkiq] Fetching sharkiq data recovered
2023-11-19 15:06:02.656 DEBUG (MainThread) [homeassistant.components.sharkiq] Finished fetching sharkiq data in 21.602 seconds (success: True)
2023-11-19 15:06:32.385 DEBUG (MainThread) [homeassistant.components.sharkiq] Updating sharkiq data
2023-11-19 15:06:32.385 DEBUG (MainThread) [homeassistant.components.sharkiq] Updating sharkiq data for device DSN AC00**MASKED**
2023-11-19 15:06:33.694 DEBUG (MainThread) [homeassistant.components.sharkiq] Finished fetching sharkiq data in 1.639 seconds (success: True)
2023-11-19 15:07:03.750 DEBUG (MainThread) [homeassistant.components.sharkiq] Updating sharkiq data

***Interesting note I was also monitoring https://ads-apiv1-sharkue1.aylanetworks.com/ and it looks like this returned a 502 result around the time of the issue. Not sure this is related since I dont see that url in the logs. (Note the 400 response is OK because it is simply checking that web server and not submitting any payload or authenticaion) No other monitors show any connection issue around that time.

image

This seems to line up with the time this last went unavailable

image
JOHLC commented 11 months ago

Here is another graph possibly indicating the issue is likely on Shark's end. Times appear to line up to when the entity goes unavailable: image

funkybunch commented 11 months ago

I was also monitoring https://ads-apiv1-sharkue1.aylanetworks.com/ and it looks like this returned a 502 result around the time of the issue.

Good catch. That URL actually has the same CNAME as the domain of the endpoint used in the integration. Ultimately meaning that they point to the same destination (Appears to be AWS US-East), and then the same 3 load balancers upstream from that (with only IPv4 addresses, by looking at the DNS route).

I'd be curious if anyone in the EU region is experiencing this issue as well. Right now all signs are pointing to an API stability issue. This is something that would at least affect North America, potentially the whole world outside the EU depending on their load balancing practices.

We won't be able to resolve the availability problem on our end, but we can at least implement a fix to get rid of (or greatly reduce) the unavailable messages.

funkybunch commented 11 months ago

That being said, the integration is accurately reflecting the availability of your devices (assuming the 3 things I mentioned earlier are configured properly).

To poll this group here, would you prefer to:

  1. Have an accurate representation of device availability, reflecting actual API availability.

OR

  1. Have the availability cleaned up, but the status of a device legitimately becoming unavailable will lag by 60 seconds or so.

To be clear, Option 1 would mean we close this issue out without any changes.

React with a 🚀 for Option 1 or a 🎉 for Option 2.

zadigre commented 11 months ago

Il more #2 🎉Le 21 nov. 2023 à 15:37, Mark Adkins @.***> a écrit : That being said, the integration is accurately reflecting the availability of your devices (assuming the 3 things I mentioned earlier are configured properly). To poll this group here, would you prefer to:

Have an accurate representation device availability reflecting actual API availability.

OR

Have the availability cleaned up, but the status of a device legitimately becoming unavailable will lag by 60 seconds or so.

To be clear, #1 would mean we close this issue out without a fix. React with a 🚀 for #1 or a 🎉 for #2.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

AkaMrBill commented 11 months ago

@funkybunch @zadigre Option #2 is by far a better choice IMHO. I really don't care when it is unavailable unless it goes "unavailable" for an hour or so. Being delayed by one minute, or more, will have no impact on me what so ever. It might for others, but I can't imagine why. I don't care if the Shark server goes off line. In fact, I would prefer to never reach out to it, I just want local control. I only use the integration to turn on lights, that's it. So, if you want to change the "unavailable" state to one that verifies the state a couple times before changing to "unavailable" that makes perfect sense to me.

DiGiSoUL84 commented 11 months ago

That being said, the integration is accurately reflecting the availability of your devices (assuming the 3 things I mentioned earlier are configured properly).

To poll this group here, would you prefer to:

  1. Have an accurate representation of device availability, reflecting actual API availability.

OR

  1. Have the availability cleaned up, but the status of a device legitimately becoming unavailable will lag by 60 seconds or so.

To be clear, Option 1 would mean we close this issue out without any changes.

React with a 🚀 for Option 1 or a 🎉 for Option 2.

The unavailability messages just clutter up my logs which I have changed my settings not to show them. Whenever I need to use the integration it works as I need to minus not having the ability to clean rooms (I know you've been working on that!) so I defer to you.

Just wanted to add that I appreciate all the hard work that has gone into all this! If there is anything you need from me in the way of testing or whatnot I would love to be of assistance. I would love to learn more. I am cursed by the desire to always tinker.

issue-triage-workflows[bot] commented 8 months ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.