Closed RaveGun closed 5 years ago
RaveGun
commented
5 years ago Hi,
It looks like none of the Google Drive add-ons out there are working.
Actually if I try to go to the http://www.googleapis.com/ I am getting: NotFound
Interesting, it might have to do something with my google account. I have no idea what it could be.
Thank you.
sabeechen
commented
5 years ago The GSuite Status Page usually has information about drive-wide outtages, but I don't see anything from the time period you posted. googleapis.com isn't a website, its just the address used for the Google Drive api so getting an error from a web browser is expected, I think.
Did the error go away after a while? It could have just been a temporary connectivity problem. If its a problem with your account, you could try the "reauthenticate" button to re-link your drive account.
I'll keep this open, since I should really surface a better error message if this happens.
RaveGun
commented
5 years ago Hi, I managed to test this several times without success. I also tried and created a new application and used the second option but it didn't work either. I did this on my phone so the error was thrown as a toast notification but I could see that it was related to oauth2.googleapi.com. This was after I got and entered the authentication code.
I would like to mention that a long time ago I created another application that I am using right now to get the calendar information in the home assistant. This works but I am seeing from time to time in the log that it was not able to query the same Google's API URL.
Maybe it has to do something with my developer account or some strange security settings.
I might try to reuse the above mentioned application and to enable the drive feature.
Thanks for the feedback.
sabeechen
commented
5 years ago I haven't heard of any other users having authentication problems so I also suspect it has to do with your account. It might be worthwhile to make sure "Hass.io Google Drive Backup" shows up under "Third Party Apps with account access" under your google security settings. You could also try creating another empty @gmail.com account and see if you can hook that one up to the add-on, that would tell for sure if the problem is with your account.
gdreelin
commented
5 years ago I have been getting this exact same error.
sabeechen
commented
5 years ago I posted this in the issue gdreedlin created, but it might help you too: I randomly stumbled into this issue, which present similarly to what you're seeing. It suggest that giving your Home Assistance instance a static IP might fix the problem, or that your router might be using a stale DNS server.
gdreelin
commented
5 years ago I posted this in the issue gdreedlin created, but it might help you too: I randomly stumbled into this issue, which present similarly to what you're seeing. It suggest that giving your Home Assistance instance a static IP might fix the problem, or that your router might be using a stale DNS server.
I currently have a static IP on HA. I manually assign all my systems with IPs only guests get DHCP.
gdreelin
commented
5 years ago Strangely enough it seemed to work the issue out, not sure why I just checked it again an all is working good.
sabeechen
commented
5 years ago My only guess is that something is wrong with your DNS configuration, but DNS problems are never easy to sort out. I'm glad it resolved out for you.
sabeechen
commented
5 years ago I intend to keep this open in case other people run into the same trouble and look for help, maybe down the line someone will be able to sort out what the root cause is.
jmart518
commented
5 years ago I too am having this issue. Tried uninstalling, reinstalling and no dice
httplib2.ServerNotFoundError: Unable to find the server at www.googleapis.com
gdreelin
commented
5 years ago After a few of your updates it just stopped. All is good now.
sabeechen
commented
5 years ago I just released v0.94, it gives a link in the big red error dialog to show the IP addresses that Google's API servers resolve to from inside the add-on's docker container. It also shows the result of trying to ping each IP address. If this is some kind of DNS resolution issue on your network, that should help to sort it out.
I'm still unable to reproduce this, even once, on any of the machines I'm using for testing :(
nikiyao
commented
5 years ago I'm having the same issue. It was working now. Today it stopped, I've tried restarting, unistalling, reauthorizing. I can't figure it out.
sabeechen
commented
5 years ago This is starting to keep me up at night :)
The more I dig into this the more it looks to me like we aren't the only ones running into this problem with Google's python libraries. The add-on uses Google's provided python libraries to talk to Google Drive, which use Python's httplib2 under the hood. It seems like like httplib2 might have some strange interaction with how docker handles DNS (the add-on runs inside a docker container Hassio creates for it). The error reports I'm seeing come in with this problem all indicate that google servers are responding to pings just fine, which further lends credibility to that idea, since since the add-on gets those results from a separate command line call avoiding the whole python network stack.
One possible solution I've considered is avoiding Google's libraries altogether and communicating with Drive using regular python3 http requests. That would be a significant rewrite of a big part of the add-on's logic where I'd potentially just end up with the same problem, but I'll try to get sense of how much work it is in reality.
sabeechen
commented
5 years ago Just super weird that it comes and goes seemingly randomly. I can see in the error reports that for most it resolves within an hour, sometimes after a few hours, and for some it never gets through. This is the worst kind of issue to debug.
sabeechen
commented
5 years ago If anyone seeing this error is willing to muck around with their home network's DNS settings, I'd be curious to know if it goes away after configuring your router to use Google's Publis DNS servers (8.8.8.8 and 8.8.4.4) and restarting the machine Home Assistant is running on.
nikiyao
commented
5 years ago So for me at least I may have found the issue. I think I'm having a problem resolving local Host names. Is there anyway to make it call for the local IP address instead?
dhzl84
commented
5 years ago I also have an issue resolving DNS. I have AdGuard running and can see that a bunch of IPs is returned to the hassio network (172.30.33.7) for oauth2.googleapis.com and www.googleapis.com but the addon doesn't seem to see them.
Strange thing is, as soon as I switch over to Pi-Hole the DNS issue is gone, so maybe AdGuard related?
nikiyao
commented
5 years ago That's weird because I run pihole and that's my issue.
jmart518
commented
5 years ago @sabeechen I went ahead and changed my router DNS to 8.8.8.8 and 8.8.4.4, disabled adguard, and hard coded my hassOS box to 8.8.8.8/8.8.4.4 for good measure.
Still getting the same error
sabeechen
commented
5 years ago @jmart518 Thanks for giving it a shot. Did you restart the Home Assistant machine too? DNS caches can stick around for a while, especially through all the layers Docker puts between a container and host environment.
jmart518
commented
5 years ago @sabeechen I sure did. First tried a simple HA restart and then did a full hassOS reboot
justinmartin
commented
5 years ago I am facing the same DNS problems. I've restarted HA, the full system, and the add-on many times.
httplib2.ServerNotFoundError: Unable to find the server at oauth2.googleapis.com
RaveGun
commented
5 years ago Hi,
Sorry it took so long. So I did some test on the dockers containers.
bash-4.4# ping oauth2.googleapis.com ping: bad address 'oauth2.googleapis.com'
bash-4.4# ping 192.168.1.1 PING 192.168.1.1 (192.168.1.1): 56 data bytes 64 bytes from 192.168.1.1: seq=0 ttl=63 time=0.655 ms 64 bytes from 192.168.1.1: seq=1 ttl=63 time=0.407 ms 64 bytes from 192.168.1.1: seq=2 ttl=63 time=0.487 ms 64 bytes from 192.168.1.1: seq=3 ttl=63 time=0.485 ms ^C --- 192.168.1.1 ping statistics --- 4 packets transmitted, 4 packets received, 0% packet loss round-trip min/avg/max = 0.407/0.508/0.655 ms bash-4.4# ping 192.168.1.90 PING 192.168.1.90 (192.168.1.90): 56 data bytes 64 bytes from 192.168.1.90: seq=0 ttl=64 time=0.245 ms 64 bytes from 192.168.1.90: seq=1 ttl=64 time=0.285 ms 64 bytes from 192.168.1.90: seq=2 ttl=64 time=0.221 ms ^C --- 192.168.1.90 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.221/0.250/0.285 ms So there seems to be no DNS resolver available at all.
bash-4.4# ping oauth2.googleapis.com PING oauth2.googleapis.com (2a00:1450:4001:815::200a): 56 data bytes ping: sendto: Address not available
bash-4.4# ping 192.168.1.1 PING 192.168.1.1 (192.168.1.1): 56 data bytes 64 bytes from 192.168.1.1: seq=0 ttl=64 time=0.654 ms 64 bytes from 192.168.1.1: seq=1 ttl=64 time=0.389 ms ^C --- 192.168.1.1 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.389/0.521/0.654 ms
bash-4.4# ping 192.168.1.90 PING 192.168.1.90 (192.168.1.90): 56 data bytes 64 bytes from 192.168.1.90: seq=0 ttl=64 time=0.249 ms 64 bytes from 192.168.1.90: seq=1 ttl=64 time=0.152 ms ^C --- 192.168.1.90 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.152/0.200/0.249 ms bash-4.4#
So it seems that the oauth2 is getting a IPv6 only and this is not enabled on my network. Since I am using a router that is know for having IPv6 security issues I cannot enable this for the time being.
I hope this helps with the debugging.
Have a great day.
dhzl84
commented
5 years ago I did some back and forth testing with Pi-Hole and AdGuard and observed the following behavior. If DNS resolution fails, there is a red box drawn in the AddOns UI showing some DNS debug data.
Strange thing is, that I also find exactly these letters in my DNS query log of AdGuard, so it seems that the AddOn wants to resolve those:
P.S. If anyone knows how to collapse images in Markdown I'd appreciate feedback.
sabeechen
commented
5 years ago @dhzl84 what you're seeing in the error dialog is a parsing error I made getting the DNS info to display there, its actually treating every character in the string "Failed to resolve host" as if it were a separate ip address. I'll be fixing it in a release later today.
@RaveGun Thats getting somewhere! I don't think it would explain everything in this thread but at least I can do something about it. I'll make a release later today with an option to ignore ipv6 addresses.
sabeechen
commented
5 years ago I've made a new release, 0.96, that includes a bunch of stuff to help resolve this issue. Please follow the instructions below if you'd like to give it a shot.
"drive_experimental": true "ignore_ipv6_addresses": true
This will enable a new library I wrote for communicating with Google Drive that ignores IPv6 addresses. Eventually I'd like to make this the default option, but I want to ensure actually fixes the problem with you guys before I witch everyone over with an update. Everything should work, but errors won't be as user friendly yet, you'll just see the raw exception text if problems come up. You'll need to set these through the hass.io addon config page, I haven't made support in the settings UI yet. I try to test thoroughly, but because its a lot of new code there could be new issues with it."drive_ipv4": "172.217.1.202" in the add-on config and restarting the add-on. This will hard-code the DNS resolution for Google's API servers. "172.217.1.202" is one of the addresses that www.googleapis.com resolves to on my network, you can lookup other ipv4 addresses that should work using a tool like this one.jmart518
commented
5 years ago Thanks for the update, I added the two new lines and received a new error:
05-05 21:20:25 ERROR Traceback (most recent call last): File "/app/backup/engine.py", line 162, in doBackupWorkflow self._checkForBackup() File "/app/backup/engine.py", line 401, in _checkForBackup self._syncSnapshots() File "/app/backup/engine.py", line 332, in _syncSnapshots self.folder_id = self.drive.getFolderId() File "/app/backup/drive.py", line 129, in getFolderId folder = self._get(folder_id) File "/app/backup/drive.py", line 145, in _get return self.drivebackend.get(id) File "/app/backup/driverequests.py", line 128, in get return self.retryRequest("GET", URL_FILES + id + "/?" + urlencode(q), is_json=True) File "/app/backup/driverequests.py", line 216, in retryRequest response = request(method, url, headers=send_headers, json=json, timeout=(30, 30), data=data, stream=stream) File "/usr/lib/python3.6/site-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, **kwargs) File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 533, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 646, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='www.googleapis.com', port=443): Max retries exceeded with url: /drive/v3/files/<redacted>?fields=id%2Cname%2CappProperties%2Csize%2Ctrashed%2CmimeType%2CmodifiedTime%2Ccapabilities (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fa0b8249828>: Failed to establish a new connection: [Errno -3] Try again',))
However, adding the last line:
"drive_ipv4": "172.217.1.202"
Appears to have resolved everything for me, just don't know how permanent/reliable the hard IP fix will be.
r3mcos3
commented
5 years ago For me to "drive_ipv4": "172.217.1.202" Has resolved the problem
RaveGun
commented
5 years ago Hi,
Thank you for the update.
The predefined drive IPv4 worked on my side too.
dhzl84
commented
5 years ago Hi, tested it with 0.96.1 but no success with DNS, only hardcoded IPv4 works for me. Anything else I could test for you?
sabeechen
commented
5 years ago Bummer, I was really really hoping this was just an ipv6 problem but its encouraging that the hard-coding works. Are you getting the same error @jmart518 posted (copied below)?
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='www.googleapis.com', port=443): Max retries exceeded with url: <long url> (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fa0b8249828>: Failed to establish a new connection: [Errno -3] Try again',)):
r3mcos3
commented
5 years ago Yes for me it was the same error
Bluhme1
commented
5 years ago Me too. Same error. But hardcoding works
dhzl84
commented
5 years ago I can also confirm the same error message without hardcoded IPv4.
r3mcos3
commented
5 years ago Did a clean install off HA and the addon without the above mentioned fixes and everything works fine.
BertrumUK
commented
5 years ago I was getting the same issue until I added
"drive_ipv4": "172.217.1.202"
Thank you for your work on this excellent addon
WildcardTech
commented
5 years ago I had to add "drive_experimental": true, "ignore_ipv6_addresses": true, "drive_ipv4": "172.217.1.202",
for it to work
sabeechen
commented
5 years ago In the latest version (v0.97) I've made the new Drive library the default, so the "experimental_drive" option no longer has any effect. I've also added failover to known good DNS servers (Google's public DNS) if the add-on runs into problems trying to resolve www.googleapis.com.
The add-on will still respect the drive_ipv4 setting if you have it set, but I suspect that you guys should be able to remove it and just make use of the DNS failover I implemented, which will also keep you up-to-date if google updates their DNS. You might see an error message still show up in the UI for a few minutes if you do this, since it will take at least on failed attempt before using the failover DNS. These options can now also be changed from the settings UI and take effect immediately.
If you still run into problems with this update, I'd be interested in getting a copy of your add-on's debug logs (link in the top right).
dhzl84
commented
5 years ago Nice to hear, I gave it a go right away. I used the DNS failover option to point to my local DNS resolver instead of google. Until now everything runs pretty smooth after updating. Any chance to see when the DNS failover kicks in?
sabeechen
commented
5 years ago Not in the UI, but in the debug logs (link at the top right of the web UI) you'll see a log message that reads
Ran into trouble reaching Google Drive's servers. We'll use alternate DNS servers on the next attempt.So I made a manual backup and restarted the addon. No DNS failover until now! thumbs up
sabeechen
commented
5 years ago I haven't heard of any problems with the latest version (which I think is a good thing) so I'm going to close this issue for now. If anyone is still running into problem, feel free to reopen this and I'll look into it further.
Padre653
commented
5 years ago 05-23 10:41:21 INFO Syncing Snapshots 05-23 10:41:21 DEBUG Making Hassio request: http://hassio/snapshots 05-23 10:41:21 DEBUG Making Google Drive request: https://www.googleapis.com/drive/v3/files/1LV8USkmPT9AkGGJ8xceqS_bDN0lXUMfk/?fields=id%2Cname%2CappProperties%2Csize%2Ctrashed%2CmimeType%2CmodifiedTime%2Ccapabilities 05-23 10:41:26 DEBUG Ran into trouble reaching Google Drive's servers. We'll use alternate DNS servers on the next attempt. 05-23 10:41:26 ERROR Unable to connect to www.googleapis.com 05-23 10:41:26 INFO Another attempt to sync will be made in 3600 seconds
It looks like the application is trying to resolve to DNS even though I have manually specified an IP address.
"max_snapshots_in_hassio": 4, "max_snapshots_in_google_drive": 3, "days_between_snapshots": 7, "use_ssl": false, "snapshot_time_of_day": "03:30", "snapshot_password": "!secret snapshot_password", "send_error_reports": false, "ignore_ipv6_addresses": true, "drive_ipv4": "172.217.9.42"
I'm running 0.97.1
Bluhme1
commented
5 years ago Same here. Worked before. Is there a problem with 0.97.1?
dhzl84
commented
5 years ago For me 0.97.1 still works as flawless as 0.97, no indication of failed DNS. I do not use: "ignore_ipv6_addresses": true, "drive_ipv4": "172.217.9.42"
sabeechen
commented
5 years ago Must have broken something in how this is configured, I'll do some digging.
sabeechen
commented
5 years ago Yup, I accidentally made it read the setting incorrectly. I've fixed it in 0.97.3 (just released).
Padre653
commented
5 years ago That fixed it, thanks!
Please add info about your configuration here, along with a brief description of what you were doing and what happened. Detail is always helpful for investigating an error. You can enable verbos logging by setting {"verbose": true} in your add-on configuration and including that here. :
Traceback (most recent call last): File "/app/backup/engine.py", line 92, in doBackupWorkflow self._checkForBackup() File "/app/backup/engine.py", line 277, in _checkForBackup self._syncSnapshots() File "/app/backup/engine.py", line 214, in _syncSnapshots self.folder_id = self.drive.getFolderId() File "/app/backup/drive.py", line 193, in getFolderId return self._createDriveFolder() File "/app/backup/drive.py", line 201, in _createDriveFolder folder = self._retryDriveServiceCall(self._drive().files().create(body=file_metadata, fields='id')) File "/app/backup/drive.py", line 65, in _drive return build(DRIVE_SERVICE, DRIVE_VERSION, credentials=self.creds) File "/usr/lib/python3.6/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper return wrapped(*args, **kwargs) File "/usr/lib/python3.6/site-packages/googleapiclient/discovery.py", line 224, in build requested_url, discovery_http, cache_discovery, cache, developerKey) File "/usr/lib/python3.6/site-packages/googleapiclient/discovery.py", line 274, in _retrieve_discovery_doc resp, content = http.request(actual_url) File "/usr/lib/python3.6/site-packages/httplib2/init.py", line 1926, in request cachekey, File "/usr/lib/python3.6/site-packages/httplib2/init.py", line 1595, in _request conn, request_uri, method, body, headers File "/usr/lib/python3.6/site-packages/httplib2/init.py", line 1508, in _conn_request raise ServerNotFoundError("Unable to find the server at %s" % conn.host) httplib2.ServerNotFoundError: Unable to find the server at www.googleapis.com