Ubiquity Service 503 HTTP Error for api.files() calls

dsouzarc commented 4 years ago

The problem

Ubiquity Service Error

After logging in (and verifying that logging in is working via printing out a list of verified devices on the iCloud account), calling api.files.dir() and api.files['com~apple~Notes'] raises a PyiCloudAPIResponseException due to Service Unavailable (503), however, Apple's System Services page shows that all systems are online.

Environment

pyiCloud release with the issue (pip show pyicloud): Version: 0.9.7
Last working pyiCloud release (if known):
Service causing this issue: Ubiquity Service
Python version (python -V): Python 2.7.16
Operating environment (project deps/Docker/Windows/etc.): macOS

Traceback/Error logs


>>> api.files.dir()

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pyicloud/services/ubiquity.py", line 43, in __getattr__
    return getattr(self.root, attr)
  File "pyicloud/services/ubiquity.py", line 20, in root
    self._root = self.get_node(0)
  File "pyicloud/services/ubiquity.py", line 29, in get_node
    request = self.session.get(self.get_node_url(node_id))
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "pyicloud/base.py", line 83, in request
    self._raise_error(response.status_code, response.reason)
  File "pyicloud/base.py", line 137, in _raise_error
    raise api_error
pyicloud.exceptions.PyiCloudAPIResponseException: Service Unavailable (503)

>>> api.files['com~apple~Notes']

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pyicloud/services/ubiquity.py", line 46, in __getitem__
    return self.root[key]
  File "pyicloud/services/ubiquity.py", line 20, in root
    self._root = self.get_node(0)
  File "pyicloud/services/ubiquity.py", line 29, in get_node
    request = self.session.get(self.get_node_url(node_id))
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "pyicloud/base.py", line 83, in request
    self._raise_error(response.status_code, response.reason)
  File "pyicloud/base.py", line 137, in _raise_error
    raise api_error
pyicloud.exceptions.PyiCloudAPIResponseException: Service Unavailable (503)

Additional information

Is there a way to see all files (including iPhone backups) stored on iCloud? Example: iterating through these purple Backup files:

ixs commented 4 years ago

Seeing the same problem here. Put a bit of debugging code in and the server replies with a 503 and the body "Bad Gateway".

The URL for the error that is causing this is https://p51-ubiquityws.icloud.com:443/ws/<dsid>/item/0...

Quentame commented 4 years ago

I don't think <dsid> is normal in the URL

ixs commented 4 years ago

I do not know. I had a look and tried seeing what icloud does with ubuiquityws.icloud.com but I did not find a single thing that uses that webservice endpoint. The icloud.com website in a browser is not using it. Things like Notes on icloud.com which I know where using ubiquityws before now seem to use ckdatabasews.

I am wondering if ubuiquityws is maybe not used at all anymore? If looking at the Drive webservice there's a /RetreiveAppLibraries endpoint that seems to be delivering App-specific folders. Maybe that's a replacement?

@Quentame You got any idea?

ixs commented 4 years ago

So, I did a bit more diging. Here's my theory:

The URL is working fine and should be https://p51-ubiquityws.icloud.com:443/ws//item/0.
The Bad Gateway error was an apple error, because it is gone.
Instead there's now "Http/1.1 Service Unavailable " being returned which looks like a "well-formed" error.

Looking further into this error, my current theory is that ubiquityws is deprecated and not usable anymore for most people. I did install MacOS 10.9 in a VM and grabbed the traffic as that is using the older ubd system according to https://eclecticlight.co/2016/02/09/icloud-some-explainers/.

What I did get was a system that does not sync any files and an error message in the traffic dump that indicates that the user was migrated. I am not sure what the replacement service is. There's a call in drive that will display the App Library which looks suspiciously like the old ubuiquity data but does not contain everything I believe. Some stuff might be in the ckdatabasews service, which seems to be true for notes at least... https://github.com/picklepete/pyicloud/pull/294 has the App Library call added but this needs further development time.

picklepete / pyicloud