[BUG] retrieval of data from apple fails (main release[2.2.0])

[Systm21]

A few days ago, I suddenly got error messages when I tried to call up location reports. At first I thought, ok, maybe the auth key has expired... So I deleted the auth file. But I just can't get the "fresh" container to run. A corresponding error message is attached below, but I can't quite figure it out... It doesn't even ask for data such as email and password :(

2024-03-02 23:06:57,911 - DEBUG - Searching for token at /app/endpoint/data/auth.json
2024-03-02 23:06:57,911 - INFO - No auth-token found.
2024-03-02 23:06:57,912 - INFO - Trying to register new device.
2024-03-02 23:06:57,918 - DEBUG - Querying http://anisette:6969 for an anisette server
2024-03-02 23:06:57,921 - DEBUG - Starting new HTTP connection (1): anisette:6969
2024-03-02 23:06:57,925 - DEBUG - http://anisette:6969 "GET / HTTP/1.1" 200 566
2024-03-02 23:06:57,928 - DEBUG - Starting new HTTPS connection (1): gsa.apple.com:443
2024-03-02 23:06:58,570 - DEBUG - https://gsa.apple.com:443 "POST /grandslam/GsService2 HTTP/1.1" 200 904
Traceback (most recent call last):
  File "/app/endpoint/mh_endpoint.py", line 159, in <module>
    apple_cryptography.registerDevice()
  File "/app/endpoint/register/apple_cryptography.py", line 76, in registerDevice
    getAuth(regenerate=True, second_factor='trusted_device' 'sms')
  File "/app/endpoint/register/apple_cryptography.py", line 49, in getAuth
    mobileme = icloud_login_mobileme(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/endpoint/register/pypush_gsa_icloud.py", line 40, in icloud_login_mobileme
    g = gsa_authenticate(username, password, second_factor)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/endpoint/register/pypush_gsa_icloud.py", line 76, in gsa_authenticate
    if r["sp"] != "s2k":
       ~^^^^^^
KeyError: 'sp'

[dchristl]

Yes, the container is still working. I've tried to reregister and it just works. I was able to reproduce the error by leaving the username or password blank. Could it be a copy and paste error?

[Systm21]

At that time I had entered the data via the shell, this seems to have changed and is now controlled via the data in the Config.ini. What do I do if my password contains a $ character?

[dchristl]

This change is "relatively" old and should noct impact anything. The keys in the config.ini will be ignored if they are empty. This can be identified when prompted for the username and password in the console. The dollar sign should not cause any issues.

[supaeasy]

I just set it up in docker on a Synology NAS yesterday. Encountered the same error, also after deleting auth file. Just enter credentials via terminal even if it doesn't ask for anything and if it doesn't work restart anisette and restart mh and try again. Container crashed two times doing that but did work eventually.

Another very similar looking error occours when trying to fetch data while being logged into the wrong apple account in mh container.

[Systm21]

I've finally got round to looking at my problem. I had to move the server to another server, among other things. Registration, 2FA, everything went fine, but I got the same error message as before the whole mess.

2024-03-10 12:45:34,808 - DEBUG - http://anisette:6969 "GET / HTTP/1.1" 200 566
2024-03-10 12:45:34,810 - DEBUG - Starting new HTTPS connection (1): gateway.icloud.com:443
2024-03-10 12:45:35,203 - DEBUG - https://gateway.icloud.com:443 "POST /acsnservice/fetch HTTP/1.1" 401 0
2024-03-10 12:45:35,203 - DEBUG - Return from fetch service:
2024-03-10 12:45:35,203 - DEBUG - 
2024-03-10 12:45:35,203 - ERROR - Unknown error occured {e}
Traceback (most recent call last):
  File "/app/endpoint/mh_endpoint.py", line 103, in do_POST
    result = json.loads(r.content.decode())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
172.19.0.4 - - [10/Mar/2024 12:45:35] "POST / HTTP/1.1" 501 -

[dchristl]

Now you get an empty response from Apple (https://gateway.icloud.com/acsnservice/fetch):

2024-03-10 12:45:35,203 - DEBUG - Return from fetch service:
2024-03-10 12:45:35,203 - DEBUG -  <-- HERE IS NORMALLY A JSON FROM APPLE-->

There is an error in my code, which suprresses the ouput of the exception. This should be fixed now. So if you restart your server , the exception should be visible.

Could you try a fetch with Biemsters version The code is mainly based on his work. If this does work, I can dig deeper

[Systm21]

2024-03-18 19:44:40,590 - DEBUG - Starting new HTTP connection (1): anisette:6969
2024-03-18 19:44:40,592 - DEBUG - http://anisette:6969 "GET / HTTP/1.1" 200 566
2024-03-18 19:44:40,594 - DEBUG - Starting new HTTPS connection (1): gateway.icloud.com:443
2024-03-18 19:44:41,167 - DEBUG - https://gateway.icloud.com:443 "POST /acsnservice/fetch HTTP/1.1" 401 0
2024-03-18 19:44:41,168 - DEBUG - Return from fetch service:
2024-03-18 19:44:41,168 - DEBUG - 
2024-03-18 19:44:41,168 - ERROR - Unknown error occured Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/app/endpoint/mh_endpoint.py", line 103, in do_POST
    result = json.loads(r.content.decode())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
172.19.0.4 - - [18/Mar/2024 19:44:41] "POST / HTTP/1.1" 501 -

[Systm21]

Could you try a fetch with Biemsters version The code is mainly based on his work. If this does work, I can dig deeper

As you can see in the screenshot, the Biemster version runs perfectly. Seems like you have a deeper bug, I bet it's the same problem as 40% of the current issues.

[Systm21]

To complete my bug report I'll add the error message in the app, I don't think it's interesting but you never know...

[dchristl]

The issue with the whole thing is that the code calling the Apple interface is identical to the biemster code. I've only expanded logging. The problem is that Apple doesn't provide a response. Are you using the same account for your tests? Could you possibly copy the auth.json from the biemster script into the Docker container and try again? Does the biemster script also run in a Docker container? Maybe there's a different network configuration there. An alternative would be to reset everything and start over.

[Systm21]

Are you using the same account for your tests?

Yes, of course.

Could you possibly copy the auth.json from the biemster script into the Docker container and try again?

I will test that later.

Does the biemster script also run in a Docker container?

No, it runs on the same Server, but without Docker. When i run your script without docker, problem persists.

Maybe there's a different network configuration there. An alternative would be to reset everything and start over.

This is actually completely out of the question, as I had to set up the container completely new on another server a few days ago, as the other one is no longer being rented. I can also rule out an IP block by Apple, because I use really little data, the server is new and the Ip works with Biemster's script.

[dchristl]

My problem is that I'd like to help you, but without reproducing the error, it will be difficult for me. I've tested everything with various accounts and server configurations. The next step would be to add even more debug output.

[Systm21]

auth.json generated with biemsters scipt works perfectly in your docker image!

[Systm21]

I've now tried a bit and played around a bit. Even if I create a fresh token with your script, it works without any problems... Until I restart the anisette or change it slightly (rebuild container etc.). After that I can't get the token to run (Error 502 from Apple Server), no matter which one. Is that normal? Is this a bug?

[dchristl]

This is really strange, can you compare the output of the anisette server, when it's working or not? This is a json normally available on port 6969 on your host or inside your anisette contaioner. It looks like: {"X-Apple-I-Client-Time":"2024-03-22T16:02:14Z","X-Apple-I-MD":"AAAABQAAABAwrJA6YrrGutnaa+13rqlFAAAABA==","X-Apple-I-MD-LU":"6BDF135AB4AB352172454570DCF8D2E08632F0890DB44737391C89F3394F552D","X-Apple-I-MD-M":"osNQm5VeYzWnYrTz8rx/yuWUHtC6NN0f1tMCB9qUhNn/7OuJDkKMTFnqVjiElRm9ixh+/9S1dMdR1iM9","X-Apple-I-MD-RINFO":"17106176","X-Apple-I-SRL-NO":"0","X-Apple-I-TimeZone":"UTC","X-Apple-Locale":"en_US","X-MMe-Client-Info":"<MacBookPro13,2> <macOS;13.1;22C65> <com.apple.AuthKit/1 (com.apple.dt.Xcode/3594.4.19)>","X-Mme-Device-Id":"F7EFCF96-590D-44EA-A655-352E9EEB3746"