Is the endpoint compatible with the Android app?

Systm21 commented 8 months ago

I have now tried a bit, but am again faced with the problem that the endpoint is online, but only throws Python errors (log from the docker container) as soon as the Android app requests data. A 502 Bad Gateway error then appears in the app. I already had this before when I called mh_endpoint.sh without a container.

New app and old Mac-based endpoint works wonders. The new endpoint shows "NOTHING TO SEE HERE" on Ip:port. I don't know what the problem is.

dchristl commented 8 months ago

Hello @Systm21 ,

I need a little bit more informations to help you solve it. I tested the app with the web and the Android-version.

Did you use the latest docker container? Try to drop it and pull it again
What does the log says. docker logs -f macless-haystack
Have you succesfully registered your device?
Did you use SSL?
Have you changed the port?

Regards, Danny

Systm21 commented 8 months ago

I can't provide a log at the moment, but I'll do so when I get the chance.

I have deleted everything old
I was able to register the "device" without any problems
you can say it has ssl to the outside, because the traffic runs from Docker into a cloudflared tunnel (http) and the domain then responds to https:// (sounds complicated, but it worked with the other Haystack projects without any problems)
I have not changed the port, outside the container it also runs into the cloudflared tunnel. -The problems only start as soon as reports are to be fetched.

dchristl commented 8 months ago

I think I need the logs, otherwise I can't see what should go wrong. My guess is the SSL or tunneling part. You can alos try to run the endpoint on your local machine first, to figure out if there is already problem or not and to exclude the tunneling.

Systm21 commented 8 months ago

----------------------------------------
2024-01-17 00:48:10,755 - DEBUG - Getting with post: b'{"ids":["here is a key"],"days":7}'
2024-01-17 00:48:10,755 - DEBUG - Querying for 7 days
2024-01-17 00:48:10,755 - DEBUG - Querying http://anisette:6969 for an anisette server
2024-01-17 00:48:10,757 - DEBUG - Starting new HTTP connection (1): anisette:6969
2024-01-17 00:48:10,759 - DEBUG - http://anisette:6969 "GET / HTTP/1.1" 200 566
2024-01-17 00:48:10,761 - DEBUG - Starting new HTTPS connection (1): gateway.icloud.com:443
2024-01-17 00:48:11,291 - DEBUG - https://gateway.icloud.com:443 "POST /acsnservice/fetch HTTP/1.1" 401 0
2024-01-17 00:48:11,292 - DEBUG - Return from fetch service:
----------------------------------------
Exception occurred during processing of request from ('172.90.80.1', 50334)
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/socketserver.py", line 318, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/local/lib/python3.12/socketserver.py", line 349, in process_request
    self.finish_request(request, client_address)
  File "/usr/local/lib/python3.12/socketserver.py", line 362, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/local/lib/python3.12/socketserver.py", line 761, in __init__
    self.handle()
  File "/usr/local/lib/python3.12/http/server.py", line 436, in handle
    self.handle_one_request()
  File "/usr/local/lib/python3.12/http/server.py", line 424, in handle_one_request
    method()
  File "/app/endpoint/mh_endpoint.py", line 67, in do_POST
    result = json.loads(r.content.decode())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
----------------------------------------

Here is the Log...

dchristl commented 8 months ago

It seems the Aple server returns an invalid response, maybe it is an empty one. After that, the endpoint seems to crash. I've added some better logging and error handling. Please simply restart the Docker container, the code updates automatically. If the error still occurs, please post the current log again.

Systm21 commented 8 months ago

2024-01-20 10:12:33,929 - DEBUG - Querying for 7 days
2024-01-20 10:12:33,934 - DEBUG - Querying http://anisette:6969 for an anisette server
2024-01-20 10:12:33,943 - DEBUG - Starting new HTTP connection (1): anisette:6969
2024-01-20 10:12:33,949 - DEBUG - http://anisette:6969 "GET / HTTP/1.1" 200 566
2024-01-20 10:12:33,953 - DEBUG - Starting new HTTPS connection (1): gateway.icloud.com:443
2024-01-20 10:12:34,226 - DEBUG - https://gateway.icloud.com:443 "POST /acsnservice/fetch HTTP/1.1" 401 0
2024-01-20 10:12:34,227 - DEBUG - Return from fetch service:
2024-01-20 10:12:34,227 - DEBUG - 
2024-01-20 10:12:34,227 - ERROR - Unknown error occured {e}
172.90.80.0 - - [20/Jan/2024 10:12:34] "POST / HTTP/1.1" 501 -

Assuming I entered the wrong password by mistake, the auth sms shouldn't have come, should it?

dchristl commented 8 months ago

It seems like your authentication didn't work. Normally, the server would terminate at the beginning in such a case. Did you possibly transfer your auth.json from another system? That could at least explain the issue. You could stop the container and perform the authentication again by deleting the auth.json:

docker stop macless-haystack
sudo rm /var/lib/docker/volumes/mh_data/_data/auth.json #maybe adjust for different setups

Restart your container in interactive mode and repeat the authentication

docker start -ai macless-haystack

If the authentication is succesful you can restart your cotainer in background:

docker restart macless-haystack

Assuming I entered the wrong password by mistake, the auth sms shouldn't have come, should it?

Yes, there should be an error, I've tried it and there is something like this:

2024-01-20 11:41:27,605 - DEBUG - Querying http://anisette:6969 for an anisette server
2024-01-20 11:41:27,607 - DEBUG - Starting new HTTP connection (1): anisette:6969
2024-01-20 11:41:27,609 - DEBUG - http://anisette:6969 "GET / HTTP/1.1" 200 566
2024-01-20 11:41:27,611 - DEBUG - Starting new HTTPS connection (1): gsa.apple.com:443
2024-01-20 11:41:28,345 - DEBUG - https://gsa.apple.com:443 "POST /grandslam/GsService2 HTTP/1.1" 200 1269
2024-01-20 11:41:28,347 - WARNING - This implementation only supports s2k. Server returned s2k_fo
Traceback (most recent call last):
  File "/app/endpoint/mh_endpoint.py", line 124, in <module>
    apple_cryptography.registerDevice()
  File "/app/endpoint/register/apple_cryptography.py", line 76, in registerDevice
    getAuth(regenerate=True, second_factor='trusted_device' 'sms')
  File "/app/endpoint/register/apple_cryptography.py", line 49, in getAuth
    mobileme = icloud_login_mobileme(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/endpoint/register/pypush_gsa_icloud.py", line 41, in icloud_login_mobileme
    pet = g["t"]["com.apple.gs.idms.pet"]["token"]
          ~^^^^^
TypeError: 'NoneType' object is not subscriptable

Systm21 commented 8 months ago

Looks like a new registration has brought the desired success. Strange, because I received the sms as normal before.

Is there any way to secure the endpoint with some kind of key or user:pass? Now that this is more than just an "experiment", maybe that makes sense.

dchristl commented 8 months ago

Great, that it works now. SIt seems strange that Apple allows registration and then denies access to the fetch service. I will try to improve the stability at this point.

Is there any way to secure the endpoint with some kind of key or user:pass?

It is one of the top-features on my list. I will added as next, when my time allows it.

dchristl / macless-haystack

Is the endpoint compatible with the Android app? #35