add functions to get account data in various formats

caver456 commented 1 year ago

for radiolog

caver456 commented 1 year ago

This issue was moving along pretty well, until trying getAccountInfo for an internet session. This happens regardless of whether the session has an associated map before the getAccountInfo call. Geometry operations, queries, etc all run correctly if a map is open - but getAccountInfo dies. Here's the entire transcript, opening a web map:

startup
12:23:24 [sartopo_python:230:INFO] Associating map 44H1C with this SartopoSession object...
12:23:24 [sartopo_python:916:WARNING] A request is about to be sent to the internet, but accountIdInternet was not specified.  The request will use accountId, but will fail if that ID does not have valid permissions at the internet host.
12:23:24 [sartopo_python:925:INFO] sending post to https://sartopo.com/api/v0/userdata
12:23:24 [sartopo_python:951:INFO] {'json': '{"map": {"center": [-120, 39], "zoom": 13}}', 'id': '.....', 'expires': 1671913524359, 'signature': '.....'}
12:23:24 [sartopo_python:330:INFO] Initial cache population begins.
12:23:24 [sartopo_python:916:WARNING] A request is about to be sent to the internet, but accountIdInternet was not specified.  The request will use accountId, but will fail if that ID does not have valid permissions at the internet host.
12:23:24 [sartopo_python:607:INFO]   Updating "ids"
12:23:24 [sartopo_python:611:INFO]   processing 38 feature(s):['007b88f3-b6ff-4bb6-9ab5-109f41c12d50', '0917e083-b549-4797-a6e3-05c866173e00', '1f784a3b-2438-4ef7-8bcf-14a967573909', '2a3d99db-442e-49b3-82f2-8e7414d30907', '364c56f2-7344-4113-859e-89ee4546643f', '46e6d2b2-4a83-4582-82d2-f81879b5be9e', '5d3c58d7-5ff2-42b2-967f-08710cf18200', '61973425-7d36-41b4-a1e1-dc868bc93fa5', '630eab7d-7a28-4aaf-baee-4c7b38390d73', '6f2c8d67-d5b5-4f8b-a042-df19e45c72ff', '707ba299-118f-40d5-97e6-3cc4537865fb', '75d2a649-9bc8-4142-82c1-c727853c17ec', '79eb8c9d-2f18-4cc9-b020-4d8b8f60324a', '8fa50984-e2be-472c-a594-8a582ae203f9', '96fd6b16-525d-49a6-9eb2-85dafcd86a9c', '9ac78930-a6f9-42a7-990b-6d1276f885f6', 'ab67b9b8-66d8-41c7-9b46-fbd8b168ded2', 'b4ee4328-4020-492d-8640-0a19f6451ed9', 'b617acfb-f00a-406f-ac4d-e9bf239f3bb9', 'b6f0002b-b582-42e2-84a6-198eca8340d9', 'b7e3e0d3-3f37-43d9-81d1-8a201a95e17f', 'c011beba-abde-4c3e-8fcb-58ba3ae918d4', 'ca5d4d55-cdcd-4639-b512-eaa2becc4cb7', 'cbb03e41-2272-4702-bb09-3aaea6412545', 'e78b10d6-63f0-4479-9953-9f033775b8f4', 'e98b91e7-0b69-46f7-9e7f-2e0c8c7f7305', 'f9d97476-b0fe-4bcd-9ca7-e7479b781edc', 'fd5f6c5f-17b4-4277-adfd-3cf2da033361', 'c5b6a59b-c1a1-4afe-afb6-638673fa9bed', '57d78d0d-ba66-4714-bd9e-435c6d5328da', '964defe6-f84e-4da5-b469-e3e5bd0f736f', 'fe22a1ed-101a-4898-9434-d57238c2d2aa', 'd1cc98d7-0bca-49d9-8f27-b988b25c18f7', '383ffb41-4605-45d8-a513-48951e9240af', 'ac9ec980-d5d3-4fbb-96c0-169d9213bf7e', '8ece58ef-b926-49ac-8f8a-3d1f25ad79f4', 'c0ef7088-bcd6-4281-b74d-0ee37d92b332', 'f7d675e0-7e58-4114-bb06-f63eec71fb69']
12:23:24 [sartopo_python:332:INFO] Initial cache population complete.
map list:
12:23:24 [sartopo_python:530:INFO] Getting account data:
12:23:24 [sartopo_python:532:INFO]   sending GET request to https://sartopo.com/sideload/account/11UEE9.json?json=%7B%22full%22%3Atrue%7D
12:23:24 [sartopo_python:534:INFO] response text:{"code":401,"message":"login required","status":"error","timestamp":1671913404813}
12:23:24 [sartopo_python:539:ERROR]   accound data request returned response code 401.  Aborting.
12:23:24 [sartopo_python:2618:CRITICAL] Uncaught exception:
Traceback (most recent call last):
  File "C:\Users\caver\Documents\GitHub\sartopo_python\sartopo_python\test.py", line 45, in <module>
    print(sts.getMapList('NCSSAR'))
  File "C:\Users\caver\Documents\GitHub\sartopo_python\sartopo_python\sartopo_python.py", line 554, in getMapList
    self.getAccountData()
  File "C:\Users\caver\Documents\GitHub\sartopo_python\sartopo_python\sartopo_python.py", line 540, in getAccountData
    raise STSException
sartopo_python.STSException
12:23:24 [sartopo_python:801:INFO] SartopoSession instance deleted for map 44H1C.

If the session already has a map, that means the signed userdata post request was already sent (during sendUserData in the first few lines of the transcript). What else is needed by getAccountData that isn't needed by the other geometry operations in the same session? Does the get request need to be signed? So far, only post requests have been signed.

caver456 commented 1 year ago

In chrome devtools, watching network traffic for the browser window, beginning with a sartopo.com window which is logged out, the only interesting things that happen during login are: 1) the google auth things 2) GET https://sartopo.com/sideload/constants.json?ts=1671818660771 - return value includes a bunch of generic icons and layers stuff 3) GET \<account>.json - same as the getAccountData request in question, but with no url-encoded payload.

Maybe 2 or 3 actually accomplish the login? Trying it out now...

caver456 commented 1 year ago

Nope, it's neither of those. 2 (sideload/constants) returns the expected json, and 3 (sideload/\<account>.json with no payload also returns 401 'login required'.

Also, the signatures in the signed requests are created using the mapID; since there's no mapID here, the signature wouldn't be of the same sort anyway.

caver456 commented 1 year ago

Had a theory: the requests.session object (self.s=requests.session() is the first thing that happens in init), and is used for most of the requests in the code, and was hopefully already logged in. The theory is that I sent the account data request NOT using self.s therefore not using the logged-in session. But, that's not the case, all the requests in the code in question are using self.s.

caver456 commented 1 year ago

Tried signing the get request in the same manner as post requests, just in case that's actually required; still got 401.

We know that this works from an interactive session - so maybe the oath login-from-google is really required. Still, seems strange that the oath login isn't required for any of the other requests.

caver456 commented 1 year ago

Tried intentionally editing the POST hashed data to see what happens when an incorrectly signed request is sent - this is the reponse:

05:19:31 [sartopo_python:1037:INFO] response code = 401 05:19:31 [sartopo_python:1096:INFO] response:<Response [401]> 05:19:31 [sartopo_python:1105:WARNING] response status other than "ok": {'code': 401, 'message': 'User not logged in', 'status': 'error', 'timestamp': 1673443172213}

So the code is the same (401) but the message is slightly different ('User not logged in' vs 'login required'). Still, this is enough to make a guess that the 401 response in the GET request is also do an incorrectly signed request. So - compare the pre-hased request text in more detail and revise the pre-has GET request body.

caver456 commented 1 year ago

Resolved this 401 mystery with the help of Matt J and Steve K.

basically: send a signed GET request to https://sartopo.com/api/v1/acct//since/0 with json=’’

The discovery that was holding up progress for months was: In the data to be hashed, make sure you include the newline after the expires timestamp string. The hashed data must have a newline after the URL tail, and another newline after the expires timestamp.

That realization should clear the logjam for this issue and for #48.

caver456 commented 5 months ago

Additional info just received: GET requests for secret maps must be signed (this part might always have been the case); team maps are now secret by default; and new throttling / speedbumps have been introduced. So: creating a new issue for signed GET requests, which will enable the fixing of this issue too. The previous comment, and emails from around that time, should show the way.

caver456 commented 1 month ago

Picking back up on this issue, now that the signed GET feature is in place.

There are actually a few tasks to take care of here:

[x] ensure proper pre-hashed data: "GET /api/v1/acct/[accountID]/since/0[newline][expires_ms][newline]" per notes from 2/16/23
[x] parse the response to generate the sorted list of map names and IDs (i.e. the point of this issue)
[x] allow mapless SartopoSession (make new function openMap; don't call it from setupSession; call it from init if needed, and allow calling it later from client code)
[ ] QA: confirm signed GET requests on maps of all sharing levels are still working before merging this branch to master

The first and third items are already working on the old temp file sartopo_python_authTest.py, but, that file also has apparently-unnecessary auth code; so, taking the important parts of that file, while minimizing modifications to and signed-GET-related code (or any other code) is the goal.

The first two items can be tested using a 'normal' session i.e. with a mapID; but, the normal use case here is probably to get the map list before any mapID is known, i.e. on a mapless session. Since splitting setupSession is a bit more invasive, the third item should be a separate commit from the first two items.

caver456 commented 1 month ago

confirmed MAI's (Mutual Aid Incidents formerly Events) work with the existing code, and appear as just another group account

caver456 commented 1 month ago

Expanded the name of this issue, to include #48 and other functions

[x] get list of maps (names and IDs) for account, sorted chronologically (the original title of this issue)
[x] #48
[x] get full nested list of maps of all accounts - can be used by downstream tools to make a full map chooser GUI
[x] list all team account memberships

caver456 commented 1 month ago

QA complete for all four permissions!

ncssar / sartopo_python

add functions to get account data in various formats #50