InvalidCookies shortly after successful request

[Abolfazl]

I have a script that runs every 3 minutes that runs Service(cookies_file=cookies_file, authenticating_account=google_email)

Recently I noticed that even after updating the cookie file, after about 15 minutes, the Service() function returns InvalidCookies.

Is anyone else having a similar issue? I cleared my cookies and tried again and I still get InvalidCookies after a short period of time after my first successful pull.

[costastf]

I have not had any other reports of failure soon after authentication. You mention running the script every 3 minutes and that after 15 minutes you get InvalidCookies but you mention getting it after your first successful report. Do you only get on report so things get invalidated after 3 minutes or do you get ~5 reports and then things fail? It might help to identify the time taken. Are you using a gsuite account or a standard gmail account?

[Abolfazl]

I have a standard gmail account. I had to clear cookies a couple times and relogin and use those cookies for google.com

When I tried yesterday, it would work for ~1-5 times (3-15 minutes) before throwing InvalidCookies. It seems to be working now though so not sure what was up.

[Abolfazl]

@costastf I started getting this problem again and I figured out how to reproduce it (but not sure how to fix).. So on my computer, I pull the Google cookies and load it into this script and it works. I have this script running on a NAS that runs every 5 minutes. If I turn my computer off right away, I am able to pull the locations just fine for hours and days. However, as soon as I turn my PC on again and load Chrome, I get InvalidCookies within 10 minutes (I am not interacting with Google Maps or Google.com in any way during these 10 minutes).

I have tried completely clearing my google cookies and then retrying and I get the same issue

[costastf]

So I guess chrome is interacting in some way, sending telemetry and invalidating the cookie. What I would suggest, is to use a Google account only for tracking (not your main gmail) use an incognito browser, make the cookies and close the tab immediately.

[Abolfazl]

That does not work out for my solution. Is there a way to disable these telemetry updates or to pull fresh cookies?

[costastf]

Have you tried making the cookies with Firefox?

[Abolfazl]

I use the Export Cookies extension on FF and I get InvalidCookies every time. When I use cookies.txt extension for Chrome, the cookies work but only for 10-15minutes (which looks to be similar issue as https://github.com/costastf/locationsharinglib/issues/112)

[costastf]

I think that export cookies on ff comments out some cookies. Do you also choose the http_only ones? Please try those too and uncomment any commented cookie and see how it goes.

[Abolfazl]

I ended up using a different extension for Firefox for exporting cookies but this one works and has been for the last day. Thanks for the suggestion :)

[carefulcomputer]

@Abolfazl which extension did you use ? I have similar problem where google cookies expire after some time. @costastf , does locationsharinglib automatically update the cookies if they change when calling google maps for location and google sends new set-cookie headers ?

[costastf]

Hi @carefulcomputer. The library uses requests session so while it runs, if the service does send a set cookie header it will be respected. But there is no export of the session cookies back to a file so you would always have the initially exported from the browser cookies in your file. I hope it makes sense.

[Abolfazl]

Hi @carefulcomputer. The library uses requests session so while it runs, if the service does send a set cookie header it will be respected. But there is no export of the session cookies back to a file so you would always have the initially exported from the browser cookies in your file. I hope it makes sense.

Would it be best to modify the script to export the saved cookies?

[costastf]

Not sure to be honest that it would help. Theoretically the cookies do not expire. On the other hand exposing a method for export wouldn't be hard, I am just not sure of the benefit.

[carefulcomputer]

@costastf here is what we are requesting, please see this makes request..

1. User exports initial set of cookies from browser and configures in config file like they currently do
2. library uses the cookies and gets locations like it currently does
3. When fetching the shared location data, if library sees 'set-cookie' headers with new set of cookies it rewrites the config file (or exports a method for other software to use) -- New

I hope this will keep the session alive for longer. Another suggestiion, (if above suggestion doesn't make sense) - update the docs and maybe share instructions like to make it explicit for the users -

1. Export the cookies using X plugin
2. Delete the cookies using Y plugin - this is important to keep the cookies alive. if the browser is logged in with cookies, even if user doesn't activly go to google software, sometimes due to google iframes on other websites, cookies get replaced and session stops working for locationsharinglib
3. Reset the cookies in the config when original cookies expire

BTW, not sure I understand your comment about "Theoretically the cookies do not expire.", because in cookie file I do see expiry date of cookies (mine is valid for about 8-9 months). I might be missing something.

[costastf]

Hi @carefulcomputer . Thanks for all the suggestions! If we end up clarifying that the cookies do get renewed (I do not have any indication of that) I could work on a feature like the one you define. For point two, I would really appreciate a PR with the documentation changes. As for the not expiring, in my mind 9 months was long enough 😊. I would not mind working with people to improve this.

[costastf]

Let me elaborate a bit on why I am not certain of any added value of cookies export from the tool. Any renewal cookie is already handled and extends the current session transparently by requests. So the service will keep extending. If for some reason it stops working and needs to be restarted the most probable reason would be cookie invalidation for some reason so saving the currently not working cookies would not make much sense. If you are thinking of a usage case where a user would want to terminate the process and restart at a later stage and not willing to go through the process of retrieving new cookies, that might be a valid usage case and of course having a manual export method for cookies is easy to implement.

[carefulcomputer]

Restarting could happen due to multiple reasons outside of invalid cookies. i am thinking of use case of Home Assistant (where this library is used), when home assistant is restarted (for bunch of reasons) it will lose the new cookies and session. Having a way for this lib to survive restarts (as long as cookies were valid before restart) and maintain the session will be awesome.

[costastf]

I understand the usage case, this is the reason why I made this library in the first place :) . The problem is how to implement that feature. We could for example export the cookies every time we retrieve shared info and the cookies are still valid but I would consider that excessive. Also automagically exporting the cookies would be fine i guess with a setting of auto_export_in_seconds or something but then location becomes a problem. If the tool exports the cookies is it supposed to pick them up automatically on start up? How could that override the configuration of the add on where the path of the cookies is specified by the user? And if we do wouldn't that create more confusion and trouble if the cookies end up being invalid? Should the tool delete the automatically created cookies if they turn out to be invalid? Do you understand my trouble? I feel like every path forward is potentially full of very confusing magic. Any suggestions? @carefulcomputer

[carefulcomputer]

i can totally understand what you are saying (i code in java and javascript). What are your thoughts about keeping a semaphore. So whenever library sees cookies which are new, it updates a semaphore that cookies have been updated and provide a method which will allow fetching. Then the user of library (in this case Home assistant) could check the semaphore at it's schedule/logic and if it is updated then it fetches the new set of cookies and write it to the file ?

[costastf]

That is pretty straight forward but still requires manual action. If that is ok then I can plan for implementing this. Of course the only tricky part is that exporting the cookies is going to be a race condition and not inherently "safe" since they might be invalidated at any point. I guess we can just explain that caveat in documentation. Can I depend on anyone on a PR for the documentation?

[Abolfazl]

The cookie expiration date is a little arbitrary, even though it says it is valid for 9 months does not mean it will actually be valid for that long.

How about this kind of logic.. If a flag is passed in to Service() function to save cookies, it will overwrite the cookies.txt input with the new cookies if it was able to successfully connect?

[carefulcomputer]

another suggestion, how about instead of semaphore, use a 'version' which gets incremented everytime there is a new cookie. that version will be updated only after new cookies are available to export. any calling app (e.g Haas) can check if version has incremented from last version it saw, and export if it has.

[carefulcomputer]

One more suggestion, how about allow the users of library to setup a callback in library which passes the new cookies to callback function when library receives new cookies ?

[costastf]

Hi all, thanks for all the suggestions. At this point in time I would like to keep things as simple as possible until we prove value for this approach so at branch https://github.com/costastf/locationsharinglib/tree/feature-observe-cookies I have implemented a logger that logs under info when there is a set cookie sent. Can you please test the code and let me know if this approach actually adds any value (getting update cookies that is)?

[costastf]

I am getting a logging like

INFO:locationsharinglib.Service:Got set cookie SIDCC=VALUE; expires=Thu, 29-Aug-2024 13:07:57 GMT; path=/; domain=.google.com; priority=high, __Secure-1PSIDCC=VALUE; expires=Thu, 29-Aug-2024 13:07:57 GMT; path=/; domain=.google.com; Secure; HttpOnly; priority=high, __Secure-3PSIDCC=VALUE; expires=Thu, 29-Aug-2024 13:07:57 GMT; path=/; domain=.google.com; Secure; HttpOnly; priority=high; SameSite=none

It would be nice if we figure out which cookie sent actually extends the session and we only respond to that instead of any advertisement cookie google sends. If you would be so kind to help me triangulate that, that would be awesome. What I am thinking is that if we actually get refresh cookies then we can implement all of your suggestions in order as long as there is valuable outcome and it is worth it. Searching for the cookies like so https://www.google.com/search?q=__Secure-1PSIDCC returns many websites that describe the cookies that they manage and most of the google ones are in those lists.

[vfont]

/> I use the Export Cookies extension on FF and I get InvalidCookies every time. When I use cookies.txt extension for Chrome, the cookies work but only for 10-15minutes (which looks to be similar issue as #112)

Hello, I think the problem is that when generating your cookies with your browser, you leave the browser open and after 15 minutes it generates new cookies, invalidating the cookies generated previously. You must close the browser so this does not happen.

[costastf]

I think so too, people should also use an extension to delete the cookies from the browser so there is no refresh. Also that is why it is suggested to use a third party account to monitor the accounts, so there is not need to be used ever again to invalidate the cookies.

[carefulcomputer]

@costastf , here is new take on Home Assistant Google map location. Would you be able to please see if you can provide a callback which could provide the new cookies back to this new Home Assistant integration ? I am wondering as long as new google cookie names match the ones provided as input to the lib, it would be the right set ?

[costastf]

Hi @carefulcomputer , thanks for the heads up, i had picked up the mention and sorry for not responding earlier, time is a little limited lately. As I mention on this thread, there is a branch where I have implemented a hook that logs when a new cookie is received so we can check whether google actually sends renewed cookies. I am personally not convinced that this is the case. I think that cookies are not refreshed in this manner and in my experience most of the issues with expiring cookies are from people that do not use a third unused account to share their location but use their personal account and then log in somewhere else invalidating the cookies.

If following the above branch someone can validate that indeed cookies are refreshed and the process still works then of course, coming up with some call back to actually export them would be easy and I would be more than happy to implement that functionality. I do not use the HA integration any more and my time is pretty pressed lately so I would appreciate it if someone with vested interest can validate the finding because at this point it would be a shame to invest time to implement a feature that would not do anything due to our missundestanding of the remote service.

I hope this makes sense and looking forward to helping solve this in a structured way. Cross mentioning the custom HA integration issue here for posterity #https://github.com/pnbruckner/ha-google-maps/issues/8#issuecomment-1925414469