search

costastf / locationsharinglib

A library to retrieve coordinates from an google account that has been shared locations of other accounts.
MIT License
170 stars 29 forks source link

No module named 'loud' #104

Closed GuyKh closed 1 year ago

GuyKh commented 1 year ago

Why am I getting this when trying to load cookies.txt

Traceback (most recent call last):
  File "/app/app.py", line 20, in <module>
    service = Service(cookies_file=cookies_file, authenticating_account=google_email)
  File "/usr/local/lib/python3.6/dist-packages/locationsharinglib/locationsharinglib.py", line 101, in __init__
    self._session = self._validate_cookie(cookies_file or '')
  File "/usr/local/lib/python3.6/dist-packages/locationsharinglib/locationsharinglib.py", line 104, in _validate_cookie
    session = self._get_authenticated_session(cookies_file)
  File "/usr/local/lib/python3.6/dist-packages/locationsharinglib/locationsharinglib.py", line 123, in _get_authenticated_session
    session.cookies.update(pickle.load(cfile))
ModuleNotFoundError: No module named 'loud'

Tried also adding loud to the requirements.txt - which ended up in 

...
    session = self._get_authenticated_session(cookies_file)
  File "/usr/local/lib/python3.6/dist-packages/locationsharinglib/locationsharinglib.py", line 123, in _get_authenticated_session
    session.cookies.update(pickle.load(cfile))
  File "/usr/local/lib/python3.6/dist-packages/loud/__init__.py", line 1, in <module>
    from loud import Loud
ImportError: cannot import name 'Loud'
costastf commented 1 year ago

Hi @GuyKh , I do not use any module called loud in the code. What I can see from the snippet you provide is that the old pickle loading mechanism is being used and not the new one clear text cookies loading one. If you use the pickle version it could be that the pickled file is actually trying to load and execute code. Please see first warning at https://docs.python.org/3/library/pickle.html . Please use the newest clear text cookies method and let me know how that works for you. You should consider the pickle method obsolete.

GuyKh commented 1 year ago

Hi @costastf, thanks for your reply.

This is the code I use I don't use the pickle explicitly, and just use a cookies.txt as extracted via various tools. Is this wrong? As you can see, this exception is thrown in the initializatoin of locationsharinglib. What should I do differently?

GuyKh commented 1 year ago

Even when running it in Python3 REPL -

>>> from locationsharinglib import Service
>>> cookies_file = 'cookies.txt'
>>> google_email = 'mymail@gmail.com'
>>> service = Service(cookies_file=cookies_file, authenticating_account=google_email)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/guyk/git/locationsharing/.venv/lib/python3.11/site-packages/locationsharinglib/locationsharinglib.py", line 102, in __init__
    self._session = self._validate_cookie(cookies_file or '')
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/guyk/git/locationsharing/.venv/lib/python3.11/site-packages/locationsharinglib/locationsharinglib.py", line 105, in _validate_cookie
    session = self._get_authenticated_session(cookies_file)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/guyk/git/locationsharing/.venv/lib/python3.11/site-packages/locationsharinglib/locationsharinglib.py", line 121, in _get_authenticated_session
    session.cookies.update(pickle.load(cfile))
                           ^^^^^^^^^^^^^^^^^^
  File "/home/guyk/git/locationsharing/.venv/lib/python3.11/site-packages/loud/__init__.py", line 1, in <module>
    from loud import Loud
ImportError: cannot import name 'Loud' from partially initialized module 'loud' (most likely due to a circular import) (/home/guyk/git/locationsharing/.venv/lib/python3.11/site-packages/loud/__init__.py)
costastf commented 1 year ago

This issue now you might be having because you added loud to your requirements. Could you please try the library out in a new virtual env and let me know if that works for you? This might be to some kind of shadowing of something on your namespaces so lets try to isolate the library and make sure that it still works. So basically if you do something like 

pipenv install locationsharinglib

and then activate the virtualenv and try again the functionality and let me know and we take it from there.

GuyKh commented 1 year ago

Empty folder with file: requirements.txt contiains locationsharinglib>=4.2.0 and cookies.txt

$ pipenv shell
...
$ pipenv install locationsharinglib
...
$ python3>
>>> from locationsharinglib import Service
>>> cookies_file = 'cookies.txt'
>>> google_email = 'mymail@gmail.com'
>>> service = Service(cookies_file=cookies_file, authenticating_account=google_email)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/guyk/git/tmp/.venv/lib/python3.11/site-packages/locationsharinglib/locationsharinglib.py", line 102, in __init__
    self._session = self._validate_cookie(cookies_file or '')
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/guyk/git/tmp/.venv/lib/python3.11/site-packages/locationsharinglib/locationsharinglib.py", line 105, in _validate_cookie
    session = self._get_authenticated_session(cookies_file)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/guyk/git/tmp/.venv/lib/python3.11/site-packages/locationsharinglib/locationsharinglib.py", line 121, in _get_authenticated_session
    session.cookies.update(pickle.load(cfile))
                           ^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'loud'
❯ pipenv graph
locationsharinglib==4.2.0
  - cachetools [required: >=5.3.0, installed: 5.3.0]
  - coloredlogs [required: >=15.0.1, installed: 15.0.1]
    - humanfriendly [required: >=9.1, installed: 10.0]
  - pytz [required: >=2022.7.1, installed: 2022.7.1]
  - requests [required: >=2.28.2, installed: 2.28.2]
    - certifi [required: >=2017.4.17, installed: 2022.12.7]
    - charset-normalizer [required: >=2,<4, installed: 3.1.0]
    - idna [required: >=2.5,<4, installed: 3.4]
    - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.15]
nose==1.3.7
costastf commented 1 year ago

Could you please also provide a snippet from your cookies file? Make sure you clear out any values from the cookies please.

GuyKh commented 1 year ago

@costastf 

cloud.google.com        FALSE   /web-risk/docs  FALSE   0       FACET_EXPERIMENT_IDS_DEVSITE    [11100011, 11100011, 11100011, 11100011, 11100011, 11100011, 11100011, 11100011, 11100011, 11100011, 11100011, 11100011, 11100011, 11100011, 11100011, 11100011, 11100011]
drive.google.com        FALSE   /drive/u/1      TRUE    1676018273      COMPASS drive=<encrypted>
mail.google.com FALSE   /mail/u/1       TRUE    0       GMAIL_AT        <encrypted>
mail.google.com FALSE   /mail/u/0       TRUE    1676806627      COMPASS gmail_ps=<encrypted>
mail.google.com FALSE   /mail/u/0       TRUE    0       GMAIL_AT        <encrypted>
mail.google.com FALSE   /sync/u/0       TRUE    1676806630      COMPASS bigtop-sync=<encrypted>
mail.google.com FALSE   /mail/u/0       TRUE    1676030252      GMAIL_IMP       <encrypted>
.google.com     TRUE    /verify TRUE    1685349304      SNID    <encrypted>
www.google.com  FALSE   /chrome FALSE   0       iid     %7B6667F302-C721-11ED-AFA1-0242AC120000%7D
drive.google.com        FALSE   /drive  TRUE    1676803253      COMPASS drive=<encrypted>
mail.google.com FALSE   /mail   TRUE    1676806630      COMPASS gmail=<encrypted>
.chrome.google.com      TRUE    /       FALSE   0       __utmc  73091649
...
costastf commented 1 year ago

Apart from the fact that you are logged in , in more than just maps, I don't see anything wrong with the cookies. I am at a bit of a loss here to be honest, not really sure what is happening. Do you mind please running it in debug mode and hopefully that will give us more info.

GuyKh commented 1 year ago

Can you help expaining how to do that?

costastf commented 1 year ago 
import logging
logging.basicConfig(level=logging.DEBUG)

before any other usage

costastf commented 1 year ago

Please be mindful of the data you paste for personal info.

GuyKh commented 1 year ago

Nothing new

>>> import logging
>>> logging.basicConfig(level=logging.DEBUG)
>>> from locationsharinglib import Service
>>> cookies_file = 'cookies.txt'
>>> google_email = 'mymail@gmail.com'
>>> service = Service(cookies_file=cookies_file, authenticating_account=google_email)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/guyk/git/tmp/.venv/lib/python3.11/site-packages/locationsharinglib/locationsharinglib.py", line 102, in __init__
    self._session = self._validate_cookie(cookies_file or '')
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/guyk/git/tmp/.venv/lib/python3.11/site-packages/locationsharinglib/locationsharinglib.py", line 105, in _validate_cookie
    session = self._get_authenticated_session(cookies_file)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/guyk/git/tmp/.venv/lib/python3.11/site-packages/locationsharinglib/locationsharinglib.py", line 121, in _get_authenticated_session
    session.cookies.update(pickle.load(cfile))
GuyKh commented 1 year ago

You also can see, that the stacktrace describes the first line of each method, no LOGGER.debug(...) before that

GuyKh commented 1 year ago

FML...

Cloned the repo. Opened pipenv shell, installing the relevant python3 version

python3
>>> import pickle
>>> with open("cookies.txt", 'rb') as cfile:
...     file = pickle.load(cfile)
...     print(file)
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
ModuleNotFoundError: No module named 'loud'

What the flying ***k?

GuyKh commented 1 year ago

OK. I got an idea.

>>> import pickle
>>> f = open("README.rst", "rb")
>>> pickle.load(f)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
_pickle.UnpicklingError: invalid load key, '='.

Meaning - there's something about the "cookies.txt" file that screws us over.

GuyKh commented 1 year ago

Deeper "investigation" lead me to believe that the problem is with pickle.load Removing the first line from cookies.txt leads me to this:

>>> pickle.load(f)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
_pickle.UnpicklingError: could not find MARK
GuyKh commented 1 year ago

Any chance to use something other than pickle? For some reason it doesn't like this format or something

GuyKh commented 1 year ago

How do you feel about this approach:

https://scrapfly.io/blog/save-and-load-cookies-in-requests-python/

session = requests.session()
cookies = json.loads(Path("cookies.json").read_text())  # save them to file as JSON
cookies = requests.utils.cookiejar_from_dict(cookies)  # turn dict to cookiejar
session.cookies.update(cookies)  # load cookiejar to current session
costastf commented 1 year ago

Can you please give https://github.com/costastf/locationsharinglib/tree/bug-pickle-load-fails a try?

GuyKh commented 1 year ago 
>>> service = Service(cookies_file=cookies_file, authenticating_account=google_email)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/guyk/git/locationsharinglib/locationsharinglib/locationsharinglib.py", line 102, in __init__
    self._session = self._validate_cookie(cookies_file or '')
  File "/home/guyk/git/locationsharinglib/locationsharinglib/locationsharinglib.py", line 105, in _validate_cookie
    session = self._get_authenticated_session(cookies_file)
  File "/home/guyk/git/locationsharinglib/locationsharinglib/locationsharinglib.py", line 121, in _get_authenticated_session
    session.cookies.update(pickle.load(cfile))
TypeError: a bytes-like object is required, not 'str'
costastf commented 1 year ago

🤦 Can you please try https://github.com/costastf/locationsharinglib/tree/feature-remove-pickle-loading

GuyKh commented 1 year ago

There is some progress, but I'm getting:

ERROR:locationsharinglib.Service:__init__() takes from 7 to 11 positional arguments but 24 were given
ERROR:locationsharinglib.Service:Things broke...
Traceback (most recent call last):
  File "/home/guyk/git/locationsharinglib/locationsharinglib/locationsharinglib.py", line 127, in _get_session_from_cookie_file
    cookies = [Cookie(*line.strip().split()) for line in text.splitlines()
  File "/home/guyk/git/locationsharinglib/locationsharinglib/locationsharinglib.py", line 127, in <listcomp>
    cookies = [Cookie(*line.strip().split()) for line in text.splitlines()
TypeError: __init__() takes from 7 to 11 positional arguments but 24 were given
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/guyk/git/locationsharinglib/locationsharinglib/locationsharinglib.py", line 100, in __init__
    self._session = self._validate_cookie(cookies_file or '')
  File "/home/guyk/git/locationsharinglib/locationsharinglib/locationsharinglib.py", line 103, in _validate_cookie
    session = self._get_authenticated_session(cookies_file)
  File "/home/guyk/git/locationsharinglib/locationsharinglib/locationsharinglib.py", line 117, in _get_authenticated_session
    session = self._get_session_from_cookie_file(cfile)
  File "/home/guyk/git/locationsharinglib/locationsharinglib/locationsharinglib.py", line 135, in _get_session_from_cookie_file
    raise InvalidCookies(message) from None
locationsharinglib.locationsharinglibexceptions.InvalidCookies: Could not properly load cookie text file.

My little investigation:

>>> file = open(cookies_file, 'r', encoding='utf-8')
>>> text = file.read()
>>> text
'cloud.google.com\tFALSE\t/web-risk/docs\tFALSE\t0\tFACET_EXPERIMENT_IDS_DEVSITE\t....`

Perhaps it's reading \n as \\n and \t as \\t and so on

costastf commented 1 year ago 
TypeError: __init__() takes from 7 to 11 positional arguments but 24 were given

That means that there are cookies with many more than expected values. Could you please use an incognito browser and log in only to maps.google.com, export the cookies and close that browser without logging out and try with those please? If might be worth looking into a more generic approach to loading cookies like you proposed and not this targeted one I currently implement.

GuyKh commented 1 year ago

No sir. I think it's related to the fact that you do file.read() and not file.readlines() -- this would get a lot of \n in a long string

>>> read_lines[0].split("\t")
['cloud.google.com', 'FALSE', '/web-risk/docs', 'FALSE', '0', 'FACET_EXPERIMENT_IDS_DEVSITE', '[11100011, 11100011, 11100011, 11100011, 11100011, 11100011, 11100011, 11100011, 11100011, 11100011, 11100011, 11100011, 11100011, 11100011, 11100011, 11100011, 11100011]\n']

Note the \n in the end

costastf commented 1 year ago

Notice in line 127 right after the read I do a list comprehension on splitlines() which is effectively that same. What fails is the instantiation of the cookie object that is getting more values than it expects. Have you tried with only cookies from maps and nothing else?

GuyKh commented 1 year ago

Scratch that.

I think the reason is that 

lines[0].strip().split()
['cloud.google.com', 'FALSE', '/web-risk/docs', 'FALSE', '0', 'FACET_EXPERIMENT_IDS_DEVSITE', '[11100011,', '11100011,', '11100011,', '11100011,', '11100011,', '11100011,', '11100011,', '11100011,', '11100011,', '11100011,', '11100011,', '39300364,', '11100011,', '11100011,', '11100011,', '11100011,', '11100011]']

i.e. - you're not splitting by "\t" but by spaces and the array gets split into a long list of values

costastf commented 1 year ago

This cookie is not a maps cookie, could you please try with only maps cookies for now?

GuyKh commented 1 year ago

Maps cookies are with www.google.com domain.

Using it - it looks somewhat better:

DEBUG:locationsharinglib.Service:Validating access to personal account...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/guyk/git/locationsharinglib/locationsharinglib/locationsharinglib.py", line 100, in __init__
    self._session = self._validate_cookie(cookies_file or '')
  File "/home/guyk/git/locationsharinglib/locationsharinglib/locationsharinglib.py", line 111, in _validate_cookie
    raise InvalidCookies(message)
locationsharinglib.locationsharinglibexceptions.InvalidCookies: The cookies provided do not provide a valid session, could not reach personal account page.Please create another cookie file and try again.
GuyKh commented 1 year ago

Still wasn't able to make it work :(

costastf commented 1 year ago

Well, these cookies are loaded but do not provide a valid session. Can you try another extension and browser maybe?

costastf commented 1 year ago

Do you mind trying out https://chrome.google.com/webstore/detail/open-cookiestxt/gdocmgbfkjnnpapoeobnolbbkoibbcif/related?hl=en as suggested by @janwillemb please?

GuyKh commented 1 year ago

Same result. No exceptions though

>>> from locationsharinglib import Service
>>> cookies_file = 'www.google.com_cookies.txt'
>>> google_email = 'mail@gmail.com'
>>> service = Service(cookies_file=cookies_file, authenticating_account=google_email)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/guyk/git/locationsharinglib/locationsharinglib/locationsharinglib.py", line 100, in __init__
    self._session = self._validate_cookie(cookies_file or '')
  File "/home/guyk/git/locationsharinglib/locationsharinglib/locationsharinglib.py", line 111, in _validate_cookie
    raise InvalidCookies(message)
locationsharinglib.locationsharinglibexceptions.InvalidCookies: The cookies provided do not provide a valid session, could not reach personal account page.Please create another cookie file and try again.

File comes with headers:

 Netscape HTTP Cookie File
# http://curl.haxx.se/rfc/cookie_spec.html
# This is a generated file! Do not edit.

# Domain        Subdomain       Path    Secure  Expire  Name    Value
.chrome.google.com      TRUE    /       FALSE   0       __utmc  73091649
....
GuyKh commented 1 year ago

Long story short:

  1. Cookies should be imported for .google.com domains, No module named 'loud' happend because of using cloud.google.com domain cookie.
  2. Due to some unknown reason, the current logic doesn't work with locale google logic - so even correct cookies are fail to work due to redirect by google. @costastf is aware and working on a fix which would implement a different verification logic.
costastf commented 1 year ago

@GuyKh can you please give https://github.com/costastf/locationsharinglib/tree/feature-remove-pickle-loading a try and let me know? I think that I have identified a very easy and nice way to identify the validity of the cookies provided. The sixth field of the response (which you get even if you are not authenticated, which is what made it tricky) would be 'GgA=' if the cookies are invalid. I have done quite a bit of testing and it checks out for me, but I would appreciate a bit more feedback. If this is ok, i will release a new major version with this new logic and the removal of the pickle loading.

GuyKh commented 1 year ago

Works great

costastf commented 1 year ago

@GuyKh can you give it another go with the explicit cookies check? It should provide more messages in case of errors.

GuyKh commented 1 year ago

Latest commit works great 👍🏻

costastf commented 1 year ago

Hi @GuyKh , v5.0.0 should be handling your case much better. Closing this for now, if you have any other problems please feel free to open it again. Thanks for reporting and for your help!

costastf commented 1 year ago

v5.0.1 also handles the foreign cookie issues nicer.