search

n8henrie / pycookiecheat

Borrow cookies from your browser's authenticated session for use in Python scripts.
https://n8henrie.com/2013/11/use-chromes-cookies-for-easier-downloading-with-python-requests/
MIT License
744 stars 111 forks source link

Can't decrypt 'v11' cookies (v10 cookies are fine) #64

Closed Saroumane closed 6 months ago

Saroumane commented 6 months ago

My Issue

According to my tests, I can decrypt 'v10' cookies, but not 'v11'

For v11 cookies, I have the classic error : 

 File "/usr/lib/python3.12/site-packages/pycookiecheat/chrome.py", line 328, in chrome_cookies
    row["value"] = chrome_decrypt(
                   ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pycookiecheat/chrome.py", line 78, in chrome_decrypt
    return clean(decrypted)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pycookiecheat/chrome.py", line 50, in clean
    return decrypted[:-last].decode("utf8")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 2: invalid start byte

WHYT

I did many tests to reproduce the problem with cookies from different websites.

n8henrie commented 6 months ago

Have you looked through this thread by chance? https://github.com/n8henrie/pycookiecheat/issues/12

v11 cookies have been working.

Are you using something like the gnome secret store? What is your DE?

Saroumane commented 6 months ago

Thanks for the quick reply ! Sorry I copy pasted the wrong log, here is the right one : it's an invalid continuation byte, not an 'invalid start byte'

File "/usr/lib/python3.12/site-packages/pycookiecheat/chrome.py", line 331, in chrome_cookies
    row["value"] = chrome_decrypt(
                   ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pycookiecheat/chrome.py", line 78, in chrome_decrypt
    return clean(decrypted)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pycookiecheat/chrome.py", line 50, in clean
    return decrypted[:-last].decode("utf8")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf2 in position 0: invalid continuation byte

Edit : 2nd mistake, for v10 cookies I was calling chrome_cookies() without defining the browser type, and my 'cookie_file' value was overwritten with an invalid path. I don't understand what cookie value I was getting, but probably not the one from my Cookies file ! I have corrected that, and now I see I cannot decrypt v10 cookies. (I also have an invalid continuation byte)

After re-reading the code, I get it now : cookie_file as a parameter has priority over the cookie_file defined by the browser type. So decrypting v10 cookies is working...

In case it could help, I printed the config right before entering the "ciphering" part of chrome.py ( kdf = PBKDF2HMAC( ...) : 

{'cookie_file': '~/.config/chromium/Default/Cookies',
 'init_vector': b'                ',
 'iterations': 1,
 'length': 16,
 'my_pass': b'xxxxxxxx',
 'salt': b'saltysalt'}

I redacted my_pass, but I checked the Gnome keyring, and my_pass matches my Chromium Safe Storage password. The cookie_file is also OK.

If I understand https://github.com/n8henrie/pycookiecheat/issues/12 correctly, I should replace the 'saltysalt' value by 'salt', right ? It's 'saltysalt' for v10 and 'salt' for v11 ? => replying to myself, 'salt' instead of 'saltysalt' makes no differences. (And I misread this issue)

Further checks inspired from other issues :

$ python3 -c 'import sys; print(sys.getdefaultencoding())'
utf-8

$ cp Cookies{,.bak}
$ sqlite3 Cookies.bak 'PRAGMA encoding;'
UTF-8

So I have no idea of what to do now.

n8henrie commented 6 months ago

If I understand https://github.com/n8henrie/pycookiecheat/issues/12 correctly, I should replace the 'saltysalt' value by 'salt', right ? It's 'saltysalt' for v10 and 'salt' for v11 ?

I don't think so -- you might be looking at salt=salt which is setting the parameter salt to the value of the variable named salt.

Saroumane commented 6 months ago

For the record : I was editing my previous comment while you were answering : _After re-reading the code, I get it now : cookie_file as a parameter has priority over the cookiefile defined by the browser type. So decrypting v10 cookies is working...

n8henrie commented 6 months ago

To clarify after your last post, is your issue resolved? Or are you still having issues with the v11 cookies?

The UTF8 errors usually mean that the encryption key (password) is incorrect.

Saroumane commented 6 months ago

I still have the issue with v11 cookies but I gave up, I use now the rookie python/javascript module.

n8henrie commented 6 months ago

No worries.

I'm having a hard time reproducing because my Chromium (123, archlinux) is only producing v10 cookies. It sounds like yours is producing both v10 and v11 -- do you know if it is specific websites that are making the v11 ones?

Saroumane commented 6 months ago

Out of 17 cookies I have for github.com , 3 are 'v10-encrypted' ' (their data begin with 76h 31h 30h), the 14 remaining are 'v11-encrypted' (their data begin with 76h 31h 31h) I checked these values by opening ~/.config/chromium/Default/Cookies file with sqlitebrowser

Idea : maybe you did not installed a gnome or kde keyring, and that's why your chromium has no access to a 'keyring key' (that is used for v11 encrypting) ?

n8henrie commented 6 months ago

Thank you. Only showing v10 for GitHub.com -- must be the absence of a gnome / kde keyring (I use i3 as my DE). I'll tinker around some more and see if I can reproduce.

n8henrie commented 6 months ago

It took me a while to set up an environment to try to replicate, but I finally have something workable.

I did discover an issue related to enums not giving me the string that I expected when coerced to a string (there's a fair change this is due to a homebrew StrEnum and may not be an issue once my MSPV is >= 3.11).

That issue should be resolved with 14001b8910d2236c29f2772b4b0511169a5287a5

However, that was giving me a totally different error, so I'm not sure whether or not it would help with your issue here.

Otherwise, for my future reference, my process to get gnome keyring working on my Arch machine (on i3) was:

With all that done, I can verify that I have v11 keys, and that pycookiecheat seems to find them without issue, as long as it's installed in a way to give it access to the system gobject library. Specifically, my first python in my path is a nix-installed python, so I was very confused for a while why it wasn't getting access to e.g. gi.repository; installing with /usr/bin/python -m venv was the obvious workaround once I realized the issue.

$ /usr/bin/python -m venv --system-site-packages .venv
$ source .venv/bin/activate
$ python -m pip install pycookiecheat
$ python -m pycookiecheat --url https://github.com --browser chromium  | jq -r keys[]
__Host-user_session_same_site
_device_id
_octo
disabled_global_site_banners
dotcom_user
has_recent_activity
logged_in
saved_user_sessions
user_session

I can confirm that user_session is a v11 cookie with sqlite:

$ sqlite3 -readonly ~/.config/chromium/Default/Cookies \
    "SELECT name,host_key FROM cookies WHERE host_key LIKE 'github.com' AND encrypted_value LIKE 'v10%';"
name                          host_key
----------------------------  ----------
disabled_global_site_banners  github.com
_device_id                    github.com
$
$ sqlite3 -readonly ~/.config/chromium/Default/Cookies \
    "SELECT name,host_key FROM cookies WHERE host_key LIKE 'github.com' AND encrypted_value LIKE 'v11%';"
name                           host_key
-----------------------------  ----------
__Host-user_session_same_site  github.com
has_recent_activity            github.com
saved_user_sessions            github.com
user_session                   github.com

I'm happy to see if we can sort out your issue, but since it sounds like you've moved on to another project, I'm going to close this as "cannot reproduce" -- thank you for your time and input!

Saroumane commented 6 months ago

Thanks for taking the time to analyse this

mgedmin commented 5 months ago

Hi! Different user here, seeing the same issue once I manually patch the source so it can find my Chromium cookies file (#65).

For me the cause seem to be a failure to access they key material from libsecret. If I add a debug print at the end of get_linux_config(), I can see that key_material is None (would be nice to print an error message and abort at this point, instead of emitting the UTF-8 decode error).

Also, -vvv shows a

INFO:pycookiecheat.chrome:Was not able to import Secret from gi.repository

which is strange, given that the gir1.2-secret-1 package is installed and /usr/lib/x86_64-linux-gnu/girepository-1.0/Secret-1.typelib exists, and that this succeeds from a REPL:

$ python3
Python 3.12.3 (main, Apr 10 2024, 05:33:47) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import gi
>>> gi.require_version('Secret', '1')
>>> from gi.repository import Secret

I wonder if the virtualenv that pipx set up for me is missing some pacakges, like pygobject? Yup, that was it

pipx inject pycookiecheat pygobject fixed everything and now I can see the cookies!

n8henrie commented 5 months ago

I can see that key_material is None (would be nice to print an error message and abort at this point, instead of emitting the UTF-8 decode error)

Unfortunately this would be problematic because when key_material is None, that is the trigger to fall back to the default of peanuts (https://github.com/n8henrie/pycookiecheat/blob/cbfe5a641cded4df68732c77fcd0e93d5355a303/src/pycookiecheat/chrome.py#L173), which is an entirely normal and expected state of affairs (I would not want to emit an annoying message by default here).