search

gauteh / lieer

Fast email-fetching and sending and two-way tag synchronization between notmuch and GMail
http://lieer.gaute.vetsj.com
Other
494 stars 60 forks source link

remote: oauth2: migrate from deprecated oauth2client library #242

Closed granquet closed 8 months ago

granquet commented 11 months ago

oauth2client lib is deprecated and hasn't been updated in a while.

replace it with google-auth.

granquet commented 11 months ago

I'm not a python developer... but I've tried :)

the "key storage" looked a bit smart with the "oauth2client.file" library, now it's just a simple wrapper that writes the credentials in json format to the disk. not sure if there's anything better to do? opinions?

Also, the "argparser" from the "oauth2client.tools" doesn't seem to have an equivalent... opinions? :)

Hope this helps, Guillaume.

PS: on Gentoo, the oauth2client python package has been removed... which triggered the need to provide the patch. I expect other distros will remove it also.

gauteh commented 11 months ago

Nice, have you tested this throughly? I don't have a good setup for that at the moment.

granquet commented 11 months ago

Nice, have you tested this throughly? I don't have a good setup for that at the moment.

I should have given that precision in the previous message. I have tested these use cases:

I'm struggling to test out the refresh case, can't seem to generate short lived credentials? I haven't looked yet at testing out the case with a provided CLIENT_SECRET_FILE.

I also haven't tried to re-use the previously created credential file by oauth2client, json looks different, so unless the new lib implements backward compatibility, user will probably have to delete their old credential file and re-request and authorization.

gauteh commented 11 months ago

That would be nice to test, or if possible, manually migrate to a new file (use a different file name so that users can roll-back).

yeled commented 11 months ago

I've massaged this a bit locally.. The first issue with this PR is that I get timeouts on my large sync. Is there a way to increase it do you think?

receiving metadata:  31%|███████████████████████████████████████▉                                                                                       | 222482/707725 [1:39:00<1:28:59, 90.88it/s]Traceback (most recent call last):
  File "/home/yeled/src/gmailieer/env3/bin/gmi", line 4, in <module>
    __import__('pkg_resources').run_script('lieer==1.5', 'gmi')
  File "/home/yeled/src/gmailieer/env3/lib/python3.9/site-packages/pkg_resources/__init__.py", line 666, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/home/yeled/src/gmailieer/env3/lib/python3.9/site-packages/pkg_resources/__init__.py", line 1469, in run_script
    exec(script_code, namespace, namespace)
  File "/home/yeled/src/gmailieer/env3/lib/python3.9/site-packages/lieer-1.5-py3.9.egg/EGG-INFO/scripts/gmi", line 24, in <module>
  File "/home/yeled/src/gmailieer/env3/lib/python3.9/site-packages/lieer-1.5-py3.9.egg/lieer/gmailieer.py", line 229, in main
  File "/home/yeled/src/gmailieer/env3/lib/python3.9/site-packages/lieer-1.5-py3.9.egg/lieer/gmailieer.py", line 309, in sync
  File "/home/yeled/src/gmailieer/env3/lib/python3.9/site-packages/lieer-1.5-py3.9.egg/lieer/gmailieer.py", line 351, in push
  File "/home/yeled/src/gmailieer/env3/lib/python3.9/site-packages/lieer-1.5-py3.9.egg/lieer/remote.py", line 133, in func_wrap
  File "/home/yeled/src/gmailieer/env3/lib/python3.9/site-packages/lieer-1.5-py3.9.egg/lieer/remote.py", line 334, in get_messages
  File "/home/yeled/src/gmailieer/env3/lib/python3.9/site-packages/google_api_python_client-2.89.0-py3.9.egg/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/home/yeled/src/gmailieer/env3/lib/python3.9/site-packages/google_api_python_client-2.89.0-py3.9.egg/googleapiclient/http.py", line 1566, in execute
    self._execute(http, self._order, self._requests)
  File "/home/yeled/src/gmailieer/env3/lib/python3.9/site-packages/google_api_python_client-2.89.0-py3.9.egg/googleapiclient/http.py", line 1496, in _execute
    resp, content = http.request(
  File "/home/yeled/src/gmailieer/env3/lib/python3.9/site-packages/google_auth_httplib2-0.1.0-py3.9.egg/google_auth_httplib2.py", line 218, in request
    response, content = self.http.request(
  File "/home/yeled/src/gmailieer/env3/lib/python3.9/site-packages/httplib2-0.22.0-py3.9.egg/httplib2/__init__.py", line 1724, in request
    (response, content) = self._request(
  File "/home/yeled/src/gmailieer/env3/lib/python3.9/site-packages/httplib2-0.22.0-py3.9.egg/httplib2/__init__.py", line 1444, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/home/yeled/src/gmailieer/env3/lib/python3.9/site-packages/httplib2-0.22.0-py3.9.egg/httplib2/__init__.py", line 1396, in _conn_request
    response = conn.getresponse()
  File "/usr/lib/python3.9/http/client.py", line 1347, in getresponse
    response.begin()
  File "/usr/lib/python3.9/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.9/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.9/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.9/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out
granquet commented 10 months ago

Sorry I'm not very reactive... I have spent some time trying to understand how to manage tokens on the google API... I've yet to understand how to get a token to expire to test the refresh token usecase. Anyone able to direct me towards some documentation how I can request a new token and make it expire so that I can test the refresh method?

I've massaged this a bit locally.. The first issue with this PR is that I get timeouts on my large sync. Is there a way to increase it do you think?

I haven't been able to reproduce this, my inbox is roughly 10GB big and my internet connection is far from what you would call "fast". Not sure here what exactly times out from the log?


(lieer-venv) testlieer@seychelles ~/mail/ranquet.guillaume@gmail.com $ gmi sync
push: everything is up-to-date.
pull: partial synchronization.. (hid: 3993042)
pull: historyId is too old, full sync required.
fetching messages: 33070it [01:08, 485.95it/s]                                                                                                                                                
removing deleted: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  6.37it/s]
receiving content: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 543/543 [00:39<00:00, 13.65it/s]
receiving metadata:  11%|██████████████▎                                                                                                                 | 3651/32527 [00:44<05:41, 84.46it/s]remote: user rate error, increasing delay to 1
remote: waiting 1.0 seconds..
[...]
waiting 1.0 seconds..
remote: decreasing delay to 0
receiving metadata: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 32527/32527 [14:35<00:00, 37.16it/s]
pull: complete, removing resume file
current historyId: 4040625, current revision: 118781
(lieer-venv) testlieer@seychelles ~/mail/ranquet.guillaume@gmail.com $
granquet commented 10 months ago

I have been able to test further, it has been a pain to navigate the google cloud web interface... anyway...

but what's tested (and works):

I haven't tested the migration from the previous json file, as I've deleted them from my system and the oauth2client python library is no more part of my system (I could reinstall using pip/venv though)

jaeyeom commented 10 months ago

At least, this solved my problem and I could successfully gmi init with granquet's repo. Thank you so much!

skangas commented 8 months ago

I've used this to do a full pull from scratch, including the whole authentication flow.

I've also tried syncing and sending emails. So far, so good.

gauteh commented 8 months ago

Sounds good, looks like this can probably be merged soon since the current solution is not really working anymore. Might require reauthorization, but that is not the end of the world.

skangas commented 8 months ago

The list of requirements in README.md should probably be updated too.

gauteh commented 8 months ago

Thanks. Finally got around to merging this.