Breaking change when upgraded from 2.23.3 to 2.27.0

dzejeu commented 5 months ago

Error description

We use luigi in our project and after recent update of our python dependencies we started to receive following errors.

/usr/local/lib/python3.10/site-packages/luigi/worker.py  701 Luigi unexpected framework error while scheduling MyTask()
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/luigi/worker.py", line 796, in add
    current = queue.get()
  File "<string>", line 2, in get
  File "/usr/local/lib/python3.10/multiprocessing/managers.py", line 833, in _callmethod
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError:
---------------------------------------------------------------------------
Unserializable message: Traceback (most recent call last):
  File "/usr/local/lib/python3.10/multiprocessing/managers.py", line 308, in serve_client
    send(msg)
  File "/usr/local/lib/python3.10/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/local/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: cannot pickle '_thread.lock' object

Workaround

I found out that dependency that caused it was google-auth. Bringing it down from 2.27.0 to 2.23.2 while keeping other dependencies updated solved the issue. It looks like this PR may be related to this behaviour.

Background

We use luigi with parallel_scheduling=true option which means multiprocessing.Queue is used underneath and it requires every object in it to be pickle-able.

Environment details

OS: Debian 12 / Ubuntu 20.04 (same issue on both)
Python version: 3.10
pip version: 23.3.1
google-auth version: 2.27.0
luigi version: 3.5.0

clundin25 commented 5 months ago

Hi @dzejeu,

Can you share a repro?

I just tested on the latest commit inmain, and I can pickle this code, so my repro is missing some details.

❯ git rev-parse HEAD && python -c 'import pickle; from google.auth._refresh_worker import RefreshThreadManager; q = pickle.dumps(RefreshThreadManager()); print(q); print(pickle.loads(q))'
30743b0937fe0c45c77c824514a0c27539f5745b
b'\x80\x04\x95T\x00\x00\x00\x00\x00\x00\x00\x8c\x1bgoogle.auth._refresh_worker\x94\x8c\x14RefreshThreadManager\x94\x93\x94)\x81\x94}\x94(\x8c\x07_worker\x94N\x8c\x05_lock\x94Nub.'
<google.auth._refresh_worker.RefreshThreadManager object at 0x7ff1321f6610>

dzejeu commented 5 months ago

Hey @clundin25! I can't really tell how to reproduce because logs I attached is all I got from luigi - I have no idea what is being pickled underneath and digging into that would require significant amount of time (since it's not my code that is failing - it's luigi's internals).

However I applied your fix from PR on my failing env with 2.27.0 installed and it solved the issue 🙂

clundin25 commented 5 months ago

@dzejeu Thank you for pulling that PR in to verify if it fixed the issue!

clundin25 commented 4 months ago

v2.28.1 will publish soon containing the fix from the PR. Please re-open if you have further issues!

dzejeu commented 4 months ago

Thanks for handling it so quick!

googleapis / google-auth-library-python