googleapis / python-bigquery

Apache License 2.0
733 stars 299 forks source link

RecursionError when pickling Table object #2001

Open adinamarca opened 1 month ago

adinamarca commented 1 month ago

Hello.

Environment details

Steps to reproduce

  1. Create a valid instance of Table object.
  2. Using pickle, dumps the Table object.
  3. Using pickle, loads the Table object.

Code example

from warehouse.connectors import AccountAnalytics4Adminapi
from pickle import dumps, loads

def test_pickle():
    conn = AccountAnalytics4Adminapi("someaccount")
    conn.fetch()
    conn.transform()
    conn.load()

    try:
        pickled = dumps(conn)
        unpickled = loads(pickled)
    except Exception as e:
        assert False, f"Failed to pickle/unpickle object: {e}"

    assert conn == unpickled, "Pickled and unpickled objects are not equal."

Stack trace

Traceback (most recent call last):
  File "/Users/some_user/Desktop/proyectos/warehouse/repo/warehouse/tests/test_pickle.py", line 12, in test_pickle
    unpickled = loads(pickled)
                ^^^^^^^^^^^^^^
  File "/Users/some_user/.pyenv/versions/3.12.4/envs/warehouse/lib/python3.12/site-packages/google/cloud/bigquery/table.py", line 1478, in __getattr__
    value = self._xxx_field_to_index.get(name)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/some_user/.pyenv/versions/3.12.4/envs/warehouse/lib/python3.12/site-packages/google/cloud/bigquery/table.py", line 1478, in __getattr__
    value = self._xxx_field_to_index.get(name)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/some_user/.pyenv/versions/3.12.4/envs/warehouse/lib/python3.12/site-packages/google/cloud/bigquery/table.py", line 1478, in __getattr__
    value = self._xxx_field_to_index.get(name)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  [Previous line repeated 995 more times]
RecursionError: maximum recursion depth exceeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/some_user/Desktop/proyectos/warehouse/repo/tests.py", line 3, in <module>
    test_pickle.test_pickle()
  File "/Users/some_user/Desktop/proyectos/warehouse/repo/warehouse/tests/test_pickle.py", line 14, in test_pickle
    assert False, f"Failed to pickle/unpickle object: {e}"
           ^^^^^
AssertionError: Failed to pickle/unpickle object: maximum recursion depth exceeded

I was using Airflow, and in a task a maximum recursion depth exceeded exception was raised. Also I checked this in my personal computer.

Im not expert, but I was checking the code and I see that in the "Row" object (which probably is used by Table), there is a recursion when calling __getattr__ because it uses the get method from _xxx_field_to_index, and the get method also uses the same get method from _xxx_field_to_index, which leads to recursion (my guess).

Thanks!