pydiverse / pydiverse.pipedag

A data pipeline orchestration library for rapid iterative development with automatic cache invalidation allowing users to focus writing their tasks in pandas, polars, sqlalchemy, ibis, and alike.
https://pydiversepipedag.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
19 stars 3 forks source link

Watch duckdb 0.10.1 and sqlalchemy 2.0.29 compatibility issues #183

Open windiana42 opened 5 months ago

windiana42 commented 5 months ago

Currently, I have strange errors with python-duckdb 0.10.1:

...
  File "/envs/test_duckdb/lib/python3.11/site-packages/pydiverse/pipedag/backend/table/sql/sql.py", line 491, in reflect_table
    tbl = sa.Table(
          ^^^^^^^^^
  File "<string>", line 2, in __new__
  File "/envs/test_duckdb/lib/python3.11/site-packages/sqlalchemy/util/deprecations.py", line 281, in warned
    return fn(*args, **kwargs)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^
...
  File "/envs/test_duckdb/lib/python3.11/site-packages/duckdb_engine/__init__.py", line 294, in do_rollback
    super().do_rollback(connection)
  File "/envs/test_duckdb/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 692, in do_rollback
    dbapi_connection.rollback()
sqlalchemy.exc.ProgrammingError: (duckdb.duckdb.InvalidInputException) Invalid Input Error: Attempting to execute an unsuccessful or closed pending query result
Error: FATAL Error: Failed: database has been invalidated because of a previous fatal error. The database must be restarted prior to being used again.

And with sqlalchemy 2.0.29:

...
  File "/envs/test_duckdb/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 919, in do_execute
    cursor.execute(statement, parameters)
  File "/envs/test_duckdb/lib/python3.11/site-packages/duckdb_engine/__init__.py", line 162, in execute
    self.__c.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (duckdb.duckdb.CatalogException) Catalog Error: Table with name pg_collation does not exist!
Did you mean "pg_class"?

Downgrading to python-duckdb 0.10.0 and sqlalchemy 2.0.28 works. We need to see whether problems resolve with newer versions or whether changes need to be made on pipedag side.

windiana42 commented 5 months ago

python-duckdb 0.10.2 is out and solves one of the two problems.