A data pipeline orchestration library for rapid iterative development with automatic cache invalidation allowing users to focus writing their tasks in pandas, polars, sqlalchemy, ibis, and alike.
Currently, I have strange errors with python-duckdb 0.10.1:
...
File "/envs/test_duckdb/lib/python3.11/site-packages/pydiverse/pipedag/backend/table/sql/sql.py", line 491, in reflect_table
tbl = sa.Table(
^^^^^^^^^
File "<string>", line 2, in __new__
File "/envs/test_duckdb/lib/python3.11/site-packages/sqlalchemy/util/deprecations.py", line 281, in warned
return fn(*args, **kwargs) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^
...
File "/envs/test_duckdb/lib/python3.11/site-packages/duckdb_engine/__init__.py", line 294, in do_rollback
super().do_rollback(connection)
File "/envs/test_duckdb/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 692, in do_rollback
dbapi_connection.rollback()
sqlalchemy.exc.ProgrammingError: (duckdb.duckdb.InvalidInputException) Invalid Input Error: Attempting to execute an unsuccessful or closed pending query result
Error: FATAL Error: Failed: database has been invalidated because of a previous fatal error. The database must be restarted prior to being used again.
And with sqlalchemy 2.0.29:
...
File "/envs/test_duckdb/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 919, in do_execute
cursor.execute(statement, parameters)
File "/envs/test_duckdb/lib/python3.11/site-packages/duckdb_engine/__init__.py", line 162, in execute
self.__c.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (duckdb.duckdb.CatalogException) Catalog Error: Table with name pg_collation does not exist!
Did you mean "pg_class"?
Downgrading to python-duckdb 0.10.0 and sqlalchemy 2.0.28 works. We need to see whether problems resolve with newer versions or whether changes need to be made on pipedag side.
Currently, I have strange errors with python-duckdb 0.10.1:
And with sqlalchemy 2.0.29:
Downgrading to python-duckdb 0.10.0 and sqlalchemy 2.0.28 works. We need to see whether problems resolve with newer versions or whether changes need to be made on pipedag side.