Mause / duckdb_engine

SQLAlchemy driver for DuckDB
MIT License
326 stars 35 forks source link

[Bug]: Long column names are truncated #910

Open cpcloud opened 6 months ago

cpcloud commented 6 months ago

What happened?

Observed in https://github.com/ibis-project/ibis/issues/8393.

Long column names appear to be truncated when using duckdb_engine to fetch results.

Reproducible example:

import sqlalchemy as sa

long_name = "a_very_very_very_very_very_very_very_very_very_long_column_name"

eng = sa.create_engine("duckdb:///test.db")

with eng.begin() as con:
    con.exec_driver_sql(f"CREATE OR REPLACE TABLE t ({long_name} INT)")

t = sa.Table("t", sa.MetaData(), autoload_with=eng)
t1 = sa.select(t.c[0]).select_from(t).subquery()
query = sa.select(t1.c[long_name])

with eng.begin() as con:
    result = con.execute(query)
    batch = result.connection.connection.fetch_record_batch()

assert batch.schema.names[0] == long_name, batch.schema.names[0]

DuckDB Engine Version

0.10.0

DuckDB Version

0.9.2, 0.10.0

SQLAlchemy Version

1.4.51

Relevant log output

No response

Code of Conduct

Mause commented 6 months ago

Looks like this is a default inherited from Postgres: https://github.com/sqlalchemy/sqlalchemy/blob/1c58fe53b6fd069cbb82955ddaf9eb5405076146/lib/sqlalchemy/dialects/postgresql/base.py#L2941

cpcloud commented 6 months ago

I wonder if this is another nail in the coffin for subclassing from that dialect.

In the next Ibis release we're moving away from SQLAlchemy entirely to avoid the extra layer and myriad dependencies (which Ibis already has plenty of!).

duckdb_engine has been great to work with!

Mause commented 6 months ago

Given that it's in the "base" Postgres dialect, rather than the psycopg2 one, I'm not sure it is

I saw in the linked issue, congrats!