search

pydiverse / pydiverse.pipedag

A data pipeline orchestration library for rapid iterative development with automatic cache invalidation allowing users to focus writing their tasks in pandas, polars, sqlalchemy, ibis, and alike.
https://pydiversepipedag.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
15 stars 2 forks source link

Apply database default case to table names during materialization #110

Open windiana42 opened 11 months ago

windiana42 commented 11 months ago

DB2 has uppercase as default table case convention. However, SQLAlchemy has lowercase and translates it. Thus we should act like SQLAlchemy and translate all-lowercase table names to all-uppercase table names for DB2.

Reproduction example:

@materialize
def task():
  df = pd.DataFrame(dict(a=[1,2,3]))
  return dag.Table(df, "_lowercase_name")

Both the table name "_lowercase_name" should be uppercased and the column name a.

windiana42 commented 11 months ago

I am guessing that the problem is caused by the leading underscore character. Sometimes, default table case works.