search

pydiverse / pydiverse.pipedag

A data pipeline orchestration library for rapid iterative development with automatic cache invalidation allowing users to focus writing their tasks in pandas, polars, sqlalchemy, ibis, and alike.
https://pydiversepipedag.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
12 stars 2 forks source link

Refactor DB2 dependency in generic code. #158

Closed windiana42 closed 3 months ago

windiana42 commented 4 months ago

Polymorphism is used instead.

Checklist

windiana42 commented 4 months ago

@nicolasmueller I prefer the polymorphism solution to getting rid of the DB2 dependency in generic code

windiana42 commented 3 months ago

Currently the signature of get_create_table_suffix is inconsistent: The generic table takes table: Table while the DB2 version takes get_create_table_suffix: str | None. So either we have to adjust the generic version or fold resolve_materialization_details_label into the DB2 version.

I'd be in favor of the first version since the function is used without resolve_materialization_details_label in copy_raw_sql_tables_to_transaction.

Oops. Yes the idea to simplify usage by just providing a table object is more tricky. So I went back to the old interface. One could consider to have take Table | Stage . I would think this is nicer encapsulation. But not nice enough to refactor it now.