search

pydiverse / pydiverse.pipedag

A data pipeline orchestration library for rapid iterative development with automatic cache invalidation allowing users to focus writing their tasks in pandas, polars, sqlalchemy, ibis, and alike.
https://pydiversepipedag.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
19 stars 3 forks source link

Support nullable and non_nullable parameters in pipedag.Table() and refactor ddl/table-store/hook logic #160

Closed windiana42 closed 6 months ago

windiana42 commented 6 months ago

The initial idea of this PR is to add a nullable and non_nullable parameter to pipedag.Table(). It should allow specifying nullability both positive and negative. For most dialects it is beneficial to create an empty table, then set nullability, and finally fill the table.

This PR turned into a somewhat bigger refactoring. DB2 had some create empty table and then fill it logic in DDL code, but if it is not a single dialect, I prefer such complex control in the hook and table store code. Also the generic/specific split of how derived classes can customize the generic algorithm was redesigned.

Checklist

windiana42 commented 6 months ago

@NMAC427 FYI: a bigger refactoring of moving logic between DDL, table store, and hook is included in this PR

windiana42 commented 6 months ago

@NMAC427 can you please confirm that the requested changes are completed?