Closed nicolasmueller closed 8 months ago
@windiana42 Should specifying a compression method for a backend that does not support it result in an exception? Currently, we just print a warning.
@windiana42 Could you have a look at this again? Now, the materialization_details
can be configured in pipedag.yaml
. For now DB2 compression and table spaces are supported (not yet unlogged Postgres tables).
Some notes:
materialization_details
to the cached table metadata so that they are applied when a table is copied to a new schema.CREATE TABLE
statement when writing a pandas dataframe to DB2 when using materialization details, since the table space cannot easily be changed after the creation (see CreateTableWithSuffix
)I also had to add the materialization_details to the cached table metadata so that they are applied when a table is copied to a new schema.
We store a version number of our metadata in the metadata itself. Shall we bump this version number? Maybe we say we don't need it if we are backwards compatible with tables without materialization_details specified.
We store a version number of our metadata in the metadata itself. Shall we bump this version number? Maybe we say we don't need it if we are backwards compatible with tables without materialization_details specified.
It's probably better style to bump it, so I'll bump it. For DuckDB it seems necessary anyway to refresh the cache after this change.
Checklist
docs/source/changelog.md
entry