delta-io / delta-kernel-rs

A native Delta implementation for integration with any query engine
Apache License 2.0
144 stars 41 forks source link

write-path generate_logical_to_physical always removes partition columns #462

Open zachschuermann opened 1 week ago

zachschuermann commented 1 week ago

See comment:

    // Generate the logical-to-physical transform expression which must be evaluated on every data
    // chunk before writing. At the moment, this is a transaction-wide expression.
    fn generate_logical_to_physical(&self) -> Expression {
        // for now, we just pass through all the columns except partition columns.
        // note this is _incorrect_ if table config deems we need partition columns.
        ...
    }

This should actually be based on table properties instead of always removing