Open emanueledomingo opened 5 days ago
I am additionally have the same issue when writing new partitions with type timestamp in 0.18.1
, when time stamps are structured as for example: '2015-10-30T06:40:15.000Z', year-month-dayThour-min-month.000Z
We fixed a longstanding bug where timestamps where incorrect, this has now been correct and was a backwards incomatible change in some areas, additionally the pyarrow engine however incorrectly writes UTC timestamps as Z, this is something we cannot configure in pyarrow
Is there a way to migrate the schema from "timestamp" to "timestamp_ntz" without recreating the table? (and load all the historical data)
I tried with schema_mode: overwrite
but i get the same error. It seems that delta is unable to write the new "timestamp_ntz" type over the legacy "timestamp".
@emanueledomingo easiest is to recreate the table at the moment
Environment
Delta-rs version:
Binding: 0.18.0
Environment:
Bug
What happened:
I have a table written with
delta-rs 0.10.0
. The schema is:I'm triyng to write a new partition on that table with the following schema:
But i get:
DeltaError: Generic DeltaTable error: Writer features must be specified for writerversion >= 7, please specify: TimestampWithoutTimezone
.With deltalake 0.16.2 worked fine. Now i dumped to 0.18.0 and i get this error with tables created with an old delta-rs client.
If i the table is created with newer delta-rs client, this doesn't happen.
How to reproduce it:
deltalake==0.10.0
dl.version # 0.10.0
ta = pa.Table.from_pydict( { "Date": ["2023-01-01", "2023-01-02"], "Timestamp": ["2023-01-01T14:37:35.386235", "2023-01-01T14:37:35.386235", "2023-01-01T14:37:35.386235"] } )
ta = ta.cast( pa.schema( [ ("Date", pa.date32(), False), ("Timestamp", pa.timestamp("us"), False), ] ) ) dl.write_deltalake( table_or_uri="tmp/table", mode="overwrite", data=ta, )
More details: