arrow-rs doesn't support converting Duration arrays to/from Parquet format. To get around this, Synapse uses a hack where Duration arrays are cast to int64 before writing to disk, and when reading from disk a Cast node is inserted into the execution plan to convert back to duration.
This works for scalar arrays, but not for Duration tensors, which are represented as fixed-size lists, since the arrow-rscast method doesn't support fixed-sized lists.
This should be fixed upstream in the near future since there is significant work being done in datafusion to better support fixed-size lists:
apache/arrow-datafusion#6560
In the meantime only scalar Duration fields are supported.
arrow-rsdoesn't support converting Duration arrays to/from Parquet format. To get around this, Synapse uses a hack where Duration arrays are cast toint64before writing to disk, and when reading from disk a Cast node is inserted into the execution plan to convert back to duration.This works for scalar arrays, but not for Duration tensors, which are represented as fixed-size lists, since the
arrow-rscastmethod doesn't support fixed-sized lists.This should be fixed upstream in the near future since there is significant work being done in
datafusionto better support fixed-size lists: apache/arrow-datafusion#6560In the meantime only scalar Duration fields are supported.