Given that the high level API for writing data into the Delta tables currently has memory performance issues, re-use our existing optimized writing logic to perform the writes instead.
Simply upload the files, and commit the transaction for a new table version, in accordance with the Delta protocol.
Note that this implementation lacks the partitioning columns mechanism that is present in the Delta writers, but since we don't use those atm anyway it should be a good temporary solution.
Given that the high level API for writing data into the Delta tables currently has memory performance issues, re-use our existing optimized writing logic to perform the writes instead.
Simply upload the files, and commit the transaction for a new table version, in accordance with the Delta protocol.
Note that this implementation lacks the partitioning columns mechanism that is present in the Delta writers, but since we don't use those atm anyway it should be a good temporary solution.