search

buoyant-data / oxbow

Collection of AWS Lambdas for creating and managing Delta tables
https://www.buoyantdata.com
GNU Affero General Public License v3.0
21 stars 7 forks source link

Add partition to existing delta table #21

Open quartox opened 5 months ago

quartox commented 5 months ago

Would this be a good solution to add partitions from parquet to a delta table? E.g. if you wait until a full date partition has been populated and then added dt=2024-05-07 it looks like it would find all relevant parquet files. I think I might be able to use the rust library directly but I was also wondering if you would be open to a contribution that makes this a cli option.

rtyler commented 2 months ago

@quartox technically the CLI could be extended to handle this scenario since the Lambda already handles this scenario.