we project use deltalake as a TableProvider and insert into deltalake result a lot of little files.
my code kind of like this.
let table_path = self.table_path.clone();
let data = self.data.clone();
let total_rows: usize = data.iter().map(|r| r.num_rows()).sum();
let stream = futures::stream::once(async move {
let delta_table = open_table(&table_path).await?;
let _ = DeltaOps(delta_table).write(data).await?;
info!("Inserted into {} {} rows", table_path, total_rows);
Ok(RecordBatch::new_empty(Arc::new(Schema::empty())))
})
.boxed();
how to append exists file instead create new files everytime.
we project use
deltalake
as aTableProvider
and insert into deltalake result a lot of little files.my code kind of like this.
how to append exists file instead create new files everytime.