delta-io / delta-rs

A native Rust library for Delta Lake, with bindings into Python

https://delta-io.github.io/delta-rs/

Apache License 2.0

1.98k stars 364 forks source link

Show progress bar during delta operations #1989

Open ion-elgreco opened 6 months ago

ion-elgreco commented 6 months ago

Description

Use Case It woud we be nice if can follow the progress of the delta operations, currently there is no feedback while writing or doing any kind of other operation.

Related Issue(s)

rtyler commented 6 months ago

@ion-elgreco I assume this is meant from the Python side of things? The way I have seen this typically handled is that there is the "normal" API and then a "progress API" which has a callback mechanism that can be optionally invoked by the library so the user can have feedback during longer operations.

Do you have an API suggestion in mind to propose?

ion-elgreco commented 6 months ago

I think it could also make sense for Rust users. We probably need to build it in rust and then expose a callback mechanism to Python.

I don't have any api design in mind yet, the idea just popped in my head.

We probably need to check at what level of granularity we want to trace things.

roeap commented 6 months ago

the related issue https://github.com/delta-io/delta-rs/issues/1048 mentions some very rough ideas on how to do this.