delta-io / delta-rs

A native Rust library for Delta Lake, with bindings into Python

https://delta-io.github.io/delta-rs/

Apache License 2.0

2.07k stars 367 forks source link

collect metrics in all operations #1152

Open roeap opened 1 year ago

roeap commented 1 year ago

Description

Some operations e.g. vacuum and optimize already collect metrics that are added to the commit history. Other operations do not yet collect these metrics. We should collect the documented metrics on all operations according to the reference.

Use Case

Related Issue(s)

rtyler commented 7 months ago

I believe this may be covered by #2032 which @ion-elgreco has proposed

ion-elgreco commented 7 months ago

Actually this one is partially covered by this one: https://github.com/delta-io/delta-rs/pull/2009

I will cross check which ones we still don't track metrics where we should though!

ion-elgreco commented 7 months ago

Current state:

Write: doesn't collect any metrics
Update: rewrite_ms is missing
FSCK: we give the removed_file_paths, but spark-delta uses num_removed_files @rtyler shall we change this to simply the count of the vec here?
Convert: we don't write the correct metrics here since we create operation underneath
Optimize: we collect metrics but totally different ones

The ones I didn't mentioned we collect the same metrics for.