[Feature]: Support UI to monitor the health status of Iceberg tables

projectnessie / nessie

Nessie: Transactional Catalog for Data Lakes with Git-like semantics

https://projectnessie.org

Apache License 2.0

979 stars 125 forks source link

[Feature]: Support UI to monitor the health status of Iceberg tables #9314

Open nqvuong1998 opened 1 month ago

nqvuong1998 commented 1 month ago

Description

Nessie should be able to assess the health of a table and alert users to any issues. The assessment of a table's health may involve two factors:

File Status: This includes the condition of data files and metadata files in the table. Overly fragmented files and an excessive number of delete files can indicate poor query performance.
Data Quality: The data in the table should adhere to user-defined rules.

Expected Use Cases

To monitor Iceberg workloads

Requested Changes in public API

No response

nqvuong1998 commented 1 month ago

cc @snazy @ajantha-bhat

dimas-b commented 1 month ago

Hi @nqvuong1998 : By "UI" do you mean the Nessie OSS UI or some general / custom UI?

nqvuong1998 commented 1 month ago

Hi @dimas-b , Currently, the Nessie OSS UI only supports displaying branches and commits. It would be beneficial to include table metrics such as snapshots, partitions, and files to assess the health of a table and alert users to potential issues.

dimas-b commented 1 month ago

Well, the Nessie OSS UI does not really have people who can improve it ATM :shrug: ... but contributions are certainly welcome.

For day-to-day activities, the new CLI is actually more feature-rich than the Web UI and hopefully easier to maintain.