search

dlt-hub / dlt

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
https://dlthub.com/docs
Apache License 2.0
2.3k stars 150 forks source link

upgrade built-in streamlit app #459

Open rudolfix opened 1 year ago

rudolfix commented 1 year ago

Background We generate a Streamlit app with dlt pipeline ... show. It needs updates to catch up with existing features.

Tasks

    • [ ] display the staging destination information
    • [ ] display the staging dataset information (if there are any merge resources)
    • [x] allow to select schemas if there's more than 1
    • [x] display more information on a resource: primary keys, merge keys and other hints

look at dlt pipeline commands and port some of the functionalities. there are nice data structures below

    • [ ] display source state for current schema
    • [ ] display resource state for particular resource
    • [ ] list extracted, normalized and loaded packages and allow to see the job, their statuses and errors inside
    • [ ] display last run trace

Bonus task: Use decompose method of the DltSource to show default execution DAG. you can use networkx to vis (make it optional dependency though)

munish7771 commented 1 year ago

can you assign this to me? @rudolfix

sultaniman commented 5 months ago

@rudolfix should we keep this open to work on the remaining points?