owid / etl

A compute graph for loading and transforming OWID's data
https://docs.owid.io/projects/etl
MIT License
58 stars 18 forks source link

:hammer: Refactor chart-sync #2861

Closed Marigold closed 5 days ago

Marigold commented 1 week ago

This PR addresses a couple of problems. @lucasrodes if you find it unreadable, I can split it into multiple PRs.

Fix chart-leakage from data & metadata

See https://github.com/owid/etl/issues/2880. I split function modified_charts_by_admin into two - one for chart configs and the other for data & metadata updates. It feels much more intuitive now. It was just rubbish before... so I recommend to take a fresh look at the new logic (check out the docstring for more details). It's not perfect, but a step forward.

Refactor chart-sync

TODO after merging

owidbot commented 1 week ago
Quick links (staging server): Site Admin Wizard

Login: ssh owid@staging-site-refactor-chartsync

chart-diff: ❌
  • 0/1 reviewed charts
    • Modified: 0/1
    • New: 0/0
data-diff: ✅ No differences found ```diff Legend: +New ~Modified -Removed =Identical Details Hint: Run this locally with etl diff REMOTE data/ --include yourdataset --verbose --snippet ``` Automatically updated datasets matching _weekly_wildfires|excess_mortality|covid|fluid|flunet|country_profile|garden/ihme_gbd/2019/gbd_risk_ are not included

Edited: 2024-06-25 07:10:32 UTC Execution time: 3.79 seconds