search

owid / etl

A compute graph for loading and transforming OWID's data
https://docs.owid.io/projects/etl
MIT License
78 stars 21 forks source link

:hammer: Remove --step-update and make committing explicit #3205

Closed Marigold closed 1 month ago

Marigold commented 1 month ago

Revert etl pr ... --step-update and update ETL docs. User has to manually commit generated files now and insert the diff link in the description.

owidbot commented 1 month ago
Quick links (staging server): Site Admin Wizard

Login: ssh owid@staging-site-revert-step-update

chart-diff: ✅ No charts for review.
data-diff: ❌ Found differences ```diff = Dataset garden/who/latest/monkeypox = Table monkeypox ~ Dim country - - Removed values: 43 / 85208 (0.05%) date country 2024-07-11 Cote d'Ivoire 2024-07-17 Cote d'Ivoire 2024-08-03 Cote d'Ivoire 2024-08-06 Cote d'Ivoire 2024-08-12 Cote d'Ivoire ~ Dim date - - Removed values: 43 / 85208 (0.05%) country date Cote d'Ivoire 2024-07-11 Cote d'Ivoire 2024-07-17 Cote d'Ivoire 2024-08-03 Cote d'Ivoire 2024-08-06 Cote d'Ivoire 2024-08-12 ~ Column annotation (changed data) - - Removed values: 43 / 85208 (0.05%) country date annotation Cote d'Ivoire 2024-07-11 Cote d'Ivoire 2024-07-17 Cote d'Ivoire 2024-08-03 Cote d'Ivoire 2024-08-06 Cote d'Ivoire 2024-08-12 ~ Column iso_code (changed data) - - Removed values: 43 / 85208 (0.05%) country date iso_code Cote d'Ivoire 2024-07-11 CIV Cote d'Ivoire 2024-07-17 CIV Cote d'Ivoire 2024-08-03 CIV Cote d'Ivoire 2024-08-06 CIV Cote d'Ivoire 2024-08-12 CIV ~ Column new_cases (changed data) - - Removed values: 43 / 85208 (0.05%) country date new_cases Cote d'Ivoire 2024-07-11 0 Cote d'Ivoire 2024-07-17 0 Cote d'Ivoire 2024-08-03 0 Cote d'Ivoire 2024-08-06 0 Cote d'Ivoire 2024-08-12 0 ~ Changed values: 5 / 85208 (0.01%) country date new_cases - new_cases + Africa 2024-07-07 157 156 Africa 2024-07-28 220 218 Africa 2024-08-04 97 94 Africa 2024-08-11 88 84 Africa 2024-08-18 90 72 ~ Column new_cases_per_million (changed data) - - Removed values: 43 / 85208 (0.05%) country date new_cases_per_million Cote d'Ivoire 2024-07-11 0.0 Cote d'Ivoire 2024-07-17 0.0 Cote d'Ivoire 2024-08-03 0.0 Cote d'Ivoire 2024-08-06 0.0 Cote d'Ivoire 2024-08-12 0.0 ~ Changed values: 5 / 85208 (0.01%) country date new_cases_per_million - new_cases_per_million + Africa 2024-07-07 0.105 0.104 Africa 2024-07-28 0.147 0.146 Africa 2024-08-04 0.065 0.063 Africa 2024-08-11 0.059 0.056 Africa 2024-08-18 0.060 0.048 ~ Column new_cases_smoothed (changed data) - - Removed values: 43 / 85208 (0.05%) country date new_cases_smoothed Cote d'Ivoire 2024-07-11 0.14 Cote d'Ivoire 2024-07-17 0.00 Cote d'Ivoire 2024-08-03 0.29 Cote d'Ivoire 2024-08-06 0.43 Cote d'Ivoire 2024-08-12 0.57 ~ Changed values: 35 / 85208 (0.04%) country date new_cases_smoothed - new_cases_smoothed + Africa 2024-08-06 13.86 13.43 Africa 2024-08-10 13.86 13.43 Africa 2024-08-12 12.57 12.00 Africa 2024-08-18 12.86 10.29 Africa 2024-08-19 12.86 10.29 ~ Column new_cases_smoothed_per_million (changed data) - - Removed values: 43 / 85208 (0.05%) country date new_cases_smoothed_per_million Cote d'Ivoire 2024-07-11 0.005 Cote d'Ivoire 2024-07-17 0.000 Cote d'Ivoire 2024-08-03 0.010 Cote d'Ivoire 2024-08-06 0.014 Cote d'Ivoire 2024-08-12 0.019 ~ Changed values: 7 / 85208 (0.01%) country date new_cases_smoothed_per_million - new_cases_smoothed_per_million + Africa 2024-08-18 0.009 0.007 Africa 2024-08-19 0.009 0.007 Africa 2024-08-20 0.009 0.007 Africa 2024-08-21 0.009 0.007 Africa 2024-08-24 0.009 0.007 ~ Column new_deaths (changed data) - - Removed values: 43 / 85208 (0.05%) country date new_deaths Cote d'Ivoire 2024-07-11 0 Cote d'Ivoire 2024-07-17 0 Cote d'Ivoire 2024-08-03 0 Cote d'Ivoire 2024-08-06 0 Cote d'Ivoire 2024-08-12 0 ~ Changed values: 1 / 85208 (0.00%) country date new_deaths - new_deaths + Africa 2024-08-18 1 0 ~ Column new_deaths_per_million (changed data) - - Removed values: 43 / 85208 (0.05%) country date new_deaths_per_million Cote d'Ivoire 2024-07-11 0.0 Cote d'Ivoire 2024-07-17 0.0 Cote d'Ivoire 2024-08-03 0.0 Cote d'Ivoire 2024-08-06 0.0 Cote d'Ivoire 2024-08-12 0.0 ~ Changed values: 1 / 85208 (0.00%) country date new_deaths_per_million - new_deaths_per_million + Africa 2024-08-18 0.00067 0.0 ~ Column new_deaths_smoothed (changed data) - - Removed values: 43 / 85208 (0.05%) country date new_deaths_smoothed Cote d'Ivoire 2024-07-11 0.0 Cote d'Ivoire 2024-07-17 0.0 Cote d'Ivoire 2024-08-03 0.0 Cote d'Ivoire 2024-08-06 0.0 Cote d'Ivoire 2024-08-12 0.0 ~ Changed values: 7 / 85208 (0.01%) country date new_deaths_smoothed - new_deaths_smoothed + Africa 2024-08-18 0.14 0.0 Africa 2024-08-19 0.14 0.0 Africa 2024-08-20 0.14 0.0 Africa 2024-08-21 0.14 0.0 Africa 2024-08-24 0.14 0.0 ~ Column new_deaths_smoothed_per_million (changed data) - - Removed values: 43 / 85208 (0.05%) country date new_deaths_smoothed_per_million Cote d'Ivoire 2024-07-11 0.0 Cote d'Ivoire 2024-07-17 0.0 Cote d'Ivoire 2024-08-03 0.0 Cote d'Ivoire 2024-08-06 0.0 Cote d'Ivoire 2024-08-12 0.0 ~ Changed values: 7 / 85208 (0.01%) country date new_deaths_smoothed_per_million - new_deaths_smoothed_per_million + Africa 2024-08-18 0.00009 0.0 Africa 2024-08-19 0.00009 0.0 Africa 2024-08-20 0.00009 0.0 Africa 2024-08-21 0.00009 0.0 Africa 2024-08-24 0.00009 0.0 ~ Column total_cases (changed data) - - Removed values: 43 / 85208 (0.05%) country date total_cases Cote d'Ivoire 2024-07-11 1 Cote d'Ivoire 2024-07-17 1 Cote d'Ivoire 2024-08-03 3 Cote d'Ivoire 2024-08-06 6 Cote d'Ivoire 2024-08-12 10 ~ Changed values: 50 / 85208 (0.06%) country date total_cases - total_cases + Africa 2024-07-09 5065 5064 Africa 2024-07-17 5239 5238 Africa 2024-07-18 5239 5238 Africa 2024-08-04 5787 5781 Africa 2024-08-17 5875 5865 ~ Column total_cases_per_million (changed data) - - Removed values: 43 / 85208 (0.05%) country date total_cases_per_million Cote d'Ivoire 2024-07-11 0.034 Cote d'Ivoire 2024-07-17 0.034 Cote d'Ivoire 2024-08-03 0.101 Cote d'Ivoire 2024-08-06 0.202 Cote d'Ivoire 2024-08-12 0.337 ~ Changed values: 41 / 85208 (0.05%) country date total_cases_per_million - total_cases_per_million + Africa 2024-07-16 3.501 3.500 Africa 2024-07-26 3.653 3.652 Africa 2024-08-10 3.861 3.857 Africa 2024-08-14 3.919 3.912 Africa 2024-08-20 3.977 3.959 ~ Column total_deaths (changed data) - - Removed values: 43 / 85208 (0.05%) country date total_deaths Cote d'Ivoire 2024-07-11 0 Cote d'Ivoire 2024-07-17 0 Cote d'Ivoire 2024-08-03 0 Cote d'Ivoire 2024-08-06 0 Cote d'Ivoire 2024-08-12 0 ~ Changed values: 8 / 85208 (0.01%) country date total_deaths - total_deaths + Africa 2024-08-19 49 48 Africa 2024-08-20 49 48 Africa 2024-08-21 49 48 Africa 2024-08-24 49 48 Africa 2024-08-25 49 48 ~ Column total_deaths_per_million (changed data) - - Removed values: 43 / 85208 (0.05%) country date total_deaths_per_million Cote d'Ivoire 2024-07-11 0.0 Cote d'Ivoire 2024-07-17 0.0 Cote d'Ivoire 2024-08-03 0.0 Cote d'Ivoire 2024-08-06 0.0 Cote d'Ivoire 2024-08-12 0.0 ~ Changed values: 8 / 85208 (0.01%) country date total_deaths_per_million - total_deaths_per_million + Africa 2024-08-19 0.03267 0.03201 Africa 2024-08-20 0.03267 0.03200 Africa 2024-08-21 0.03267 0.03200 Africa 2024-08-24 0.03266 0.03200 Africa 2024-08-25 0.03266 0.03199 Legend: +New ~Modified -Removed =Identical Details Hint: Run this locally with etl diff REMOTE data/ --include yourdataset --verbose --snippet ``` Automatically updated datasets matching _weekly_wildfires|excess_mortality|covid|fluid|flunet|country_profile|garden/ihme_gbd/2019/gbd_risk_ are not included

Edited: 2024-08-27 09:35:30 UTC Execution time: 12.84 seconds

Marigold commented 1 month ago

Maybe we could simply ask for confirmation (instead of reverting everything). What do you think?

I've actually changed my mind after our discussion. It's nicer to have ETL dashboard and git operations separated. It's just two extra commands that the user has to run, and it's way more transparent. If we decide to integrate them together, we'd better do it all in ETL dashboard without having to run CLI.