owid / etl

A compute graph for loading and transforming OWID's data
https://docs.owid.io/projects/etl
MIT License
78 stars 21 forks source link

:bug: fix duplicate values in covid sequences #3180

Closed Marigold closed 1 month ago

owidbot commented 1 month ago
Quick links (staging server): Site Admin Wizard

Login: ssh owid@staging-site-fix-covid-sequence

chart-diff: ✅ No charts for review.
data-diff: ❌ Found differences ```diff = Dataset garden/who/latest/monkeypox = Table monkeypox ~ Dim country + + New values: 366 / 85201 (0.43%) date country 2024-08-09 Barbados 2022-05-03 Brazil 2022-06-05 Brazil 2024-08-12 Colombia 2024-08-11 Paraguay - - Removed values: 29 / 85201 (0.03%) date country 2022-06-30 Bahamas 2022-07-09 Bahamas 2022-07-11 Bahamas 2022-07-18 Bahamas 2022-07-21 Bahamas ~ Dim date + + New values: 366 / 85201 (0.43%) country date Barbados 2024-08-09 Brazil 2022-05-03 Brazil 2022-06-05 Colombia 2024-08-12 Paraguay 2024-08-11 - - Removed values: 29 / 85201 (0.03%) country date Bahamas 2022-06-30 Bahamas 2022-07-09 Bahamas 2022-07-11 Bahamas 2022-07-18 Bahamas 2022-07-21 ~ Column annotation (new data, changed data) + + New values: 366 / 85201 (0.43%) country date annotation Barbados 2024-08-09 Brazil 2022-05-03 Brazil 2022-06-05 Colombia 2024-08-12 Paraguay 2024-08-11 - - Removed values: 29 / 85201 (0.03%) country date annotation Bahamas 2022-06-30 Bahamas 2022-07-09 Bahamas 2022-07-11 Bahamas 2022-07-18 Bahamas 2022-07-21 ~ Column iso_code (new data, changed data) + + New values: 366 / 85201 (0.43%) country date iso_code Barbados 2024-08-09 BRB Brazil 2022-05-03 BRA Brazil 2022-06-05 BRA Colombia 2024-08-12 COL Paraguay 2024-08-11 PRY - - Removed values: 29 / 85201 (0.03%) country date iso_code Bahamas 2022-06-30 BHS Bahamas 2022-07-09 BHS Bahamas 2022-07-11 BHS Bahamas 2022-07-18 BHS Bahamas 2022-07-21 BHS ~ Column new_cases (new data, changed data) + + New values: 366 / 85201 (0.43%) country date new_cases Barbados 2024-08-09 0 Brazil 2022-05-03 716 Brazil 2022-06-05 0 Colombia 2024-08-12 0 Paraguay 2024-08-11 0 - - Removed values: 29 / 85201 (0.03%) country date new_cases Bahamas 2022-06-30 0 Bahamas 2022-07-09 0 Bahamas 2022-07-11 0 Bahamas 2022-07-18 0 Bahamas 2022-07-21 0 ~ Changed values: 849 / 85201 (1.00%) country date new_cases - new_cases + Canada 2024-06-25 6 7 Ecuador 2022-07-12 0 2 Ecuador 2022-12-20 0 11 Ecuador 2023-07-25 0 2 North America 2022-09-20 1209 1214 ~ Column new_cases_per_million (new data, changed data) + + New values: 366 / 85201 (0.43%) country date new_cases_per_million Barbados 2024-08-09 0.000 Brazil 2022-05-03 3.328 Brazil 2022-06-05 0.000 Colombia 2024-08-12 0.000 Paraguay 2024-08-11 0.000 - - Removed values: 29 / 85201 (0.03%) country date new_cases_per_million Bahamas 2022-06-30 0.0 Bahamas 2022-07-09 0.0 Bahamas 2022-07-11 0.0 Bahamas 2022-07-18 0.0 Bahamas 2022-07-21 0.0 ~ Changed values: 828 / 85201 (0.97%) country date new_cases_per_million - new_cases_per_million + Asia 2023-07-03 0.000 0.004 Brazil 2024-02-13 0.000 0.152 Mexico 2024-05-07 0.015 0.000 United States 2022-09-20 2.643 2.657 United States 2022-10-04 1.656 1.668 ~ Column new_cases_smoothed (new data, changed data) + + New values: 366 / 85201 (0.43%) country date new_cases_smoothed Barbados 2024-08-09 0.000000 Brazil 2022-05-03 102.290001 Brazil 2022-06-05 0.570000 Colombia 2024-08-12 0.140000 Paraguay 2024-08-11 0.000000 - - Removed values: 29 / 85201 (0.03%) country date new_cases_smoothed Bahamas 2022-06-30 0.14 Bahamas 2022-07-09 0.00 Bahamas 2022-07-11 0.00 Bahamas 2022-07-18 0.00 Bahamas 2022-07-21 0.00 ~ Changed values: 5743 / 85201 (6.74%) country date new_cases_smoothed - new_cases_smoothed + Africa 2024-07-22 32.430000 33.000000 Canada 2024-06-25 0.860000 1.000000 Ecuador 2023-10-11 0.000000 0.290000 Mexico 2024-07-06 0.710000 0.140000 United States 2022-08-26 308.709991 313.140015 ~ Column new_cases_smoothed_per_million (new data, changed data) + + New values: 366 / 85201 (0.43%) country date new_cases_smoothed_per_million Barbados 2024-08-09 0.000 Brazil 2022-05-03 0.475 Brazil 2022-06-05 0.003 Colombia 2024-08-12 0.003 Paraguay 2024-08-11 0.000 - - Removed values: 29 / 85201 (0.03%) country date new_cases_smoothed_per_million Bahamas 2022-06-30 0.341 Bahamas 2022-07-09 0.000 Bahamas 2022-07-11 0.000 Bahamas 2022-07-18 0.000 Bahamas 2022-07-21 0.000 ~ Changed values: 4796 / 85201 (5.63%) country date new_cases_smoothed_per_million - new_cases_smoothed_per_million + Brazil 2023-09-02 0.000 0.009 Ecuador 2022-12-25 0.000 0.087 Peru 2023-08-12 0.000 0.013 United States 2022-05-31 0.022 0.020 United States 2023-10-08 0.021 0.022 ~ Column new_deaths (new data, changed data) + + New values: 366 / 85201 (0.43%) country date new_deaths Barbados 2024-08-09 0 Brazil 2022-05-03 2 Brazil 2022-06-05 0 Colombia 2024-08-12 0 Paraguay 2024-08-11 0 - - Removed values: 29 / 85201 (0.03%) country date new_deaths Bahamas 2022-06-30 0 Bahamas 2022-07-09 0 Bahamas 2022-07-11 0 Bahamas 2022-07-18 0 Bahamas 2022-07-21 0 ~ Changed values: 85 / 85201 (0.10%) country date new_deaths - new_deaths + Brazil 2022-07-12 0 3 Brazil 2022-11-29 2 0 South America 2022-09-20 0 1 South America 2022-11-29 4 2 World 2022-08-02 8 7 ~ Column new_deaths_per_million (new data, changed data) + + New values: 366 / 85201 (0.43%) country date new_deaths_per_million Barbados 2024-08-09 0.0000 Brazil 2022-05-03 0.0093 Brazil 2022-06-05 0.0000 Colombia 2024-08-12 0.0000 Paraguay 2024-08-11 0.0000 - - Removed values: 29 / 85201 (0.03%) country date new_deaths_per_million Bahamas 2022-06-30 0.0 Bahamas 2022-07-09 0.0 Bahamas 2022-07-11 0.0 Bahamas 2022-07-18 0.0 Bahamas 2022-07-21 0.0 ~ Changed values: 85 / 85201 (0.10%) country date new_deaths_per_million - new_deaths_per_million + Brazil 2022-07-12 0.00000 0.01393 Brazil 2022-11-29 0.00927 0.00000 South America 2022-09-20 0.00000 0.00229 South America 2022-11-29 0.00913 0.00457 World 2022-08-02 0.00100 0.00088 ~ Column new_deaths_smoothed (new data, changed data) + + New values: 366 / 85201 (0.43%) country date new_deaths_smoothed Barbados 2024-08-09 0.00 Brazil 2022-05-03 0.29 Brazil 2022-06-05 0.00 Colombia 2024-08-12 0.00 Paraguay 2024-08-11 0.00 - - Removed values: 29 / 85201 (0.03%) country date new_deaths_smoothed Bahamas 2022-06-30 0.0 Bahamas 2022-07-09 0.0 Bahamas 2022-07-11 0.0 Bahamas 2022-07-18 0.0 Bahamas 2022-07-21 0.0 ~ Changed values: 592 / 85201 (0.69%) country date new_deaths_smoothed - new_deaths_smoothed + Ecuador 2022-09-22 0.00 0.14 South America 2022-07-05 0.00 0.29 South America 2022-08-04 0.14 0.00 South America 2022-12-06 0.29 0.14 World 2024-05-11 0.29 0.00 ~ Column new_deaths_smoothed_per_million (new data, changed data) + + New values: 366 / 85201 (0.43%) country date new_deaths_smoothed_per_million Barbados 2024-08-09 0.00000 Brazil 2022-05-03 0.00135 Brazil 2022-06-05 0.00000 Colombia 2024-08-12 0.00000 Paraguay 2024-08-11 0.00000 - - Removed values: 29 / 85201 (0.03%) country date new_deaths_smoothed_per_million Bahamas 2022-06-30 0.0 Bahamas 2022-07-09 0.0 Bahamas 2022-07-11 0.0 Bahamas 2022-07-18 0.0 Bahamas 2022-07-21 0.0 ~ Changed values: 592 / 85201 (0.69%) country date new_deaths_smoothed_per_million - new_deaths_smoothed_per_million + Ecuador 2022-09-22 0.00000 0.00776 South America 2022-07-05 0.00000 0.00066 South America 2022-08-04 0.00032 0.00000 South America 2022-12-06 0.00066 0.00032 World 2024-05-11 0.00004 0.00000 ~ Column suspected_cases_cumulative (new data) + + New values: 366 / 85201 (0.43%) country date suspected_cases_cumulative Barbados 2024-08-09 Brazil 2022-05-03 Brazil 2022-06-05 Colombia 2024-08-12 Paraguay 2024-08-11 - - Removed values: 29 / 85201 (0.03%) country date suspected_cases_cumulative Bahamas 2022-06-30 Bahamas 2022-07-09 Bahamas 2022-07-11 Bahamas 2022-07-18 Bahamas 2022-07-21 ~ Column total_cases (new data, changed data) + + New values: 366 / 85201 (0.43%) country date total_cases Barbados 2024-08-09 1 Brazil 2022-05-03 716 Brazil 2022-06-05 884 Colombia 2024-08-12 4257 Paraguay 2024-08-11 126 - - Removed values: 29 / 85201 (0.03%) country date total_cases Bahamas 2022-06-30 1 Bahamas 2022-07-09 1 Bahamas 2022-07-11 1 Bahamas 2022-07-18 1 Bahamas 2022-07-21 1 ~ Changed values: 8124 / 85201 (9.54%) country date total_cases - total_cases + Bahamas 2023-02-17 2 1 Bahamas 2023-06-04 2 3 Brazil 2023-03-22 10890 10586 Brazil 2024-03-01 11212 11474 North America 2024-03-06 39090 39161 ~ Column total_cases_per_million (new data, changed data) + + New values: 366 / 85201 (0.43%) country date total_cases_per_million Barbados 2024-08-09 3.542000 Brazil 2022-05-03 3.328000 Brazil 2022-06-05 4.107000 Colombia 2024-08-12 81.283997 Paraguay 2024-08-11 18.112000 - - Removed values: 29 / 85201 (0.03%) country date total_cases_per_million Bahamas 2022-06-30 2.439 Bahamas 2022-07-09 2.439 Bahamas 2022-07-11 2.439 Bahamas 2022-07-18 2.438 Bahamas 2022-07-21 2.438 ~ Changed values: 8102 / 85201 (9.51%) country date total_cases_per_million - total_cases_per_million + Africa 2024-08-01 3.790000 3.796000 Ecuador 2023-04-26 29.356001 35.634998 Panama 2023-06-15 52.625999 54.417000 South America 2023-02-03 49.981998 49.668999 United States 2023-03-27 89.477997 89.462997 ~ Column total_deaths (new data, changed data) + + New values: 366 / 85201 (0.43%) country date total_deaths Barbados 2024-08-09 0 Brazil 2022-05-03 2 Brazil 2022-06-05 2 Colombia 2024-08-12 0 Paraguay 2024-08-11 0 - - Removed values: 29 / 85201 (0.03%) country date total_deaths Bahamas 2022-06-30 0 Bahamas 2022-07-09 0 Bahamas 2022-07-11 0 Bahamas 2022-07-18 0 Bahamas 2022-07-21 0 ~ Changed values: 6052 / 85201 (7.10%) country date total_deaths - total_deaths + Brazil 2023-12-07 16 15 Chile 2023-06-10 2 3 South America 2023-10-01 44 45 World 2022-08-20 35 43 World 2023-05-08 160 165 ~ Column total_deaths_per_million (new data, changed data) + + New values: 366 / 85201 (0.43%) country date total_deaths_per_million Barbados 2024-08-09 0.00000 Brazil 2022-05-03 0.00930 Brazil 2022-06-05 0.00929 Colombia 2024-08-12 0.00000 Paraguay 2024-08-11 0.00000 - - Removed values: 29 / 85201 (0.03%) country date total_deaths_per_million Bahamas 2022-06-30 0.0 Bahamas 2022-07-09 0.0 Bahamas 2022-07-11 0.0 Bahamas 2022-07-18 0.0 Bahamas 2022-07-21 0.0 ~ Changed values: 6052 / 85201 (7.10%) country date total_deaths_per_million - total_deaths_per_million + Brazil 2023-12-07 0.07375 0.06914 Chile 2023-06-10 0.10189 0.15284 South America 2023-10-01 0.09988 0.10215 World 2022-08-20 0.00438 0.00539 World 2023-05-08 0.01991 0.02054 Legend: +New ~Modified -Removed =Identical Details Hint: Run this locally with etl diff REMOTE data/ --include yourdataset --verbose --snippet ``` Automatically updated datasets matching _weekly_wildfires|excess_mortality|covid|fluid|flunet|country_profile|garden/ihme_gbd/2019/gbd_risk_ are not included

Edited: 2024-08-23 07:38:19 UTC Execution time: 14.54 seconds