search

owid / etl

A compute graph for loading and transforming OWID's data
https://docs.owid.io/projects/etl
MIT License
86 stars 23 forks source link

🐛 famines: missing values in World population affected death rates #3479

Closed veronikasamborska1994 closed 3 weeks ago

owidbot commented 3 weeks ago
Quick links (staging server): Site Admin Wizard

Login: ssh owid@staging-site-famines-bug

chart-diff: ✅
  • 2/2 reviewed charts
  • Modified: 0/0
  • New: 2/2
  • Rejected: 0
data-diff: ❌ Found differences ```diff = Dataset garden/wpf/2024-10-03/famines = Table famines ~ Dim famine_name + + New values: 4 / 75 (5.33%) date famine_name 1937,1938,1939,1940,1941,1942,1943,1944,1945 China 1937-1945 1942,1943,1944,1945 Indonesia 1942-1945 1910,1911,1912,1913,1913,1914,1914,1915,1916,1917,1918,1919 Somaliland, African Red Sea Region 1910-1919 1941,1942,1943,1944 USSR (Hungerplan) 1941-1944 - - Removed values: 2 / 75 (2.67%) date famine_name 1910,1911,1912,1913,1913,1914,1914,1915,1916,1917,1918,1919 Somaliland 1910-1919 1932,1933,1934,1941,1942,1943,1944,1941,1942,1943,1944,1945,1946,1947 Ukraine 1932-1947 ~ Dim date + + New values: 4 / 75 (5.33%) famine_name date China 1937-1945 1937,1938,1939,1940,1941,1942,1943,1944,1945 Indonesia 1942-1945 1942,1943,1944,1945 Somaliland, African Red Sea Region 1910-1919 1910,1911,1912,1913,1913,1914,1914,1915,1916,1917,1918,1919 USSR (Hungerplan) 1941-1944 1941,1942,1943,1944 - - Removed values: 2 / 75 (2.67%) famine_name date Somaliland 1910-1919 1910,1911,1912,1913,1913,1914,1914,1915,1916,1917,1918,1919 Ukraine 1932-1947 1932,1933,1934,1941,1942,1943,1944,1941,1942,1943,1944,1945,1946,1947 ~ Column conflict (new data, changed data) + + New values: 4 / 75 (5.33%) famine_name date conflict China 1937-1945 1937,1938,1939,1940,1941,1942,1943,1944,1945 1.0 Indonesia 1942-1945 1942,1943,1944,1945 1.0 Somaliland, African Red Sea Region 1910-1919 1910,1911,1912,1913,1913,1914,1914,1915,1916,1917,1918,1919 1.0 USSR (Hungerplan) 1941-1944 1941,1942,1943,1944 1.0 - - Removed values: 2 / 75 (2.67%) famine_name date conflict Somaliland 1910-1919 1910,1911,1912,1913,1913,1914,1914,1915,1916,1917,1918,1919 1.0 Ukraine 1932-1947 1932,1933,1934,1941,1942,1943,1944,1941,1942,1943,1944,1945,1946,1947 1.0 ~ Column country (new data, changed data) + + New values: 4 / 75 (5.33%) famine_name date country China 1937-1945 1937,1938,1939,1940,1941,1942,1943,1944,1945 China Indonesia 1942-1945 1942,1943,1944,1945 Indonesia Somaliland, African Red Sea Region 1910-1919 1910,1911,1912,1913,1913,1914,1914,1915,1916,1917,1918,1919 Somaliland, African Red Sea Region USSR (Hungerplan) 1941-1944 1941,1942,1943,1944 USSR - - Removed values: 2 / 75 (2.67%) famine_name date country Somaliland 1910-1919 1910,1911,1912,1913,1913,1914,1914,1915,1916,1917,1918,1919 Somaliland Ukraine 1932-1947 1932,1933,1934,1941,1942,1943,1944,1941,1942,1943,1944,1945,1946,1947 Ukraine ~ Column external_factors (new data, changed data) + + New values: 4 / 75 (5.33%) famine_name date external_factors China 1937-1945 1937,1938,1939,1940,1941,1942,1943,1944,1945 1.0 Indonesia 1942-1945 1942,1943,1944,1945 1.0 Somaliland, African Red Sea Region 1910-1919 1910,1911,1912,1913,1913,1914,1914,1915,1916,1917,1918,1919 1.0 USSR (Hungerplan) 1941-1944 1941,1942,1943,1944 1.0 - - Removed values: 2 / 75 (2.67%) famine_name date external_factors Somaliland 1910-1919 1910,1911,1912,1913,1913,1914,1914,1915,1916,1917,1918,1919 1.0 Ukraine 1932-1947 1932,1933,1934,1941,1942,1943,1944,1941,1942,1943,1944,1945,1946,1947 1.0 ~ Column government_policy_overall (new data, changed data) + + New values: 4 / 75 (5.33%) famine_name date government_policy_overall China 1937-1945 1937,1938,1939,1940,1941,1942,1943,1944,1945 0.0 Indonesia 1942-1945 1942,1943,1944,1945 1.0 Somaliland, African Red Sea Region 1910-1919 1910,1911,1912,1913,1913,1914,1914,1915,1916,1917,1918,1919 1.0 USSR (Hungerplan) 1941-1944 1941,1942,1943,1944 1.0 - - Removed values: 2 / 75 (2.67%) famine_name date government_policy_overall Somaliland 1910-1919 1910,1911,1912,1913,1913,1914,1914,1915,1916,1917,1918,1919 0.0 Ukraine 1932-1947 1932,1933,1934,1941,1942,1943,1944,1941,1942,1943,1944,1945,1946,1947 1.0 ~ Column region (new data, changed data) + + New values: 4 / 75 (5.33%) famine_name date region China 1937-1945 1937,1938,1939,1940,1941,1942,1943,1944,1945 Asia Indonesia 1942-1945 1942,1943,1944,1945 Asia Somaliland, African Red Sea Region 1910-1919 1910,1911,1912,1913,1913,1914,1914,1915,1916,1917,1918,1919 Africa USSR (Hungerplan) 1941-1944 1941,1942,1943,1944 Asia - - Removed values: 2 / 75 (2.67%) famine_name date region Somaliland 1910-1919 1910,1911,1912,1913,1913,1914,1914,1915,1916,1917,1918,1919 Africa Ukraine 1932-1947 1932,1933,1934,1941,1942,1943,1944,1941,1942,1943,1944,1945,1946,1947 Europe ~ Column wpf_authoritative_mortality_estimate (changed metadata, new data, changed data) - - description_processing: The deaths were assumed to have been distributed evenly over the duration of the famine. + + New values: 4 / 75 (5.33%) famine_name date wpf_authoritative_mortality_estimate China 1937-1945 1937,1938,1939,1940,1941,1942,1943,1944,1945 5000000 Indonesia 1942-1945 1942,1943,1944,1945 2400000 Somaliland, African Red Sea Region 1910-1919 1910,1911,1912,1913,1913,1914,1914,1915,1916,1917,1918,1919 100000 USSR (Hungerplan) 1941-1944 1941,1942,1943,1944 10000000 - - Removed values: 2 / 75 (2.67%) famine_name date wpf_authoritative_mortality_estimate Somaliland 1910-1919 1910,1911,1912,1913,1913,1914,1914,1915,1916,1917,1918,1919 100000 Ukraine 1932-1947 1932,1933,1934,1941,1942,1943,1944,1941,1942,1943,1944,1945,1946,1947 14600000 = Dataset garden/wpf/2024-10-03/famines_by_factor = Table famines_by_factor ~ Dim year + + New values: 4 / 290 (1.38%) country year Asia 1938 World 1938 Asia 1939 Asia 1940 ~ Dim country + + New values: 4 / 290 (1.38%) year country 1938 Asia 1938 World 1939 Asia 1940 Asia ~ Column sum_conflict_mortality_factor (new data, changed data) + + New values: 4 / 290 (1.38%) year country sum_conflict_mortality_factor 1938 Asia 555555.5625 1938 World 555555.5625 1939 Asia 555555.5625 1940 Asia 555555.5625 ~ Changed values: 29 / 290 (10.00%) year country sum_conflict_mortality_factor - sum_conflict_mortality_factor + 1933 Europe 2209523.75 1.166667e+06 1941 Europe 3742857.25 2.700000e+06 1942 Asia 825000.00 4.480556e+06 1944 Europe 3692857.25 2.650000e+06 1945 Europe 1276190.50 2.333333e+05 ~ Column sum_conflict_mortality_not_a_factor (new data) + + New values: 4 / 290 (1.38%) year country sum_conflict_mortality_not_a_factor 1938 Asia 0.0 1938 World 0.0 1939 Asia 0.0 1940 Asia 0.0 ~ Column sum_external_factors_mortality_factor (new data, changed data) + + New values: 4 / 290 (1.38%) year country sum_external_factors_mortality_factor 1938 Asia 555555.5625 1938 World 555555.5625 1939 Asia 555555.5625 1940 Asia 555555.5625 ~ Changed values: 29 / 290 (10.00%) year country sum_external_factors_mortality_factor - sum_external_factors_mortality_factor + 1933 Europe 2209523.750 1166666.625 1941 Europe 1242857.125 200000.000 1942 Asia 825000.000 4480555.500 1944 Europe 1192857.125 150000.000 1945 Europe 1192857.125 150000.000 ~ Column sum_external_factors_mortality_not_a_factor (new data) + + New values: 4 / 290 (1.38%) year country sum_external_factors_mortality_not_a_factor 1938 Asia 0.0 1938 World 0.0 1939 Asia 0.0 1940 Asia 0.0 ~ Column sum_government_policy_overall_mortality_factor (new data, changed data) + + New values: 4 / 290 (1.38%) year country sum_government_policy_overall_mortality_factor 1938 Asia 0.0 1938 World 0.0 1939 Asia 0.0 1940 Asia 0.0 ~ Changed values: 45 / 290 (15.52%) year country sum_government_policy_overall_mortality_factor - sum_government_policy_overall_mortality_factor + 1912 Africa 0.00 8.333333e+03 1915 Africa 0.00 8.333333e+03 1941 Europe 3742857.25 2.700000e+06 1942 Europe 3742857.25 2.700000e+06 1944 World 5017857.00 7.075000e+06 ~ Column sum_government_policy_overall_mortality_not_a_factor (new data, changed data) + + New values: 4 / 290 (1.38%) year country sum_government_policy_overall_mortality_not_a_factor 1938 Asia 555555.5625 1938 World 555555.5625 1939 Asia 555555.5625 1940 Asia 555555.5625 ~ Changed values: 34 / 290 (11.72%) year country sum_government_policy_overall_mortality_not_a_factor - sum_government_policy_overall_mortality_not_a_factor + 1911 Africa 8333.333008 0.00000 1915 World 441250.000000 432916.65625 1918 World 492983.343750 484650.00000 1937 Asia 0.000000 555555.56250 1941 World 0.000000 555555.56250 + Dataset garden/wpf/2024-10-03/famines_by_place + + Table famines_by_place + + Column wpf_authoritative_mortality_estimate + + Column decadal_famine_deaths = Dataset garden/wpf/2024-10-03/famines_by_regime_gdp = Table famines ~ Dim famine_name + + New values: 27 / 304 (8.88%) year famine_name 1939 China 1937-1945 1942 China 1937-1945 1911 Somaliland, African Red Sea Region 1910-1919 1914 Somaliland, African Red Sea Region 1910-1919 1942 USSR (Hungerplan) 1941-1944 - - Removed values: 20 / 304 (6.58%) year famine_name 1911 Somaliland 1910-1919 1918 Somaliland 1910-1919 1932 Ukraine 1932-1947 1946 Ukraine 1932-1947 1947 Ukraine 1932-1947 ~ Dim year + + New values: 27 / 304 (8.88%) famine_name year China 1937-1945 1939 China 1937-1945 1942 Somaliland, African Red Sea Region 1910-1919 1911 Somaliland, African Red Sea Region 1910-1919 1914 USSR (Hungerplan) 1941-1944 1942 - - Removed values: 20 / 304 (6.58%) famine_name year Somaliland 1910-1919 1911 Somaliland 1910-1919 1918 Ukraine 1932-1947 1932 Ukraine 1932-1947 1946 Ukraine 1932-1947 1947 ~ Column gdp_per_capita (new data, changed data) + + New values: 27 / 304 (8.88%) famine_name year gdp_per_capita China 1937-1945 1939 NaN China 1937-1945 1942 NaN Somaliland, African Red Sea Region 1910-1919 1911 NaN Somaliland, African Red Sea Region 1910-1919 1914 NaN USSR (Hungerplan) 1941-1944 1942 NaN - - Removed values: 20 / 304 (6.58%) famine_name year gdp_per_capita Somaliland 1910-1919 1911 NaN Somaliland 1910-1919 1918 NaN Ukraine 1932-1947 1932 2294.0 Ukraine 1932-1947 1946 3049.0 Ukraine 1932-1947 1947 3389.0 ~ Column regime_redux_row_owid (new data, changed data) + + New values: 27 / 304 (8.88%) famine_name year regime_redux_row_owid China 1937-1945 1939 3.0 China 1937-1945 1942 3.0 Somaliland, African Red Sea Region 1910-1919 1911 3.0 Somaliland, African Red Sea Region 1910-1919 1914 3.0 USSR (Hungerplan) 1941-1944 1942 3.0 - - Removed values: 20 / 304 (6.58%) famine_name year regime_redux_row_owid Somaliland 1910-1919 1911 3.0 Somaliland 1910-1919 1918 3.0 Ukraine 1932-1947 1932 3.0 Ukraine 1932-1947 1946 3.0 Ukraine 1932-1947 1947 3.0 ~ Column region (new data, changed data) + + New values: 27 / 304 (8.88%) famine_name year region China 1937-1945 1939 Asia China 1937-1945 1942 Asia Somaliland, African Red Sea Region 1910-1919 1911 Africa Somaliland, African Red Sea Region 1910-1919 1914 Africa USSR (Hungerplan) 1941-1944 1942 Asia - - Removed values: 20 / 304 (6.58%) famine_name year region Somaliland 1910-1919 1911 Africa Somaliland 1910-1919 1918 Africa Ukraine 1932-1947 1932 Europe Ukraine 1932-1947 1946 Europe Ukraine 1932-1947 1947 Europe ~ Column wpf_authoritative_mortality_estimate (new data, changed data) + + New values: 27 / 304 (8.88%) famine_name year wpf_authoritative_mortality_estimate China 1937-1945 1939 5000000 China 1937-1945 1942 5000000 Somaliland, African Red Sea Region 1910-1919 1911 100000 Somaliland, African Red Sea Region 1910-1919 1914 100000 USSR (Hungerplan) 1941-1944 1942 10000000 - - Removed values: 20 / 304 (6.58%) famine_name year wpf_authoritative_mortality_estimate Somaliland 1910-1919 1911 100000 Somaliland 1910-1919 1918 100000 Ukraine 1932-1947 1932 14600000 Ukraine 1932-1947 1946 14600000 Ukraine 1932-1947 1947 14600000 = Dataset garden/wpf/2024-10-03/total_famines_by_year_decade = Table total_famines_by_year_decade ~ Column decadal_famine_count (changed data) ~ Changed values: 5 / 924 (0.54%) year country decadal_famine_count - decadal_famine_count + 1930 Asia 8 11 1930 Europe 7 4 1940 Asia 7 21 1940 Europe 33 26 1940 World 40 47 ~ Column decadal_famine_deaths (changed data) ~ Changed values: 6 / 924 (0.65%) year country decadal_famine_deaths - decadal_famine_deaths + 1930 Asia 10833333.0 12500000.0 1930 Europe 6678571.5 3550000.0 1930 World 17511904.0 16050000.0 1940 Asia 7300000.0 23033334.0 1940 World 27000000.0 35433332.0 ~ Column decadal_famine_deaths_rate (changed data) ~ Changed values: 20 / 924 (2.16%) year country decadal_famine_deaths_rate - decadal_famine_deaths_rate + 1880 World 163.139023 164.872116 1930 World 831.364136 778.128601 1940 Europe 3740.337402 2354.323975 2010 World 14.331697 14.408136 2020 World 6.893198 6.931939 ~ Column famine_count (changed data) ~ Changed values: 33 / 924 (3.57%) year country famine_count - famine_count + 1933 Europe 2 1 1939 World 1 2 1942 Asia 1 4 1943 Asia 2 5 1944 Europe 5 4 ~ Column famine_deaths (changed data) ~ Changed values: 33 / 924 (3.57%) year country famine_deaths - famine_deaths + 1933 Europe 2209523.75 1166666.6250 1939 World 50000.00 605555.5625 1942 Asia 825000.00 4480555.5000 1943 Asia 3825000.00 7480555.5000 1944 Europe 3692857.25 2650000.0000 ~ Column famine_deaths_per_rate (changed data) ~ Changed values: 150 / 924 (16.23%) year country famine_deaths_per_rate - famine_deaths_per_rate + 1877 World NaN 317.494904 1904 World NaN 11.682498 1932 World NaN 79.223045 1973 World 5.530327 5.559914 1981 World 1.324861 1.331714 Legend: +New ~Modified -Removed =Identical Details Hint: Run this locally with etl diff REMOTE data/ --include yourdataset --verbose --snippet ``` Automatically updated datasets matching _weekly_wildfires|excess_mortality|covid|fluid|flunet|country_profile|garden/ihme_gbd/2019/gbd_risk_ are not included

Edited: 2024-11-01 14:20:08 UTC Execution time: 4.41 seconds