owid / owid-grapher

A platform for creating interactive data visualizations
https://ourworldindata.org
MIT License
1.36k stars 230 forks source link

check and fix `is_indexable` for charts tagged with children of Living Conditions, Community and Wellbeing and Poverty and Economic Development #2816

Closed mlbrgl closed 10 months ago

mlbrgl commented 10 months ago

Charts that should be indexable: datasette query

From live db to avoid stale results from datasette query:

2111    official-development-assistance-as-a-share-of-donor-gross-national-income
2113    use-of-crf-tools-by-providers-of-dev-cooperation
3246    leverage-ratio-of-banks-in-the-uk-and-us
3863    govt-expenditure-share-budget
4089    medical-ventilators-per-100000
4520    households-actual-and-imputed-rent-as-share-of-gdp-oecd
4544    healthcare-access-quality-un
4545    community-health-workers
4549    net-oda-per-capita
5234    money-committed-to-public-private-partnerships-for-infrastructure
5359    official-development-assistance-gni-share
5406    bank-capital-to-total-assets-ratio
5586    expenditure-on-cultural-and-natural-heritage-per-capita
5653    public-healthcare-spending-share-gdp
6905    dealt-with-anxiety-depression-took-prescribed-medication
6951    ihme--who-comparison
7016    share-covered-by-one-social-protection-benefit
185 the-share-of-people-living-in-extreme-poverty-vs-gdp-per-capita
225 gdp-per-capita-worldbank
379 income-inequality-and-growth-in-european-sub-national-regions
449 gdp-per-capita-in-international-and-market-dollars
517 national-poverty-lines-vs-gdp-per-capita
518 nominal-wages-consumer-prices-and-real-wages-in-the-uk-since-1750
1009    englands-economy-over-the-long-run-gdp-per-capita-vs-population
1540    total-wealth-per-capita-by-income-groups-in-2014-us-dollars
1566    total-wealth-by-asset-group-in-millions-2014-us-dollars
1567    wealth-per-capita-by-asset-group-with-natural-capital-disaggregated-in-2014-us-dollars
1671    gdp-per-person-employed-constant-ppp
1886    drowning-death-rate-vs-gdp-per-capita
2903    globalization-over-5-centuries
3149    gini-index-around-2015-1990-2015-countries-vs-gini-index-around-1990-1990-2015-countries
3150    gini-index-around-2015-vs-gini-index-around-2000
3231    gini-index-of-income-in-2015-vs-1990-gcip-including-non-survey-years
3233    gini-index-of-consumption-in-2015-vs-1990-gcip-survey-years-only
3234    gini-index-of-income-in-2015-vs-1990-gcip-survey-years-only
3243    catching-up-to-the-us-during-the-golden-age-and-beyond
3244    the-household-debt-to-income-ratio-and-house-prices-in-the-us
3248    average-tariff-rates-per-cent
3250    catching-up-and-stagnating-manufacturing-wages-relative-to-the-us
3251    inequality-in-wealth-earnings-and-disposable-income-us-sweden-and-japan
3311    the-growth-rate-of-productivity-over-the-long-run-14002004
3359    change-of-gdp-per-capita-extremely-poor
3456    revenue-from-corporate-income-taxes-gdp
3796    one-person-households-vs-gdp-per-capita
3941    englands-economy-over-the-long-run-gdp-vs-population
4009    share-of-the-population-that-is-70-years-and-over-vs-gdp-per-capita
4059    hospital-beds-vs-gdp
4327    total-gdp-vs-gdp-per-capita-maddison
4408    maddison-data-gdp-per-capita-in-2011us-slopechart
4410    government-expenditure-vs-gdp
4433    ppp-conversion-factor-for-private-consumption
4462    q2-gdp-growth-vs-confirmed-deaths-due-to-covid-19-per-million-people
4478    q2-gdp-growth-vs-confirmed-cases-per-million-people
4479    economic-decline-in-the-second-quarter-of-2020
4487    coal-vs-gdp-per-capita
4522    actual-and-imputed-rents-as-share-of-gdp
4523    actual-and-imputed-housing-rents-as-share-of-gdp
4539    vegetable-supply-per-person-vs-gdp-per-capita
4654    per-capita-electricity-consumption-vs-gdp-per-capita
4669    energy-use-per-person-vs-gdp-per-capita
4696    share-of-investment-in-gdp-vs-gdp-per-capita
4699    historical-share-of-population-under2-vs-gdp-per-capita
4704    share-of-gov-expenditure-in-gdp-vs-gdp-per-capita
4792    reconstruction-of-historical-global-extreme-poverty-rates-1820-2017-roser-and-hasell-2021-and-world-bank2020
4798    regional-pov-rate
4901    antidepressants-per-capita-vs-gdp
5042    historical-national-accounts-estimates-of-the-distribution-of-people-living-at-different-income-thresholds-globally
5057    use-of-interpolation-and-extrapolation-on-maddison-gdp-per-capita-data
5076    reconstruction-of-historical-poverty-trends-by-country
5165    annual-working-hours-per-worker-vs-gdp-per-capita-long-run
5180    pm25-exposure-gdp
5203    national-strategy-for-youth-employment
5227    sustainable-fisheries-as-a-proportion-of-gdp
5231    countries-with-a-signed-bilateral-investment-treaty
5233    mechanisms-to-enhance-policy-for-sustainable-development
5247    recipients-of-development-coordination
5297    industry-value-added-as-share-of-total-value-added-klems-2008
5385    gross-public-sector-debt-as-a-proportion-of-gdp
5392    merchandise-trade-as-a-proportion-of-gdp
5395    annual-gdp-growth
5400    current-account-balance-as-a-proportion-of-gdp
5401    net-portfolio-investment
5402    annual-growth-of-imports-of-goods-and-services
5403    broad-money-to-total-reserves-ratio
5405    annual-growth-of-exports-of-goods-and-services
5408    annual-broad-money-growth
5409    total-reserves-in-months-of-imports
5410    annual-growth-of-the-gross-capital-formation
5411    external-debt-as-a-share-of-gni
5413    annual-growth-of-general-government-final-consumption-expenditure
5500    mean-daily-per-capita-expenditure-vs-daily-gdp-per-capita-reference-lines
5574    countries-reporting-progress-in-multi-stakeholder-development-effectiveness-monitoring-frameworks-in-support-of-the-sdgs
5894    gini-coefficient-wid-vs-pip-data
6045    mean-daily-expenditure-per-capita-vs-gdp-per-capita-2011-ppp
6046    median-daily-per-capita-expenditure-vs-gdp-per-capita-2011-ppp
6049    share-in-poverty-relative-to-different-poverty-thresholds-2011-ppp
6153    mean-income-or-expenditure-per-day-2011-vs-2017-international-
6199    daily-mean-income-2011-ppp
6221    global-inequality-between-world-citizens-and-its-components
6226    world-income-distribution---van-zanden-et-al-eds-2-14-oecd
6301    gdp-per-capita-constructed-gdp-per-capita-wdi-gdp-per-capita-maddison
6356    share-of-electricity-produced-from-coal-vs-gdp-per-capita
6360    comparing-equivalized-income
6475    gdp-comparison
6499    pip-gini-1993-vs-2015-all-obs
6501    pip-gini-1980-vs-2018-all-obs
6502    pip-gini-1993-vs-2015-matching-obs
6503    pip-gini-1980-vs-2018-matching-obs
6504    wid-top-1-1993-vs-2015-all-obs
6505    wid-top-1-1993-vs-2015-matching-obs
6506    wid-top-1-1980-vs-2018-all-obs
6507    wid-top-1-1980-vs-2018-matching-obs
6512    change-in-inequality-according-to-two-metrics-1993-2015
6643    share-of-global-merchandise-exports
6663    proportional-change-in-gini-coefficient-between-1993-and-2015
6664    proportional-change-in-top-1-share-between-1993-and-2015
6665    proportional-change-in-gini-coefficient-between-1980-and-2018
6666    proportional-change-in-top-1-share-between-1980-and-2018
6667    absolute-change-in-top-1-share-between-1993-and-2015
6952    consumer-price-index
3230    gini-index-of-consumption-in-2015-vs-1990-gcip
5152    daily-median-income
5306    share-of-population-living-on-less-than-half-of-the-median-income
6362    luxembourg-income-study-lis-2022
7129    pip-and-wid
6426    share-of-agricultural-land-owners-that-are-women
6427    degree-to-which-the-legal-framework-guarantees-womens-equal-rights-to-land-ownership-andor-control
6763    age-of-retirement-equal-partial-pension
5579    debt-service-of-exports-of-goods-services
5580    adequacy-of-unemployment-benefits
6538    share-of-population-satisfied-with-government-services
6618    government-spending-on-mental-health-as-a-share-of-spending-on-health
3724    proportion-of-unemployed-using-specific-search-methods
3731    job-seeker-job-search-pew
4806    share-deforestation-exported
5201    level-of-national-compliance-with-labor-rights
5211    companies-publishing-sustainability-reports-minimum-requirements
5244    manufacturing-share-of-total-employment
5621    food-exports-ukraine-russia
5624    fertilizer-exports-russia-ukraine
6644    share-of-global-services-exports
413 share-multi-poverty
417 mpi-vs-consumption
5952    intensity-of-multidimensional-poverty-national
5997    share-of-population-in-multidimensional-poverty-urban-vs-rural-cme
6260    mpi-vs-consumption-over-time
6261    mpi-vs-consumption-365
236 world-population-in-extreme-poverty-absolute
328 data-deprivation-poverty-surveys-per-decade
383 total-population-living-in-extreme-poverty-by-world-region
392 share-of-population-living-with-less-than-190-and-320-per-day
418 poverty-vs-life-expectancy
419 poverty-vs-mean-schooling
424 extreme-poverty-eu-2008
437 comparison-extreme-poverty-rich-countries
446 poverty-headcount-rate-vs-poverty-gap
448 size-poverty-gap-countries
4700    historical-share-of-population-living-on-less-than-5-per-day
4701    historical-share-of-population-living-on-less-than-per-day2
4707    historical-share-of-population-living-on-less-than-5-per-day-roser-hasell
4863    share-in-poverty-relative-to-different-poverty-thresholds
4876    distribution-of-population-between-different-poverty-thresholds-up-to-30-dollars
5111    share-of-population-in-extreme-poverty
5113    share-living-with-less-than-320-int--per-day
5139    above-or-below-extreme-poverty-line-world-bank
5169    poverty-gap-index-at-190-int-per-day-povcal
5181    total-population-in-extreme-poverty
5925    share-of-population-living-in-extreme-poverty
6047    distribution-of-population-between-different-poverty-thresholds-stacke-bar-2011-ppp
6048    distribution-of-population-poverty-thresholds-2011-ppp
6050    share-of-population-in-extreme-poverty-2011-ppp
6051    mpi-vs-consumption-2011-ppp
6052    poverty-share-on-less-than-30-per-day-2011-ppp
2631    depression-by-level-of-education-employment
4596    working-hours-uk-millennium
579 life-satisfaction-vs-life-expectancy
3768    people-who-report-having-friends-or-relatives-they-can-count-on
3860    safety-walking-alone
5839    happiness-cantril-ladder-marimekko
6418    share-of-survey-respondents-who-say-they-do-not-trust-at-all-people-they-meet-for-the-first-time
3305    the-duration-of-democracy-and-working-hours-2014
4683    average-effective-retirement-men
3861    share-of-victims-who-reported-crime-to-the-police
5876    political-participation-index-eiu
6419    share-of-survey-respondents-who-agree-most-people-would-try-to-take-advantage-of-you
6609    share-of-people-who-find-political-decision-making-inclusive-and-responsive
3927    marriage-rate-per-1000-inhabitants
4634    us-child-care-time-sex
3507    self-reported-loneliness-older-adults
3779    one-person-households
3781    adults-living-alone
4599    time-spent-with-relationships-by-age-us
4620    time-spent-sports
4621    time-spent-sports-usa
mlbrgl commented 10 months ago

Update query: UPDATE charts SET is_indexable = 1 WHERE charts.id IN (2111,2113,3246,3863,4089,4520,4544,4545,4549,5234,5359,5406,5586,5653,6905,6951,7016,185,225,379,449,517,518,1009,1540,1566,1567,1671,1886,2903,3149,3150,3231,3233,3234,3243,3244,3248,3250,3251,3311,3359,3456,3796,3941,4009,4059,4327,4408,4410,4433,4462,4478,4479,4487,4522,4523,4539,4654,4669,4696,4699,4704,4792,4798,4901,5042,5057,5076,5165,5180,5203,5227,5231,5233,5247,5297,5385,5392,5395,5400,5401,5402,5403,5405,5408,5409,5410,5411,5413,5500,5574,5894,6045,6046,6049,6153,6199,6221,6226,6301,6356,6360,6475,6499,6501,6502,6503,6504,6505,6506,6507,6512,6643,6663,6664,6665,6666,6667,6952,3230,5152,5306,6362,7129,6426,6427,6763,5579,5580,6538,6618,3724,3731,4806,5201,5211,5244,5621,5624,6644,413,417,5952,5997,6260,6261,236,328,383,392,418,419,424,437,446,448,4700,4701,4707,4863,4876,5111,5113,5139,5169,5181,5925,6047,6048,6050,6051,6052,2631,4596,579,3768,3860,5839,6418,3305,4683,3861,5876,6419,6609,3927,4634,3507,3779,3781,4599,4620,4621)

mlbrgl commented 10 months ago

@spoonerf it occurred to me after running the update query that one of these charts could be unlisted, in which case they should have been excluded from the update.

Checking for charts that are both unlisted and is_indexable against the live DB returns nothing after the update, so we're good:

SELECT DISTINCT charts.id FROM charts LEFT JOIN chart_tags ON charts.id = chart_tags.chartId LEFT JOIN tags ON tags.id = chart_tags.tagId WHERE charts.is_indexable = 1 AND tags.name="Unlisted"

spoonerf commented 10 months ago

Ah good thinking @mlbrgl! Thanks for the fix. So after the indexing runs in a few hours these charts should reappear in Search?

mlbrgl commented 10 months ago

Yes! (they did)