owid / etl

A compute graph for loading and transforming OWID's data
https://docs.owid.io/projects/etl
MIT License
85 stars 23 forks source link

:bar_chart: aggregate migrants between regions #3557

Closed antea04 closed 1 day ago

antea04 commented 4 days ago

belongs to owid/owid-issues#1722

The goal of this PR is to create an indicator that shows aggregated migrants who moved between different regions (in absolute numbers)

owidbot commented 4 days ago
Quick links (staging server): Site Dev Site Preview Admin Wizard Docs

Login: ssh owid@staging-site-migration-subregions

chart-diff: ✅
  • 1/1 reviewed charts
  • Modified: 0/0
  • New: 1/1
  • Rejected: 0
data-diff: ❌ Found differences ```diff = Dataset garden/antibiotics/2024-10-25/esvac_sales_corrected = Table esvac_sales_corrected ⚠ Error: Index must be unique. = Dataset garden/artificial_intelligence/2023-06-14/ai_deepfakes = Table ai_deepfakes ⚠ Error: Index must be unique. ⚠ Error: Index must be unique. = Dataset garden/artificial_intelligence/2024-02-15/epoch_llms = Table epoch_llms ~ Column dataset_size__tokens (changed metadata) - - Owen, David. (2023). Large Language Model performance and compute, Epoch (2023) [Data set]. In Extrapolating performance in language modeling benchmarks. Published online at epoch.ai. Retrieved from: 'https://epoch.ai/blog/extrapolating-performance-in-language-modelling-benchmarks' . ? ^^^^ ^^^ + + Owen, David. (2023). Large Language Model performance and compute, Epoch (2023) [Data set]. In Extrapolating performance in language modeling benchmarks. Published online at epochai.org. Retrieved from: 'https://epochai.org/blog/extrapolating-performance-in-language-modelling-benchmarks' . ? ^^^^^^^ ^^^^^^ - - url_main: https://epoch.ai/blog/extrapolating-performance-in-language-modelling-benchmarks ? - + + url_main: https://epochai.org/blog/extrapolating-performance-in-language-modelling-benchmarks ? ++++ - - url: https://epoch.ai/blog/extrapolating-performance-in-language-modelling-benchmarks ? - + + url: https://epochai.org/blog/extrapolating-performance-in-language-modelling-benchmarks ? ++++ ~ Column mmlu_avg (changed metadata) - - Owen, David. (2023). Large Language Model performance and compute, Epoch (2023) [Data set]. In Extrapolating performance in language modeling benchmarks. Published online at epoch.ai. Retrieved from: 'https://epoch.ai/blog/extrapolating-performance-in-language-modelling-benchmarks' . ? ^^^^ ^^^ + + Owen, David. (2023). Large Language Model performance and compute, Epoch (2023) [Data set]. In Extrapolating performance in language modeling benchmarks. Published online at epochai.org. Retrieved from: 'https://epochai.org/blog/extrapolating-performance-in-language-modelling-benchmarks' . ? ^^^^^^^ ^^^^^^ - - url_main: https://epoch.ai/blog/extrapolating-performance-in-language-modelling-benchmarks ? - + + url_main: https://epochai.org/blog/extrapolating-performance-in-language-modelling-benchmarks ? ++++ - - url: https://epoch.ai/blog/extrapolating-performance-in-language-modelling-benchmarks ? - + + url: https://epochai.org/blog/extrapolating-performance-in-language-modelling-benchmarks ? ++++ ~ Column model_size__parameters (changed metadata) - - Owen, David. (2023). Large Language Model performance and compute, Epoch (2023) [Data set]. In Extrapolating performance in language modeling benchmarks. Published online at epoch.ai. Retrieved from: 'https://epoch.ai/blog/extrapolating-performance-in-language-modelling-benchmarks' . ? ^^^^ ^^^ + + Owen, David. (2023). Large Language Model performance and compute, Epoch (2023) [Data set]. In Extrapolating performance in language modeling benchmarks. Published online at epochai.org. Retrieved from: 'https://epochai.org/blog/extrapolating-performance-in-language-modelling-benchmarks' . ? ^^^^^^^ ^^^^^^ - - url_main: https://epoch.ai/blog/extrapolating-performance-in-language-modelling-benchmarks ? - + + url_main: https://epochai.org/blog/extrapolating-performance-in-language-modelling-benchmarks ? ++++ - - url: https://epoch.ai/blog/extrapolating-performance-in-language-modelling-benchmarks ? - + + url: https://epochai.org/blog/extrapolating-performance-in-language-modelling-benchmarks ? ++++ ~ Column organisation (changed metadata) - - Owen, David. (2023). Large Language Model performance and compute, Epoch (2023) [Data set]. In Extrapolating performance in language modeling benchmarks. Published online at epoch.ai. Retrieved from: 'https://epoch.ai/blog/extrapolating-performance-in-language-modelling-benchmarks' . ? ^^^^ ^^^ + + Owen, David. (2023). Large Language Model performance and compute, Epoch (2023) [Data set]. In Extrapolating performance in language modeling benchmarks. Published online at epochai.org. Retrieved from: 'https://epochai.org/blog/extrapolating-performance-in-language-modelling-benchmarks' . ? ^^^^^^^ ^^^^^^ - - url_main: https://epoch.ai/blog/extrapolating-performance-in-language-modelling-benchmarks ? - + + url_main: https://epochai.org/blog/extrapolating-performance-in-language-modelling-benchmarks ? ++++ - - url: https://epoch.ai/blog/extrapolating-performance-in-language-modelling-benchmarks ? - + + url: https://epochai.org/blog/extrapolating-performance-in-language-modelling-benchmarks ? ++++ ~ Column training_computation_petaflop (changed metadata) - - Owen, David. (2023). Large Language Model performance and compute, Epoch (2023) [Data set]. In Extrapolating performance in language modeling benchmarks. Published online at epoch.ai. Retrieved from: 'https://epoch.ai/blog/extrapolating-performance-in-language-modelling-benchmarks' . ? ^^^^ ^^^ + + Owen, David. (2023). Large Language Model performance and compute, Epoch (2023) [Data set]. In Extrapolating performance in language modeling benchmarks. Published online at epochai.org. Retrieved from: 'https://epochai.org/blog/extrapolating-performance-in-language-modelling-benchmarks' . ? ^^^^^^^ ^^^^^^ - - url_main: https://epoch.ai/blog/extrapolating-performance-in-language-modelling-benchmarks ? - + + url_main: https://epochai.org/blog/extrapolating-performance-in-language-modelling-benchmarks ? ++++ - - url: https://epoch.ai/blog/extrapolating-performance-in-language-modelling-benchmarks ? - + + url: https://epochai.org/blog/extrapolating-performance-in-language-modelling-benchmarks ? ++++ = Dataset garden/artificial_intelligence/2024-06-06/epoch_compute_cost = Table epoch_compute_cost ~ Column cost__inflation_adjusted (changed metadata) - - url_main: https://epoch.ai/blog/how-much-does-it-cost-to-train-frontier-ai-models ? - + + url_main: https://epochai.org/blog/how-much-does-it-cost-to-train-frontier-ai-models ? ++++ ~ Column domain (changed metadata) - - url_main: https://epoch.ai/blog/how-much-does-it-cost-to-train-frontier-ai-models ? - + + url_main: https://epochai.org/blog/how-much-does-it-cost-to-train-frontier-ai-models ? ++++ ~ Column publication_date (changed metadata) - - url_main: https://epoch.ai/blog/how-much-does-it-cost-to-train-frontier-ai-models ? - + + url_main: https://epochai.org/blog/how-much-does-it-cost-to-train-frontier-ai-models ? ++++ = Dataset garden/artificial_intelligence/2024-11-03/epoch = Table epoch ~ Column domain (changed metadata) - - Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epoch.ai/data/epochdb/visualization’ ? - + + Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epochai.org/data/epochdb/visualization’ ? ++++ - - url_main: https://epoch.ai/mlinputs/visualization ? - + + url_main: https://epochai.org/mlinputs/visualization ? ++++ - - url_download: https://epoch.ai/data/epochdb/notable_ai_models.csv ? - + + url_download: https://epochai.org/data/epochdb/notable_ai_models.csv ? ++++ ~ Column organization_categorization (changed metadata) - - Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epoch.ai/data/epochdb/visualization’ ? - + + Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epochai.org/data/epochdb/visualization’ ? ++++ - - url_main: https://epoch.ai/mlinputs/visualization ? - + + url_main: https://epochai.org/mlinputs/visualization ? ++++ - - url_download: https://epoch.ai/data/epochdb/notable_ai_models.csv ? - + + url_download: https://epochai.org/data/epochdb/notable_ai_models.csv ? ++++ ~ Column parameters (changed metadata) - - Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epoch.ai/data/epochdb/visualization’ ? - + + Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epochai.org/data/epochdb/visualization’ ? ++++ - - url_main: https://epoch.ai/mlinputs/visualization ? - + + url_main: https://epochai.org/mlinputs/visualization ? ++++ - - url_download: https://epoch.ai/data/epochdb/notable_ai_models.csv ? - + + url_download: https://epochai.org/data/epochdb/notable_ai_models.csv ? ++++ ~ Column publication_date (changed metadata) - - Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epoch.ai/data/epochdb/visualization’ ? - + + Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epochai.org/data/epochdb/visualization’ ? ++++ - - url_main: https://epoch.ai/mlinputs/visualization ? - + + url_main: https://epochai.org/mlinputs/visualization ? ++++ - - url_download: https://epoch.ai/data/epochdb/notable_ai_models.csv ? - + + url_download: https://epochai.org/data/epochdb/notable_ai_models.csv ? ++++ ~ Column training_computation_petaflop (changed metadata) - - Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epoch.ai/data/epochdb/visualization’ ? - + + Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epochai.org/data/epochdb/visualization’ ? ++++ - - url_main: https://epoch.ai/mlinputs/visualization ? - + + url_main: https://epochai.org/mlinputs/visualization ? ++++ - - url_download: https://epoch.ai/data/epochdb/notable_ai_models.csv ? - + + url_download: https://epochai.org/data/epochdb/notable_ai_models.csv ? ++++ ~ Column training_dataset_size__datapoints (changed metadata) - - Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epoch.ai/data/epochdb/visualization’ ? - + + Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epochai.org/data/epochdb/visualization’ ? ++++ - - url_main: https://epoch.ai/mlinputs/visualization ? - + + url_main: https://epochai.org/mlinputs/visualization ? ++++ - - url_download: https://epoch.ai/data/epochdb/notable_ai_models.csv ? - + + url_download: https://epochai.org/data/epochdb/notable_ai_models.csv ? ++++ = Dataset garden/artificial_intelligence/2024-11-03/epoch_aggregates_affiliation = Table epoch_aggregates_affiliation ~ Column cumulative_count (changed metadata) - - Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epoch.ai/data/epochdb/visualization’ ? - + + Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epochai.org/data/epochdb/visualization’ ? ++++ - - url_main: https://epoch.ai/mlinputs/visualization ? - + + url_main: https://epochai.org/mlinputs/visualization ? ++++ - - url_download: https://epoch.ai/data/epochdb/notable_ai_models.csv ? - + + url_download: https://epochai.org/data/epochdb/notable_ai_models.csv ? ++++ ~ Column yearly_count (changed metadata) - - Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epoch.ai/data/epochdb/visualization’ ? - + + Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epochai.org/data/epochdb/visualization’ ? ++++ - - url_main: https://epoch.ai/mlinputs/visualization ? - + + url_main: https://epochai.org/mlinputs/visualization ? ++++ - - url_download: https://epoch.ai/data/epochdb/notable_ai_models.csv ? - + + url_download: https://epochai.org/data/epochdb/notable_ai_models.csv ? ++++ = Dataset garden/artificial_intelligence/2024-11-03/epoch_aggregates_domain = Table epoch_aggregates_domain ~ Column cumulative_count (changed metadata) - - Describes the specific area, application, or field in which an AI system is designed to operate. An AI system can operate in more than one domain, thus contributing to the count for multiple domains. The 2024 data is incomplete and was last updated 03 November 2024. ? ^^ + + Describes the specific area, application, or field in which an AI system is designed to operate. An AI system can operate in more than one domain, thus contributing to the count for multiple domains. The 2024 data is incomplete and was last updated 6 November 2024. ? ^ - - Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epoch.ai/data/epochdb/visualization’ ? - + + Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epochai.org/data/epochdb/visualization’ ? ++++ - - url_main: https://epoch.ai/mlinputs/visualization ? - + + url_main: https://epochai.org/mlinputs/visualization ? ++++ - - url_download: https://epoch.ai/data/epochdb/notable_ai_models.csv ? - + + url_download: https://epochai.org/data/epochdb/notable_ai_models.csv ? ++++ ~ Column yearly_count (changed metadata) - - Describes the specific area, application, or field in which an AI system is designed to operate. An AI system can operate in more than one domain, thus contributing to the count for multiple domains. The 2024 data is incomplete and was last updated 03 November 2024. ? ^^ + + Describes the specific area, application, or field in which an AI system is designed to operate. An AI system can operate in more than one domain, thus contributing to the count for multiple domains. The 2024 data is incomplete and was last updated 6 November 2024. ? ^ - - Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epoch.ai/data/epochdb/visualization’ ? - + + Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epochai.org/data/epochdb/visualization’ ? ++++ - - url_main: https://epoch.ai/mlinputs/visualization ? - + + url_main: https://epochai.org/mlinputs/visualization ? ++++ - - url_download: https://epoch.ai/data/epochdb/notable_ai_models.csv ? - + + url_download: https://epochai.org/data/epochdb/notable_ai_models.csv ? ++++ = Dataset garden/artificial_intelligence/2024-11-03/epoch_compute_intensive = Table epoch_compute_intensive ~ Column domain (changed metadata) - - Robi Rahman, David Owen and Josh You (2024), "Tracking Compute-Intensive AI Models". Published online at epochai.org. Retrieved from: 'https://epoch.ai/blog/tracking-compute-intensive-ai-models' ? ^^^ + + Robi Rahman, David Owen and Josh You (2024), "Tracking Compute-Intensive AI Models". Published online at epochai.org. Retrieved from: 'https://epochai.org/blog/tracking-compute-intensive-ai-models' ? ^^^^^^ - - url_main: https://epoch.ai/blog/tracking-compute-intensive-ai-models ? - + + url_main: https://epochai.org/blog/tracking-compute-intensive-ai-models ? ++++ - - url_download: https://epoch.ai/data/epochdb/large_scale_ai_models.csv ? - + + url_download: https://epochai.org/data/epochdb/large_scale_ai_models.csv ? ++++ - - url: https://epoch.ai/blog/how-much-does-it-cost-to-train-frontier-ai-models ? - + + url: https://epochai.org/blog/how-much-does-it-cost-to-train-frontier-ai-models ? ++++ ~ Column parameters (changed metadata) - - Robi Rahman, David Owen and Josh You (2024), "Tracking Compute-Intensive AI Models". Published online at epochai.org. Retrieved from: 'https://epoch.ai/blog/tracking-compute-intensive-ai-models' ? ^^^ + + Robi Rahman, David Owen and Josh You (2024), "Tracking Compute-Intensive AI Models". Published online at epochai.org. Retrieved from: 'https://epochai.org/blog/tracking-compute-intensive-ai-models' ? ^^^^^^ - - url_main: https://epoch.ai/blog/tracking-compute-intensive-ai-models ? - + + url_main: https://epochai.org/blog/tracking-compute-intensive-ai-models ? ++++ - - url_download: https://epoch.ai/data/epochdb/large_scale_ai_models.csv ? - + + url_download: https://epochai.org/data/epochdb/large_scale_ai_models.csv ? ++++ - - url: https://epoch.ai/blog/how-much-does-it-cost-to-train-frontier-ai-models ? - + + url: https://epochai.org/blog/how-much-does-it-cost-to-train-frontier-ai-models ? ++++ ~ Column publication_date (changed metadata) - - Robi Rahman, David Owen and Josh You (2024), "Tracking Compute-Intensive AI Models". Published online at epochai.org. Retrieved from: 'https://epoch.ai/blog/tracking-compute-intensive-ai-models' ? ^^^ + + Robi Rahman, David Owen and Josh You (2024), "Tracking Compute-Intensive AI Models". Published online at epochai.org. Retrieved from: 'https://epochai.org/blog/tracking-compute-intensive-ai-models' ? ^^^^^^ - - url_main: https://epoch.ai/blog/tracking-compute-intensive-ai-models ? - + + url_main: https://epochai.org/blog/tracking-compute-intensive-ai-models ? ++++ - - url_download: https://epoch.ai/data/epochdb/large_scale_ai_models.csv ? - + + url_download: https://epochai.org/data/epochdb/large_scale_ai_models.csv ? ++++ - - url: https://epoch.ai/blog/how-much-does-it-cost-to-train-frontier-ai-models ? - + + url: https://epochai.org/blog/how-much-does-it-cost-to-train-frontier-ai-models ? ++++ ~ Column training_computation_petaflop (changed metadata) - - Robi Rahman, David Owen and Josh You (2024), "Tracking Compute-Intensive AI Models". Published online at epochai.org. Retrieved from: 'https://epoch.ai/blog/tracking-compute-intensive-ai-models' ? ^^^ + + Robi Rahman, David Owen and Josh You (2024), "Tracking Compute-Intensive AI Models". Published online at epochai.org. Retrieved from: 'https://epochai.org/blog/tracking-compute-intensive-ai-models' ? ^^^^^^ - - url_main: https://epoch.ai/blog/tracking-compute-intensive-ai-models ? - + + url_main: https://epochai.org/blog/tracking-compute-intensive-ai-models ? ++++ - - url_download: https://epoch.ai/data/epochdb/large_scale_ai_models.csv ? - + + url_download: https://epochai.org/data/epochdb/large_scale_ai_models.csv ? ++++ - - url: https://epoch.ai/blog/how-much-does-it-cost-to-train-frontier-ai-models ? - + + url: https://epochai.org/blog/how-much-does-it-cost-to-train-frontier-ai-models ? ++++ ~ Column training_dataset_size__datapoints (changed metadata) - - Robi Rahman, David Owen and Josh You (2024), "Tracking Compute-Intensive AI Models". Published online at epochai.org. Retrieved from: 'https://epoch.ai/blog/tracking-compute-intensive-ai-models' ? ^^^ + + Robi Rahman, David Owen and Josh You (2024), "Tracking Compute-Intensive AI Models". Published online at epochai.org. Retrieved from: 'https://epochai.org/blog/tracking-compute-intensive-ai-models' ? ^^^^^^ - - url_main: https://epoch.ai/blog/tracking-compute-intensive-ai-models ? - + + url_main: https://epochai.org/blog/tracking-compute-intensive-ai-models ? ++++ - - url_download: https://epoch.ai/data/epochdb/large_scale_ai_models.csv ? - + + url_download: https://epochai.org/data/epochdb/large_scale_ai_models.csv ? ++++ - - url: https://epoch.ai/blog/how-much-does-it-cost-to-train-frontier-ai-models ? - + + url: https://epochai.org/blog/how-much-does-it-cost-to-train-frontier-ai-models ? ++++ = Dataset garden/artificial_intelligence/2024-11-03/epoch_compute_intensive_countries = Table epoch_compute_intensive_countries ~ Column cumulative_count (changed metadata) - - Refers to the location of the primary organization with which the authors of a large-scale AI systems are affiliated. The 2024 data is incomplete and was last updated 03 November 2024. ? ^^ + + Refers to the location of the primary organization with which the authors of a large-scale AI systems are affiliated. The 2024 data is incomplete and was last updated 6 November 2024. ? ^ - - Robi Rahman, David Owen and Josh You (2024), "Tracking Compute-Intensive AI Models". Published online at epochai.org. Retrieved from: 'https://epoch.ai/blog/tracking-compute-intensive-ai-models' ? ^^^ + + Robi Rahman, David Owen and Josh You (2024), "Tracking Compute-Intensive AI Models". Published online at epochai.org. Retrieved from: 'https://epochai.org/blog/tracking-compute-intensive-ai-models' ? ^^^^^^ - - url_main: https://epoch.ai/blog/tracking-compute-intensive-ai-models ? - + + url_main: https://epochai.org/blog/tracking-compute-intensive-ai-models ? ++++ - - url_download: https://epoch.ai/data/epochdb/large_scale_ai_models.csv ? - + + url_download: https://epochai.org/data/epochdb/large_scale_ai_models.csv ? ++++ - - url: https://epoch.ai/blog/how-much-does-it-cost-to-train-frontier-ai-models ? - + + url: https://epochai.org/blog/how-much-does-it-cost-to-train-frontier-ai-models ? ++++ ~ Column yearly_count (changed metadata) - - Refers to the location of the primary organization with which the authors of a large-scale AI systems are affiliated. The 2024 data is incomplete and was last updated 03 November 2024. ? ^^ + + Refers to the location of the primary organization with which the authors of a large-scale AI systems are affiliated. The 2024 data is incomplete and was last updated 6 November 2024. ? ^ - - Robi Rahman, David Owen and Josh You (2024), "Tracking Compute-Intensive AI Models". Published online at epochai.org. Retrieved from: 'https://epoch.ai/blog/tracking-compute-intensive-ai-models' ? ^^^ + + Robi Rahman, David Owen and Josh You (2024), "Tracking Compute-Intensive AI Models". Published online at epochai.org. Retrieved from: 'https://epochai.org/blog/tracking-compute-intensive-ai-models' ? ^^^^^^ - - url_main: https://epoch.ai/blog/tracking-compute-intensive-ai-models ? - + + url_main: https://epochai.org/blog/tracking-compute-intensive-ai-models ? ++++ - - url_download: https://epoch.ai/data/epochdb/large_scale_ai_models.csv ? - + + url_download: https://epochai.org/data/epochdb/large_scale_ai_models.csv ? ++++ - - url: https://epoch.ai/blog/how-much-does-it-cost-to-train-frontier-ai-models ? - + + url: https://epochai.org/blog/how-much-does-it-cost-to-train-frontier-ai-models ? ++++ = Dataset garden/artificial_intelligence/2024-11-03/epoch_compute_intensive_domain = Table epoch_compute_intensive_domain ~ Column cumulative_count (changed metadata) - - Describes the specific area, application, or field in which a large-scale AI model is designed to operate. The 2024 data is incomplete and was last updated 03 November 2024. ? ^^ + + Describes the specific area, application, or field in which a large-scale AI model is designed to operate. The 2024 data is incomplete and was last updated 6 November 2024. ? ^ - - Robi Rahman, David Owen and Josh You (2024), "Tracking Compute-Intensive AI Models". Published online at epochai.org. Retrieved from: 'https://epoch.ai/blog/tracking-compute-intensive-ai-models' ? ^^^ + + Robi Rahman, David Owen and Josh You (2024), "Tracking Compute-Intensive AI Models". Published online at epochai.org. Retrieved from: 'https://epochai.org/blog/tracking-compute-intensive-ai-models' ? ^^^^^^ - - url_main: https://epoch.ai/blog/tracking-compute-intensive-ai-models ? - + + url_main: https://epochai.org/blog/tracking-compute-intensive-ai-models ? ++++ - - url_download: https://epoch.ai/data/epochdb/large_scale_ai_models.csv ? - + + url_download: https://epochai.org/data/epochdb/large_scale_ai_models.csv ? ++++ - - url: https://epoch.ai/blog/how-much-does-it-cost-to-train-frontier-ai-models ? - + + url: https://epochai.org/blog/how-much-does-it-cost-to-train-frontier-ai-models ? ++++ ~ Column yearly_count (changed metadata) - - Describes the specific area, application, or field in which a large-scale AI model is designed to operate. The 2024 data is incomplete and was last updated 03 November 2024. ? ^^ + + Describes the specific area, application, or field in which a large-scale AI model is designed to operate. The 2024 data is incomplete and was last updated 6 November 2024. ? ^ - - Robi Rahman, David Owen and Josh You (2024), "Tracking Compute-Intensive AI Models". Published online at epochai.org. Retrieved from: 'https://epoch.ai/blog/tracking-compute-intensive-ai-models' ? ^^^ + + Robi Rahman, David Owen and Josh You (2024), "Tracking Compute-Intensive AI Models". Published online at epochai.org. Retrieved from: 'https://epochai.org/blog/tracking-compute-intensive-ai-models' ? ^^^^^^ - - url_main: https://epoch.ai/blog/tracking-compute-intensive-ai-models ? - + + url_main: https://epochai.org/blog/tracking-compute-intensive-ai-models ? ++++ - - url_download: https://epoch.ai/data/epochdb/large_scale_ai_models.csv ? - + + url_download: https://epochai.org/data/epochdb/large_scale_ai_models.csv ? ++++ - - url: https://epoch.ai/blog/how-much-does-it-cost-to-train-frontier-ai-models ? - + + url: https://epochai.org/blog/how-much-does-it-cost-to-train-frontier-ai-models ? ++++ = Dataset garden/artificial_intelligence/2024-11-03/epoch_regressions = Table epoch_regressions ~ Column domain (changed metadata) - - Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epoch.ai/data/epochdb/visualization’ ? - + + Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epochai.org/data/epochdb/visualization’ ? ++++ - - url_main: https://epoch.ai/mlinputs/visualization ? - + + url_main: https://epochai.org/mlinputs/visualization ? ++++ - - url_download: https://epoch.ai/data/epochdb/notable_ai_models.csv ? - + + url_download: https://epochai.org/data/epochdb/notable_ai_models.csv ? ++++ ~ Column organization_categorization (changed metadata) - - Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epoch.ai/data/epochdb/visualization’ ? - + + Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epochai.org/data/epochdb/visualization’ ? ++++ - - url_main: https://epoch.ai/mlinputs/visualization ? - + + url_main: https://epochai.org/mlinputs/visualization ? ++++ - - url_download: https://epoch.ai/data/epochdb/notable_ai_models.csv ? - + + url_download: https://epochai.org/data/epochdb/notable_ai_models.csv ? ++++ ~ Column parameters (changed metadata) - - Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epoch.ai/data/epochdb/visualization’ ? - + + Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epochai.org/data/epochdb/visualization’ ? ++++ - - url_main: https://epoch.ai/mlinputs/visualization ? - + + url_main: https://epochai.org/mlinputs/visualization ? ++++ - - url_download: https://epoch.ai/data/epochdb/notable_ai_models.csv ? - + + url_download: https://epochai.org/data/epochdb/notable_ai_models.csv ? ++++ ~ Column publication_date (changed metadata) - - Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epoch.ai/data/epochdb/visualization’ ? - + + Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epochai.org/data/epochdb/visualization’ ? ++++ - - url_main: https://epoch.ai/mlinputs/visualization ? - + + url_main: https://epochai.org/mlinputs/visualization ? ++++ - - url_download: https://epoch.ai/data/epochdb/notable_ai_models.csv ? - + + url_download: https://epochai.org/data/epochdb/notable_ai_models.csv ? ++++ ~ Column training_computation_petaflop (changed metadata) - - Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epoch.ai/data/epochdb/visualization’ ? - + + Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epochai.org/data/epochdb/visualization’ ? ++++ - - url_main: https://epoch.ai/mlinputs/visualization ? - + + url_main: https://epochai.org/mlinputs/visualization ? ++++ - - url_download: https://epoch.ai/data/epochdb/notable_ai_models.csv ? - + + url_download: https://epochai.org/data/epochdb/notable_ai_models.csv ? ++++ ~ Column training_dataset_size__datapoints (changed metadata) - - Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epoch.ai/data/epochdb/visualization’ ? - + + Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epochai.org/data/epochdb/visualization’ ? ++++ - - url_main: https://epoch.ai/mlinputs/visualization ? - + + url_main: https://epochai.org/mlinputs/visualization ? ++++ - - url_download: https://epoch.ai/data/epochdb/notable_ai_models.csv ? - + + url_download: https://epochai.org/data/epochdb/notable_ai_models.csv ? ++++ = Dataset garden/education/2023-07-17/education_barro_lee_projections = Table education_barro_lee_projections ~ Dim country - - Removed values: 144 / 11802 (1.22%) year country 1994 Africa 1988 Asia 2007 North America 1987 Oceania 2003 Oceania ~ Dim year - - Removed values: 144 / 11802 (1.22%) country year Africa 1994 Asia 1988 North America 2007 Oceania 1987 Oceania 2003 = Dataset garden/education/2023-07-17/education_lee_lee = Table education_lee_lee ~ Dim country - - Removed values: 192 / 12737 (1.51%) year country 1998 Europe 1987 North America 2003 North America 2021 North America 1989 South America ~ Dim year - - Removed values: 192 / 12737 (1.51%) country year Europe 1998 North America 1987 North America 2003 North America 2021 South America 1989 = Dataset garden/education/2023-08-14/oecd_education = Table oecd_education ~ Dim country + + New values: 363 / 7238 (5.02%) year country 2009 Cayman Islands 2012 Curacao 2015 Netherlands Antilles 2007 South Sudan 2021 South Sudan - - Removed values: 125 / 7238 (1.73%) year country 2023 Barbados 2023 Djibouti 2023 Iran 2023 Kuwait 2023 Mauritius ~ Dim year + + New values: 363 / 7238 (5.02%) country year Cayman Islands 2009 Curacao 2012 Netherlands Antilles 2015 South Sudan 2007 South Sudan 2021 - - Removed values: 125 / 7238 (1.73%) country year Barbados 2023 Djibouti 2023 Iran 2023 Kuwait 2023 Mauritius 2023 ~ Column illiterate (changed metadata, new data, changed data) - - description_short: Percentage of people aged 15 and above who can read and write a short, simple statement on their everyday - - life. - - url_main: https://datacatalog.worldbank.org/search/dataset/0038480/education-statistics + + version_producer: '2023' + + url_main: https://databank.worldbank.org/reports.aspx?source=Education%20Statistics - - date_accessed: '2024-11-04' ? ^^^^ ^ + + date_accessed: '2023-07-26' ? ^ ^^^^ - - date_published: '2024-10-01' ? ^^^^^^^ + + date_published: '2023' ? ^ + + New values: 363 / 7238 (5.02%) country year illiterate Cayman Islands 2009 NaN Curacao 2012 NaN Netherlands Antilles 2015 NaN South Sudan 2007 NaN South Sudan 2021 NaN - - Removed values: 125 / 7238 (1.73%) country year illiterate Barbados 2023 Djibouti 2023 Iran 2023 Kuwait 2023 Mauritius 2023 ~ Changed values: 815 / 7238 (11.26%) country year illiterate - illiterate + Arab World (WB) 2022 24.181541 24.828468 Bangladesh 2021 24.0 NaN El Salvador 2009 16.0 15.897110 Pakistan 2010 45.0 44.624809 South Asia (WB) 2019 28.033974 28.057213 ~ Column literacy (changed metadata, new data, changed data) - - description_short: Percentage of people aged 15 and above who can read and write a short, simple statement on their everyday - - life. - - url_main: https://datacatalog.worldbank.org/search/dataset/0038480/education-statistics + + version_producer: '2023' + + url_main: https://databank.worldbank.org/reports.aspx?source=Education%20Statistics - - date_accessed: '2024-11-04' ? ^^^^ ^ + + date_accessed: '2023-07-26' ? ^ ^^^^ - - date_published: '2024-10-01' ? ^^^^^^^ + + date_published: '2023' ? ^ + + New values: 363 / 7238 (5.02%) country year literacy Cayman Islands 2009 NaN Curacao 2012 NaN Netherlands Antilles 2015 NaN South Sudan 2007 NaN South Sudan 2021 NaN - - Removed values: 125 / 7238 (1.73%) country year literacy Barbados 2023 Djibouti 2023 Iran 2023 Kuwait 2023 Mauritius 2023 ~ Changed values: 811 / 7238 (11.20%) country year literacy - literacy + Arab World (WB) 2022 75.818459 75.171532 Bangladesh 2021 76.0 NaN El Salvador 2011 85.0 85.493988 Pakistan 2011 55.0 54.738022 South Asia (WB) 2018 72.192062 72.081512 ~ Column no_formal_education (changed metadata, new data, changed data) - - description_short: Percentage of the population aged 15 and older with no formal education. - - url_main: https://datacatalog.worldbank.org/search/dataset/0038480/education-statistics + + version_producer: '2023' + + url_main: https://databank.worldbank.org/reports.aspx?source=Education%20Statistics - - date_accessed: '2024-11-04' ? ^^^^ ^ + + date_accessed: '2023-07-26' ? ^ ^^^^ - - date_published: '2024-10-01' ? ^^^^^^^ + + date_published: '2023' ? ^ + + New values: 363 / 7238 (5.02%) country year no_formal_education Cayman Islands 2009 NaN Curacao 2012 NaN Netherlands Antilles 2015 NaN South Sudan 2007 NaN South Sudan 2021 NaN - - Removed values: 125 / 7238 (1.73%) country year no_formal_education Barbados 2023 Djibouti 2023 Iran 2023 Kuwait 2023 Mauritius 2023 ~ Changed values: 1688 / 7238 (23.32%) country year no_formal_education - no_formal_education + Aruba 2035 0.03 3.0 Guinea 2040 0.39 39.0 India 2100 0.02 2.0 Luxembourg 2045 0.02 2.0 North Macedonia 2020 0.02 2.0 ~ Column population_with_basic_education (changed metadata, new data, changed data) - - description_short: Percentage of the population aged 15 and older with no formal education. - - url_main: https://datacatalog.worldbank.org/search/dataset/0038480/education-statistics + + version_producer: '2023' + + url_main: https://databank.worldbank.org/reports.aspx?source=Education%20Statistics - - date_accessed: '2024-11-04' ? ^^^^ ^ + + date_accessed: '2023-07-26' ? ^ ^^^^ - - date_published: '2024-10-01' ? ^^^^^^^ + + date_published: '2023' ? ^ + + New values: 363 / 7238 (5.02%) country year population_with_basic_education Cayman Islands 2009 NaN Curacao 2012 NaN Netherlands Antilles 2015 NaN South Sudan 2007 NaN South Sudan 2021 NaN - - Removed values: 125 / 7238 (1.73%) country year population_with_basic_education Barbados 2023 Djibouti 2023 Iran 2023 Kuwait 2023 Mauritius 2023 ~ Changed values: 1688 / 7238 (23.32%) country year population_with_basic_education - population_with_basic_education + Aruba 2035 99.970001 97.0 Guinea 2040 99.610001 61.0 India 2100 99.980003 98.0 Luxembourg 2045 99.980003 98.0 North Macedonia 2020 99.980003 98.0 = Dataset garden/education/2023-12-15/wittgenstein_center = Table wittgenstein_center_data ~ Dim country + + New values: 285 / 7307 (3.90%) year country 2012 Faroe Islands 2019 Guam 2007 North Korea 2017 North Korea 2008 Puerto Rico ~ Dim year + + New values: 285 / 7307 (3.90%) country year Faroe Islands 2012 Guam 2019 North Korea 2007 North Korea 2017 Puerto Rico 2008 ~ Column no_formal_education (changed metadata, new data, changed data) - - url_main: https://datacatalog.worldbank.org/search/dataset/0038480/education-statistics + + version_producer: '2023' + + url_main: https://databank.worldbank.org/reports.aspx?source=Education%20Statistics - - date_accessed: '2024-11-04' ? ^^^^ ^ + + date_accessed: '2023-07-26' ? ^ ^^^^ - - date_published: '2024-10-01' ? ^^^^^^^ + + date_published: '2023' ? ^ + + New values: 285 / 7307 (3.90%) country year no_formal_education Faroe Islands 2012 NaN Guam 2019 NaN North Korea 2007 NaN North Korea 2017 NaN Puerto Rico 2008 NaN ~ Changed values: 281 / 7307 (3.85%) country year no_formal_education - no_formal_education + Azerbaijan 2010 0.01 1.000000 Cote d'Ivoire 2010 0.46 46.000000 Ethiopia 2015 0.43 43.000000 Iran 2015 0.15 15.000001 Sao Tome and Principe 2010 0.09 9.000000 ~ Column population_with_basic_education (changed metadata, new data, changed data) - - url_main: https://datacatalog.worldbank.org/search/dataset/0038480/education-statistics + + version_producer: '2023' + + url_main: https://databank.worldbank.org/reports.aspx?source=Education%20Statistics - - date_accessed: '2024-11-04' ? ^^^^ ^ + + date_accessed: '2023-07-26' ? ^ ^^^^ - - date_published: '2024-10-01' ? ^^^^^^^ + + date_published: '2023' ? ^ + + New values: 285 / 7307 (3.90%) country year population_with_basic_education Faroe Islands 2012 NaN Guam 2019 NaN North Korea 2007 NaN North Korea 2017 NaN Puerto Rico 2008 NaN ~ Changed values: 281 / 7307 (3.85%) country year population_with_basic_education - population_with_basic_education + Azerbaijan 2010 99.989998 99.0 Cote d'Ivoire 2010 99.540001 54.0 Ethiopia 2015 99.57 57.0 Iran 2015 99.849998 85.0 Sao Tome and Principe 2010 99.910004 91.0 = Dataset garden/faostat/2024-03-14/faostat_fa = Table faostat_fa_flat = Table faostat_fa 2024-11-21 13:39:34 [warning ] DataFrame is highly fragmented. This is usually the result of calling `frame.insert` many times, which has poor performance. Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()` category=PerformanceWarning filename=/home/owid/etl/lib/catalog/owid/catalog/tables.py lineno=405 2024-11-21 13:40:08 [warning ] DataFrame is highly fragmented. This is usually the result of calling `frame.insert` many times, which has poor performance. Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()` category=PerformanceWarning filename=/home/owid/etl/lib/catalog/owid/catalog/tables.py lineno=405 2024-11-21 13:40:50 [warning ] DataFrame is highly fragmented. This is usually the result of calling `frame.in ...diff too long, truncated... ``` Automatically updated datasets matching _weekly_wildfires|excess_mortality|covid|fluid|flunet|country_profile|garden/ihme_gbd/2019/gbd_risk_ are not included

Edited: 2024-11-21 14:05:57 UTC Execution time: 4.71 seconds

antea04 commented 1 day ago

Thanks for the review, @paarriagadap! I've addressed your comments, I'll go ahead and merge ☺️