ecwood / GCAM-CDR-modeling

1 stars 0 forks source link

Learn How to Decrease the Number of Running Regions in GCAM #18

Closed ecwood closed 1 year ago

ecwood commented 1 year ago

Simply commenting out a line in gcam-core/input/gcamdata/inst/extdata/common/GCAM_region_names.csv did not work

ecwood commented 1 year ago

Files that contain GCAM_region_ID:

aglu/A_bio_frac_prod_R.csv:GCAM_region_ID,ag,For,Mill
aglu/A_soil_time_scale_R.csv:GCAM_region_ID,soilTimeScale
aglu/LDS/L123.LC_bm2_R_MgdFor_Yh_GLU_beforeadjust.csv:GCAM_region_ID,Land_Type,GLU,year,MgdFor
common/GCAM_region_names.csv:GCAM_region_ID,region
common/iso_GCAM_regID.csv:iso,country_name,region_GCAM3,GCAM_region_ID
emissions/EPA_MAC_missing_region.csv:GCAM_region_ID_missing,GCAM_region_ID_alternative
emissions/A_regions.csv:GCAM_region_ID,region,MAC_region,bio_N2O_coef,SO2_name,GAINS_region
emissions/EPA_country_map.csv:EPA_country,iso,GCAM_region_ID
energy/A_regions.csv:GCAM_region_ID,region,tradbio_region,ethanol,biodiesel,biomassOil_tech,elect_td_techchange,has_district_heat,region.class
energy/A62.calibration.csv:GCAM_region_ID,sector,year,value
energy/offshore_wind_potential_scaler.csv:GCAM_region_ID,scaler,reason
energy/IO_IRONSTL_scaled.csv:region,GCAM_region_ID,sector.name,subsector.name,technology,year,minicam.energy.input,coefficient
energy/aluminum_prod.csv:region,flow,GCAM_region_ID,year,unit_prod,value
energy/construction_feedstock.csv:region,GCAM_region_ID,flow,fuel,value,year,unit
water/A_unlimited_water_price.csv:GCAM_region_ID,water_type,X1971,X2010,X2100
ecwood commented 1 year ago

I commented out any lines with a GCAM_region_ID of 32 in those files and got this error when running make xml using driver_drake:

cd input/gcamdata && Rscript -e "devtools::load_all('.')" -e "driver_drake(write_xml=TRUE)"
Loading gcamdata
Loading required namespace: drake
GCAM Data System v5.1
Found 420 chunks
Found 4265 chunk data requirements
Found 2416 chunk data products
1452 chunk data input(s) not accounted for
▶ target common.iso_GCAM_regID
▶ target energy.A_regions
▶ target module_water_L102.water_supply_unlimited
▶ target emissions.A_regions
▶ target emissions.EPA_country_map
▶ target energy.offshore_wind_potential_scaler
▶ target aglu.A_soil_time_scale_R
▶ target module_energy_LA112.U
▶ target aglu.LDS.L123.LC_bm2_R_MgdFor_Yh_GLU_beforeadjust
▶ target energy.A62.calibration
▶ target aglu.A_bio_frac_prod_R
▶ target module_energy_LA161.Cstorage
✖ fail module_energy_LA161.Cstorage
Error: target module_energy_LA161.Cstorage failed.
diagnose(module_energy_LA161.Cstorage)$error$message:
  left_join_no_match: NA values in new data columns
diagnose(module_energy_LA161.Cstorage)$error$calls:
  gcamdata:::module_energy_LA161.Cstorage("MAKE", c(common.iso_GCAM_regID, 
    aglu.LDS.Land_type_area_ha, energy.A61.Cstorage_curves, energy.Dooley_Cstorage_RG3_MtCO2))
  Land_type_area_ha %>% filter(year == max(year)) %>% group_by(iso) %>% 
    summarise(value = sum(value * CONV_HA_BM2)) %>% ungroup() %>% 
    left_join_error_no_match(iso_GCAM_regID, by = "iso") %>% 
    group_by(region_GCAM3) %>% mutate(share = value/sum(value)) %>% 
    ungroup()
  ungroup(.)
  mutate(., share = value/sum(value))
  group_by(., region_GCAM3)
  left_join_error_no_match(., iso_GCAM_regID, by = "iso")
  stop("left_join_no_match: NA values in new data columns")
Execution halted
make: *** [Makefile:5: xml] Error 1

and this error when using the regular driver:

cd input/gcamdata && Rscript -e "devtools::load_all('.')" -e "driver(write_xml=TRUE)"
Loading gcamdata
GCAM Data System v5.1
Found 420 chunks
Found 4265 chunk data requirements
Found 2416 chunk data products
1452 chunk data input(s) not accounted for
Chunks left: 1872
[1] "module_aglu_LA100.0_LDS_preprocessing"
[1] "- make 13.30"
[1] "module_aglu_LA100.FAO_downscale_ctry"
[1] "- make 4.56"
[1] "module_aglu_LA100.GTAP_downscale_ctry"
[1] "- make 0.14"
[1] "module_aglu_LA100.IMAGE_downscale_ctry_yr"
[1] "- make 0.82"
[1] "module_aglu_LA101.ag_FAO_R_C_Y"
Error in left_join_error_no_match(., iso_GCAM_regID, by = "iso") : 
  left_join_no_match: NA values in new data columns
Calls: driver ... mutate -> filter -> left_join -> left_join_error_no_match
Execution halted
make: *** [Makefile:5: xml] Error 1
ecwood commented 1 year ago

The input files for that module are:

    return(c(FILE = "common/iso_GCAM_regID",
             FILE = "aglu/LDS/Land_type_area_ha",
             FILE = "energy/A61.Cstorage_curves",
             FILE = "energy/Dooley_Cstorage_RG3_MtCO2"))
ecwood commented 1 year ago

I edited aglu/LDS/Land_type_area_ha.csv to comment out col lines. This was the output after I did that:

cd input/gcamdata && Rscript -e "devtools::load_all('.')" -e "driver_drake(write_xml=TRUE)"
Loading gcamdata
Loading required namespace: drake
GCAM Data System v5.1
Found 420 chunks
Found 4265 chunk data requirements
Found 2416 chunk data products
1452 chunk data input(s) not accounted for
▶ target module_emissions_L152.MACC
✖ fail module_emissions_L152.MACC
Error: target module_emissions_L152.MACC failed.
diagnose(module_emissions_L152.MACC)$error$message:
  left_join_no_match: NA values in new data columns
diagnose(module_emissions_L152.MACC)$error$calls:
  gcamdata:::module_emissions_L152.MACC("MAKE", c(emissions.EPA.EPA_2019_raw, 
    emissions.EPA.EPA_2019_MACC_Ag_updated_baseline, emissions.EPA.EPA_2019_MACC_raw, 
    emissions.EPA_MACC_mapping, emissions.EPA_MACC_control_mapping, 
    emissions.EPA_MAC_missing_region, emissions.EPA_country_map))
  EPA_ag %>% left_join_error_no_match(EPA_MACC_control_mapping, 
    by = "source") %>% left_join_error_no_match(EPA_country_map %>% 
    select(-iso) %>% rename(country = EPA_country), by = "country") %>% 
    group_by(GCAM_region_ID, Sector, Process, year) %>% summarise(value = sum(value)) %>% 
    ungroup() %>% filter(year %in% emissions.EPA_MACC_YEAR)
  filter(., year %in% emissions.EPA_MACC_YEAR)
  ungroup(.)
  summarise(., value = sum(value))
  group_by(., GCAM_region_ID, Sector, Proc
Execution halted
make: *** [Makefile:5: xml] Error 1

I believe it just ran a different file this time.

ecwood commented 1 year ago

Here are the inputs to the script that caused the error on the last run:

module_emissions_L152.MACC <- function(command, ...) {
  if(command == driver.DECLARE_INPUTS) {
    return(c(FILE = "emissions/EPA/EPA_2019_raw",
             FILE = "emissions/EPA/EPA_2019_MACC_Ag_updated_baseline",
             FILE = "emissions/EPA/EPA_2019_MACC_raw",
             FILE = "emissions/EPA_MACC_mapping",
             FILE = "emissions/EPA_MACC_control_mapping",
             FILE = "emissions/EPA_MAC_missing_region",
             FILE = "emissions/EPA_country_map"))
ecwood commented 1 year ago

The column that col was in was called iso. These are the files with iso in the header:

aglu/A_recent_feed_modifications.csv:iso,item,year,feed
aglu/Rohwer_2007_IrrigationEff.csv:country,iso,application.eff,conveyance.eff,large.scale.frac,management.eff
aglu/LDS/Water_footprint_m3.csv:iso,glu_code,SAGE_crop,water_type,value
aglu/LDS/MIRCA_rfdHA_ha.csv:iso,glu_code,mirca_crop,value
aglu/LDS/Land_type_area_ha.csv:iso,glu_code,land_type,year,value
aglu/LDS/Ref_veg_carbon_Mg_per_ha.csv:iso,glu_code,land_type,c_type,weighted_average,median_value,min_value,max_value,q1_value,q3_value
aglu/LDS/MIRCA_irrHA_ha.csv:iso,glu_code,mirca_crop,value
aglu/LDS/LDS_ag_HA_ha.csv:ctry_iso,glu_code,SAGE_crop,value
aglu/LDS/LDS_value_milUSD.csv:reglr_iso,glu_code,use_sector,value
aglu/LDS/LDS_ag_prod_t.csv:ctry_iso,glu_code,SAGE_crop,value
aglu/AGLU_ctry.csv:FAO_country,iso,GTAP_region,CROSIT_ctry,CROSIT_country_ID,MIRCA_country,IFA2002_country,IFA_region,IMAGE_region_ID,IMAGE_region_name,FAO2050_reg
common/iso_GCAM_regID.csv:iso,country_name,region_GCAM3,GCAM_region_ID
emissions/RCP_OC_2000.csv:Country,iso,agr,awb,dom,ene,ind,lcf,sav,slv,tra,wst
emissions/CEDS/GFED-CMIP6_LUC_emissions.csv:em,iso,sector,unit,1940,1941,1942,1943,1944,1945,1946,1947,1948,1949,1950,1951,1952,1953,1954,1955,1956,1957,1958,1959,1960,1961,1962,1963,1964,1965,1966,1967,1968,1969,1970,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015
emissions/CEDS/gains_iso_fuel_emissions.csv:Non.co2,iso,year,dieseloil,lightoil,natural_gas
emissions/CEDS/gains_iso_sector_emissions.csv:Non.co2,iso,year,Freight,Motorcycle,Passenger
emissions/mappings/CDIAC_nation_iso.csv:nation,iso,UN_code
emissions/RCP_BC_2000.csv:Country,iso,agr,awb,dom,ene,ind,lcf,sav,slv,tra,wst
emissions/EPA_country_map.csv:EPA_country,iso,GCAM_region_ID
energy/CEDB_ResFloorspace_chn.csv:country,iso,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015
energy/rsrc_unconv_oil_prod_bbld.csv:iso,1970,1975,1980,1985,1990,1995,2000,2005,2010
energy/mappings/UCD_ctry.csv:iso,country_name,UCD_region
energy/mappings/IEA_ctry.csv:IEA_ctry,iso,IEA_Fert_reg
energy/mappings/cement_regions.csv:iso,Worrell_region,IEA_fuelshare_region,IEA_intensity_region
energy/mappings/IAA_ctry_region.csv:iso,region_aluminum_prod,region_alumina_prod,region_aluminum_en,region_alumina_en
energy/RECS_ResFloorspace_usa.csv:region,iso,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015
energy/IEA_PCResFloorspace.csv:country,iso,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004
energy/Smith_irradiance_ctry_kwh.csv:iso,irradiance,irradiance.area,dni,dni.area
energy/steel_prod.csv:iso,year,BLASTFUR,EAF with scrap,EAF with DRI,unit_prod
energy/Other_pcflsp_m2_ctry_Yh.csv:country,iso,gcam.consumer,2004,2005,source
energy/aluminum_prod_USGS.csv:country,iso,year,value,unit
energy/A23.subsector_shrwt_nuc_R.csv:region_GCAM3,iso,supplysector,subsector,2015,2020,2035,2050
energy/Zhou_wind_supply_ctry_EJ.csv:iso,maxSubResource,mid-price,curve-exponent,base-price
energy/Odyssee_ResFloorspacePerHouse.csv:iso,Unit,Source,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009
energy/Hydropower_potential.csv:Country,iso,Theoretical_MW,Theoretical_GWh,Technical_MW,Technical_GWh,Economic_MW,Economic_GWh,Installed_MW,Installed_GWh
gcam-usa/UCS_Database.csv:514,Morrow Point,CO,514-1,United States Government,HY-NA-HY,HY,No,None,Hydropower,38.4703,-107.8744,14020006,Upper Colorado,1970,86.6,233714,31%,Morrow Point Reservoir-Gunniso,Surface Water,0,0,0,0,0,0,0,0
gcam-usa/UCS_Database.csv:514,Morrow Point,CO,514-2,United States Government,HY-NA-HY,HY,No,None,Hydropower,38.4703,-107.8744,14020006,Upper Colorado,1971,86.6,233714,31%,Morrow Point Reservoir-Gunniso,Surface Water,0,0,0,0,0,0,0,0
socioeconomics/WB_ExtraCountries_GDP_MER.csv:Country,iso,1969,1970,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017
socioeconomics/socioeconomics_ctry.csv:iso,country_name,Maddison_ctry,Downscale_from
socioeconomics/USDA_GDP_MER.csv:Country,iso,1969,1970,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017
water/mfg_water_mapping.csv:iso,continent
water/Liu_EFW_inventory.csv:country,iso,GCAM_R14,ag,irr,ind,muni,trtww,pow_irr,pow_irr_gw,pow_irr_sf,sf_shr2,gw_shr2,irr_gw_shr,irr_sf_shr,pct_ind_sf_cooling,pct_ind_sf_manu,ratio_sw_ind,trtshr,gw_sc_EI_coef,muni_sply_EI_coef,E4W_sf,E4W_gw,E4W_non-fresh,E4W_sc,E4W_trt,E4W_dist,E4W_ww_clt,E4W_ww_trt,E4W_ww_dis,E4W_ag,E4W_ind,E4W_mun
water/aquastat_ctry.csv:aquastat_ctry,iso,AusNWC_reg,Superwell_country

This should make that list a little easier to read:

aglu/A_recent_feed_modifications.csv
aglu/Rohwer_2007_IrrigationEff.csv
aglu/LDS/Water_footprint_m3.csv
aglu/LDS/MIRCA_rfdHA_ha.csv
aglu/LDS/Land_type_area_ha.csv
aglu/LDS/Ref_veg_carbon_Mg_per_ha.csv
aglu/LDS/MIRCA_irrHA_ha.csv
aglu/LDS/LDS_ag_HA_ha.csv
aglu/LDS/LDS_value_milUSD.csv
aglu/LDS/LDS_ag_prod_t.csv
aglu/AGLU_ctry.csv
common/iso_GCAM_regID.csv
emissions/RCP_OC_2000.csv
emissions/CEDS/GFED-CMIP6_LUC_emissions.csv
emissions/CEDS/gains_iso_fuel_emissions.csv
emissions/CEDS/gains_iso_sector_emissions.csv
emissions/mappings/CDIAC_nation_iso.csv
emissions/RCP_BC_2000.csv
emissions/EPA_country_map.csv
energy/CEDB_ResFloorspace_chn.csv
energy/rsrc_unconv_oil_prod_bbld.csv
energy/mappings/UCD_ctry.csv
energy/mappings/IEA_ctry.csv
energy/mappings/cement_regions.csv
energy/mappings/IAA_ctry_region.csv
energy/RECS_ResFloorspace_usa.csv
energy/IEA_PCResFloorspace.csv
energy/Smith_irradiance_ctry_kwh.csv
energy/steel_prod.csv
energy/Other_pcflsp_m2_ctry_Yh.csv
energy/aluminum_prod_USGS.csv
energy/A23.subsector_shrwt_nuc_R.csv
energy/Zhou_wind_supply_ctry_EJ.csv
energy/Odyssee_ResFloorspacePerHouse.csv
energy/Hydropower_potential.csv
gcam-usa/UCS_Database.csv
gcam-usa/UCS_Database.csv
socioeconomics/WB_ExtraCountries_GDP_MER.csv
socioeconomics/socioeconomics_ctry.csv
socioeconomics/USDA_GDP_MER.csv
water/mfg_water_mapping.csv
water/Liu_EFW_inventory.csv
water/aquastat_ctry.csv
ecwood commented 1 year ago

Celina suggested that since it is GCAM-USA, the other regions shouldn't show up. This is a bit confusing, because there are sections like

                        <market year="1990" name="Colombiacrude oil">
                                <MarketGoodOrFuel>crude oil</MarketGoodOrFuel>
                                <MarketRegion>Colombia</MarketRegion>
                                <price unit="1975$/GJ">1.37</price>
                                <demand unit="EJ">0.96984</demand>
                                <supply unit="EJ">0.9699</supply>
                                <ContainedRegion>Colombia</ContainedRegion>
                        </market>
                        <market year="2005" name="Colombiacrude oil">
                                <MarketGoodOrFuel>crude oil</MarketGoodOrFuel>
                                <MarketRegion>Colombia</MarketRegion>
                                <price unit="1975$/GJ">1.546</price>
                                <demand unit="EJ">1.15246</demand>
                                <supply unit="EJ">1.15249</supply>
                                <ContainedRegion>Colombia</ContainedRegion>
                        </market>
                        <market year="2010" name="Colombiacrude oil">
                                <MarketGoodOrFuel>crude oil</MarketGoodOrFuel>
                                <MarketRegion>Colombia</MarketRegion>
                                <price unit="1975$/GJ">3.935</price>
                                <demand unit="EJ">1.76842</demand>
                                <supply unit="EJ">1.76855</supply>
                                <ContainedRegion>Colombia</ContainedRegion>
                        </market>
                        <market year="2015" name="Colombiacrude oil">
                                <MarketGoodOrFuel>crude oil</MarketGoodOrFuel>
                                <MarketRegion>Colombia</MarketRegion>
                                <price unit="1975$/GJ">4.51</price>
                                <demand unit="EJ">2.22854</demand>
                                <supply unit="EJ">2.22867</supply>
                                <ContainedRegion>Colombia</ContainedRegion>
                        </market>
                        <market year="2020" name="Colombiacrude oil">
                                <MarketGoodOrFuel>crude oil</MarketGoodOrFuel>
                                <MarketRegion>Colombia</MarketRegion>
                                <price unit="1975$/GJ">4.86848</price>
                                <demand unit="EJ">1.9725</demand>
                                <supply unit="EJ">1.9725</supply>
                                <ContainedRegion>Colombia</ContainedRegion>
                        </market>

in debug_db.xml (for Colombia the country).

ecwood commented 1 year ago

The global regions are definitely present in the USA output. I searched using the form <region name="XXXX" type="region"> when XXXX is substituted with China and Colombia and both times I got results back. I also saw the one for the USA region. These were the first lines for China, indicating that this isn't a "minimal" (for lack of a better word) entry.

                <region name="China" type="region">
                        <demographics>
                                <populationMiniCAM year="1975">
                                        <total-population unit="thous" year="1975">930691</total-population>
                                </populationMiniCAM>
                                <populationMiniCAM year="1990">
                                        <total-population unit="thous" year="1975">1.18296e+06</total-population>
                                </populationMiniCAM>
                                <populationMiniCAM year="2005">
                                        <total-population unit="thous" year="1975">1.33803e+06</total-population>
                                </populationMiniCAM>
                                <populationMiniCAM year="2010">
                                        <total-population unit="thous" year="1975">1.37632e+06</total-population>
                                </populationMiniCAM>
                                <populationMiniCAM year="2015">
                                        <total-population unit="thous" year="1975">1.41464e+06</total-population>
                                </populationMiniCAM>
                                <populationMiniCAM year="2020">
                                        <total-population unit="thous" year="1975">1.43126e+06</total-population>
                                </populationMiniCAM>
                                <populationMiniCAM year="2025">
                                        <total-population unit="thous" year="1975">1.43768e+06</total-population>
                                </populationMiniCAM>
                                <populationMiniCAM year="2030">
                                        <total-population unit="thous" year="1975">1.43338e+06</total-population>
                                </populationMiniCAM>
                                <populationMiniCAM year="2035">
                                        <total-population unit="thous" year="1975">1.41784e+06</total-population>
                                </populationMiniCAM>
                                <populationMiniCAM year="2040">
                                        <total-population unit="thous" year="1975">1.39162e+06</total-population>
                                </populationMiniCAM>
                                <populationMiniCAM year="2045">
                                        <total-population unit="thous" year="1975">1.35631e+06</total-population>
                                </populationMiniCAM>
                                <populationMiniCAM year="2050">
                                        <total-population unit="thous" year="1975">1.31322e+06</total-population>
                                </populationMiniCAM>
                                <populationMiniCAM year="2055">
                                        <total-population unit="thous" year="1975">1.26414e+06</total-population>
                                </populationMiniCAM>
                                <populationMiniCAM year="2060">
                                        <total-population unit="thous" year="1975">1.21104e+06</total-population>
                                </populationMiniCAM>
                                <populationMiniCAM year="2065">
                                        <total-population unit="thous" year="1975">1.15606e+06</total-population>
                                </populationMiniCAM>
                                <populationMiniCAM year="2070">
                                        <total-population unit="thous" year="1975">1.10108e+06</total-population>
                                </populationMiniCAM>
                                <populationMiniCAM year="2075">
                                        <total-population unit="thous" year="1975">1.04647e+06</total-population>
                                </populationMiniCAM>
                                <populationMiniCAM year="2080">
                                        <total-population unit="thous" year="1975">992245</total-population>
                                </populationMiniCAM>
                                <populationMiniCAM year="2085">
                                        <total-population unit="thous" year="1975">939257</total-population>
                                </populationMiniCAM>
                                <populationMiniCAM year="2090">
                                        <total-population unit="thous" year="1975">889160</total-population>
                                </populationMiniCAM>
                                <populationMiniCAM year="2095">
                                        <total-population unit="thous" year="1975">842988</total-population>
                                </populationMiniCAM>
                                <populationMiniCAM year="2100">
                                        <total-population unit="thous" year="1975">800756</total-population>
                                </populationMiniCAM>
                        </demographics>
                        <supplysector name="regional biomass" type="sector">
                                <cost unit="1975$/GJ" year="1975">1.737</cost>
                                <cost unit="1975$/GJ" year="1990">1.53917</cost>
                                <cost unit="1975$/GJ" year="2005">1.58647</cost>
                                <cost unit="1975$/GJ" year="2010">1.66034</cost>
                                <cost unit="1975$/GJ" year="2015">1.71763</cost>
                                <cost unit="1975$/GJ" year="2020">1.75478</cost>
                                <cost unit="1975$/GJ" year="2025">1.77944</cost>
                                <cost unit="1975$/GJ" year="2030">1.80495</cost>
                                <cost unit="1975$/GJ" year="2035">1.81776</cost>
                                <cost unit="1975$/GJ" year="2040">1.88464</cost>
                                <cost unit="1975$/GJ" year="2045">1.95231</cost>
                                <cost unit="1975$/GJ" year="2050">2.00981</cost>
                                <cost unit="1975$/GJ" year="2055">2.05538</cost>
                                <cost unit="1975$/GJ" year="2060">2.07794</cost>
                                <cost unit="1975$/GJ" year="2065">2.0898</cost>
                                <cost unit="1975$/GJ" year="2070">2.09045</cost>
                                <cost unit="1975$/GJ" year="2075">2.08493</cost>
                                <cost unit="1975$/GJ" year="2080">2.08039</cost>
                                <cost unit="1975$/GJ" year="2085">2.07187</cost>
                                <cost unit="1975$/GJ" year="2090">2.06133</cost>
                                <cost unit="1975$/GJ" year="2095">2.04712</cost>
                                <cost unit="1975$/GJ" year="2100">2.02875</cost>
                                <subsector name="regional biomass" type="subsector">
                                        <share-weight unit="none" year="1975">1</share-weight>
                                        <share-weight unit="none" year="1990">1</share-weight>
                                        <share-weight unit="none" year="2005">1</share-weight>
                                        <share-weight unit="none" year="2010">1</share-weight>
                                        <share-weight unit="none" year="2015">1</share-weight>
                                        <share-weight unit="none" year="2020">1</share-weight>
                                        <share-weight unit="none" year="2025">1</share-weight>
                                        <share-weight unit="none" year="2030">1</share-weight>
                                        <share-weight unit="none" year="2035">1</share-weight>
                                        <share-weight unit="none" year="2040">1</share-weight>
                                        <share-weight unit="none" year="2045">1</share-weight>
                                        <share-weight unit="none" year="2050">1</share-weight>
                                        <share-weight unit="none" year="2055">1</share-weight>
                                        <share-weight unit="none" year="2060">1</share-weight>
                                        <share-weight unit="none" year="2065">1</share-weight>
                                        <share-weight unit="none" year="2070">1</share-weight>
                                        <share-weight unit="none" year="2075">1</share-weight>
                                        <share-weight unit="none" year="2080">1</share-weight>
                                        <share-weight unit="none" year="2085">1</share-weight>
                                        <share-weight unit="none" year="2090">1</share-weight>
                                        <share-weight unit="none" year="2095">1</share-weight>
                                        <share-weight unit="none" year="2100">1</share-weight>
                                        <cost unit="1975$/GJ" year="1975">1.737</cost>
                                        <cost unit="1975$/GJ" year="1990">1.53917</cost>
                                        <cost unit="1975$/GJ" year="2005">1.58647</cost>
                                        <cost unit="1975$/GJ" year="2010">1.66034</cost>
                                        <cost unit="1975$/GJ" year="2015">1.71763</cost>
                                        <cost unit="1975$/GJ" year="2020">1.75478</cost>
                                        <cost unit="1975$/GJ" year="2025">1.77944</cost>
                                        <cost unit="1975$/GJ" year="2030">1.80495</cost>
                                        <cost unit="1975$/GJ" year="2035">1.81776</cost>
                                        <cost unit="1975$/GJ" year="2040">1.88464</cost>
                                        <cost unit="1975$/GJ" year="2045">1.95231</cost>
                                        <cost unit="1975$/GJ" year="2050">2.00981</cost>
                                        <cost unit="1975$/GJ" year="2055">2.05538</cost>
                                        <cost unit="1975$/GJ" year="2060">2.07794</cost>
                                        <cost unit="1975$/GJ" year="2065">2.0898</cost>
                                        <cost unit="1975$/GJ" year="2070">2.09045</cost>
                                        <cost unit="1975$/GJ" year="2075">2.08493</cost>
                                        <cost unit="1975$/GJ" year="2080">2.08039</cost>
                                        <cost unit="1975$/GJ" year="2085">2.07187</cost>
                                        <cost unit="1975$/GJ" year="2090">2.06133</cost>
                                        <cost unit="1975$/GJ" year="2095">2.04712</cost>
                                        <cost unit="1975$/GJ" year="2100">2.02875</cost>
                                        <technology year="1975" name="regional biomass" type="technology">
                                                <share-weight>1</share-weight>
                                                <cost unit="1975$/GJ" year="1975">1.737</cost>
                                                <output-primary name="regional biomass" type="output">
                                                        <physical-output unit="EJ" vintage="1975">1.97e-05</physical-output>
                                                </output-primary>
                                                <CO2 name="CO2" type="GHG">
                                                        <emissions unit="MTC" year="1975">-0.0004531</emissions>
                                                </CO2>
                                                <input-non-energy name="non-energy" type="input">
                                                        <price-paid unit="1975$/GJ" vintage="1975">0.737</price-paid>
                                                </input-non-energy>
                                                <input-energy name="total biomass" type="input">
                                                        <demand-physical unit="EJ" vintage="1975">1.97e-05</demand-physical>
                                                        <IO-coefficient unit="unitless" vintage="1975">1</IO-coefficient>
                                                </input-energy>
                                                <keyword primary-consumption="biomass"></keyword>
                                        </technology>
                                        <technology year="1990" name="regional biomass" type="technology">
                                                <share-weight>1</share-weight>
                                                <cost unit="1975$/GJ" year="1990">1.53917</cost>
                                                <output-primary name="regional biomass" type="output">
                                                        <physical-output unit="EJ" vintage="1990">0.000705038</physical-output>
                                                </output-primary>
                                                <CO2 name="CO2" type="GHG">
ecwood commented 1 year ago

After commenting out the applicable lines in all of the files above, I got the same error:

cd input/gcamdata && Rscript -e "devtools::load_all('.')" -e "driver_drake(write_xml=TRUE)"
Loading gcamdata
Loading required namespace: drake
GCAM Data System v5.1
Found 420 chunks
Found 4265 chunk data requirements
Found 2416 chunk data products
1452 chunk data input(s) not accounted for
▶ target module_emissions_L152.MACC
✖ fail module_emissions_L152.MACC
Error: target module_emissions_L152.MACC failed.
diagnose(module_emissions_L152.MACC)$error$message:
  left_join_no_match: NA values in new data columns
diagnose(module_emissions_L152.MACC)$error$calls:
  gcamdata:::module_emissions_L152.MACC("MAKE", c(emissions.EPA.EPA_2019_raw, 
    emissions.EPA.EPA_2019_MACC_Ag_updated_baseline, emissions.EPA.EPA_2019_MACC_raw, 
    emissions.EPA_MACC_mapping, emissions.EPA_MACC_control_mapping, 
    emissions.EPA_MAC_missing_region, emissions.EPA_country_map))
  EPA_ag %>% left_join_error_no_match(EPA_MACC_control_mapping, 
    by = "source") %>% left_join_error_no_match(EPA_country_map %>% 
    select(-iso) %>% rename(country = EPA_country), by = "country") %>% 
    group_by(GCAM_region_ID, Sector, Process, year) %>% summarise(value = sum(value)) %>% 
    ungroup() %>% filter(year %in% emissions.EPA_MACC_YEAR)
  filter(., year %in% emissions.EPA_MACC_YEAR)
  ungroup(.)
  summarise(., value = sum(value))
  group_by(., GCAM_region_ID, Sector, Proc
Execution halted
make: *** [Makefile:5: xml] Error 1
ecwood commented 1 year ago

Since EPA_2019_MACC_Ag_updated_baseline.csv has a header in the form of source,country,year,gas,unit,value, I think we need to search for country as well. After commenting out the applicable country lines in the files from this comment, it got past that file:

cd input/gcamdata && Rscript -e "devtools::load_all('.')" -e "driver_drake(write_xml=TRUE)"
Loading gcamdata
Loading required namespace: drake
GCAM Data System v5.1
Found 420 chunks
Found 4265 chunk data requirements
Found 2416 chunk data products
1452 chunk data input(s) not accounted for
▶ target energy.mappings.IEA_ctry
▶ target module_energy_LA120.offshore_wind
✖ fail module_energy_LA120.offshore_wind
Error: target module_energy_LA120.offshore_wind failed.
diagnose(module_energy_LA120.offshore_wind)$error$message:
  left_join_no_match: NA values in new data columns
diagnose(module_energy_LA120.offshore_wind)$error$calls:
  gcamdata:::module_energy_LA120.offshore_wind("MAKE", c(common.iso_GCAM_regID, 
    common.GCAM_region_names, energy.A20.wind_class_CFs, energy.A20.offshore_wind_depth_cap_cost, 
    energy.NREL_offshore_energy, energy.NREL_wind_energy_distance_range, 
    energy.offshore_wind_grid_cost, energy.offshore_wind_potential_scaler, 
    L113.globaltech_capital_ATB, L113.globaltech_OMfixed_ATB))
  NREL_offshore_energy %>% select(-total) %>% left_join_error_no_match(iso_GCAM_regID %>% 
    select(country_name, GCAM_region_ID) %>% distinct(), by = c(IAM_country = "country_name")) %>% 
    gather(wind_class, resource.potential.PWh, -IAM_country, 
        -GCAM_region_ID, -depth_class, -distance_to_shore) %>% 
    mutate(resource.potential.EJ = resource.potential.PWh * 1000 
Execution halted
make: *** [Makefile:5: xml] Error 1
ecwood commented 1 year ago

I ran

ubuntu@ip-172-31-57-208:~/gcam-core/input/gcamdata/inst/extdata$ grep -r "country" *
aglu/Rohwer_2007_IrrigationEff.csv:country,iso,application.eff,conveyance.eff,large.scale.frac,management.eff
aglu/AGLU_ctry.csv:FAO_country,iso,GTAP_region,CROSIT_ctry,CROSIT_country_ID,MIRCA_country,IFA2002_country,IFA_region,IMAGE_region_ID,IMAGE_region_name,FAO2050_reg
aglu/LDS/LDS_ag_prod_t.csv:# Title: Initialization of production (t) by country/GLU/crop
aglu/LDS/LDS_ag_prod_t.csv:# Description: Initialization of production (t) by country/GLU/crop
aglu/LDS/MIRCA_irrHA_ha.csv:# Description: mirca irrigated harvested area (ha) for sage land cells in country X glu
aglu/LDS/MIRCA_irrHA_ha.csv:# Original source: MIRCA2000; country raster; new glu raster
aglu/LDS/MIRCA_rfdHA_ha.csv:# Description: mirca rainfed havested area (ha) for sage land cells in country X glu
aglu/LDS/MIRCA_rfdHA_ha.csv:# Original source: MIRCA2000; country raster; new glu raster
aglu/LDS/LDS_value_milUSD.csv:# Title: Initialization of land value (million USD) by country87/use/GLU
aglu/LDS/LDS_value_milUSD.csv:# Description: Initialization of land value (million USD) by country87/use/GLU
aglu/LDS/Land_type_area_ha.csv:# Description: area (ha) for land cells in country X glu X land type X protected category X year
aglu/LDS/Land_type_area_ha.csv:# Original source: hyde land use areas; reference veg; land cover; country raster; glu raster; hyde land area
aglu/LDS/Ref_veg_carbon_Mg_per_ha.csv:# Description: ref veg soil and veg carbon density (Mg/ha) for hyde land cells in country X glu X land type
aglu/LDS/Ref_veg_carbon_Mg_per_ha.csv:# Original source: soil c for sage pot veg; veg c for sage pot veg; reference veg; country raster; new glu raster; hyde land area
aglu/LDS/Water_footprint_m3.csv:# Description: crop average annual water volume consumed (m^3) for sage land cells in country X glu
aglu/LDS/Water_footprint_m3.csv:# Original source: water footprint network; country raster; new glu raster
aglu/LDS/Mueller_yield_levels.csv:# Title: Attainable yield levels by country and basin
aglu/LDS/Mueller_yield_levels.csv:# Description: Min, average, and max refer to the respective yields observed in grid cells within country, basin, and crop; count is the number of grid cells with harvested area for the given crop, country, and basin.
aglu/LDS/L123.LC_bm2_R_MgdFor_Yh_GLU_beforeadjust.csv:# Title: Initialization of harvested area (ha) by country/GLU/crop
aglu/LDS/LDS_ag_HA_ha.csv:# Title: Initialization of harvested area (ha) by country/GLU/crop
aglu/LDS/LDS_ag_HA_ha.csv:# Description: Initialization of harvested area (ha) by country/GLU/crop
aglu/FAO/FAO_an_Dairy_Stocks.csv:# Title: FAO dairy stocks by country, item, year
aglu/FAO/FAO_an_Dairy_Stocks.csv:country codes,countries,item codes,item,element codes,element,units,1961,1962,1963,1964,1965,1966,1967,1968,1969,1970,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017
aglu/FAO/FAO_ag_Prod_t_PRODSTAT.csv:# Title: FAO agricultural production by country, item, year
aglu/FAO/FAO_ag_Prod_t_PRODSTAT.csv:countries,country.codes,item,item.codes,element,element.codes,1961,1962,1963,1964,1965,1966,1967,1968,1969,1970,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017
aglu/FAO/FAO_ag_Imp_t_SUA.csv:# Title: FAO agricultural imports by country, item, year
aglu/FAO/FAO_ag_Imp_t_SUA.csv:countries,country codes,item,item codes,element,element codes,1961,1962,1963,1964,1965,1966,1967,1968,1969,1970,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013
aglu/FAO/FAO_harv_CL_kha_RESOURCESTAT.csv:# Title: FAO harvested cropland (temporary crops) area by country, year
aglu/FAO/FAO_harv_CL_kha_RESOURCESTAT.csv:countries,country codes,item,item codes,element,element codes,1961,1962,1963,1964,1965,1966,1967,1968,1969,1970,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016
aglu/FAO/FAO_an_Food_t_SUA.csv:# Title: FAO animal food consumption by country, item, year
aglu/FAO/FAO_an_Food_t_SUA.csv:countries,country.codes,item,item.codes,element,element.codes,1961,1962,1963,1964,1965,1966,1967,1968,1969,1970,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015
aglu/FAO/FAO_CL_kha_RESOURCESTAT.csv:# Title: FAO cropland area by country, year
aglu/FAO/FAO_CL_kha_RESOURCESTAT.csv:countries,country codes,item,item codes,element,element codes,1961,1962,1963,1964,1965,1966,1967,1968,1969,1970,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016
aglu/FAO/FAO_Fert_Cons_tN_RESOURCESTAT.csv:# Title: FAO fertilizer consumption by country, year
aglu/FAO/FAO_Fert_Cons_tN_RESOURCESTAT.csv:countries,country codes,item,item codes,element,element codes,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017
aglu/FAO/FAO_an_Prod_t_SUA.csv:# Title: FAO animal production by country, item, year
aglu/FAO/FAO_an_Prod_t_SUA.csv:countries,country.codes,item,item.codes,element,element.codes.x,1961,1962,1963,1964,1965,1966,1967,1968,1969,1970,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015
aglu/FAO/FAO_USA_an_Prod_t_PRODSTAT.csv:countries,country.codes,item,item.codes,element,element.codes,1961,1962,1963,1964,1965,1966,1967,1968,1969,1970,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017
aglu/FAO/FAO_For_Exp_m3_USD_FORESTAT.csv:countries,country codes,item,item codes,element,element codes,1961,1962,1963,1964,1965,1966,1967,1968,1969,1970,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017,2018,2019
aglu/FAO/FAO_USA_For_Exp_t_USD_FORESTAT.csv:countries,country.codes,item,item.codes,element,element.codes,1961,1962,1963,1964,1965,1966,1967,1968,1969,1970,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017,2018
aglu/FAO/FAO_fallowland_kha_RESOURCESTAT.csv:# Title: FAO fallow land area by country, year
aglu/FAO/FAO_fallowland_kha_RESOURCESTAT.csv:countries,country codes,item,item codes,element,element codes,1961,1962,1963,1964,1965,1966,1967,1968,1969,1970,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016
aglu/FAO/FAO_For_Exp_m3_FORESTAT.csv:# Title: FAO forests export qunatity by country.year
aglu/FAO/FAO_For_Exp_m3_FORESTAT.csv:countries,country codes,item,item codes,element,element codes,1961,1962,1963,1964,1965,1966,1967,1968,1969,1970,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017,2018,2019
aglu/FAO/FAO_an_Imp_t_SUA.csv:# Title: FAO animal imports by country, item, year
aglu/FAO/FAO_an_Imp_t_SUA.csv:countries,country.codes,item,item.codes,element,element.codes,1961,1962,1963,1964,1965,1966,1967,1968,1969,1970,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015
aglu/FAO/FAO_Fert_Prod_tN_RESOURCESTAT_archv.csv:# Title: FAO fertilizer production by country, year
aglu/FAO/FAO_Fert_Prod_tN_RESOURCESTAT_archv.csv:countries,country codes,item,item codes,element,element codes,1961,1962,1963,1964,1965,1966,1967,1968,1969,1970,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002
aglu/FAO/FAO_ag_Exp_t_SUA.csv:# Title: FAO agricultural exports by country, item, year
aglu/FAO/FAO_ag_Exp_t_SUA.csv:countries,country codes,item,item codes,element,element codes,1961,1962,1963,1964,1965,1966,1967,1968,1969,1970,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012
aglu/FAO/FAO_ag_Feed_t_SUA.csv:# Title: FAO agricultural feed by country, item, year
aglu/FAO/FAO_ag_Feed_t_SUA.csv:countries,country.codes,item,item.codes,element,element.codes,1961,1962,1963,1964,1965,1966,1967,1968,1969,1970,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015
aglu/FAO/FAO_an_Prod_t_PRODSTAT.csv:# Title: FAO animal product output by country, item, year (PRODSTAT database)
aglu/FAO/FAO_an_Prod_t_PRODSTAT.csv:countries,country codes,item,item codes,element,element codes,1961,1962,1963,1964,1965,1966,1967,1968,1969,1970,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015
aglu/FAO/FAO_For_Imp_m3_FORESTAT.csv:# Title: FAO forestry imports (roundwood total) by country.year
aglu/FAO/FAO_For_Imp_m3_FORESTAT.csv:countries,country codes,item,item codes,element,element codes,1961,1962,1963,1964,1965,1966,1967,1968,1969,1970,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017,2018,2019
aglu/FAO/FAO_Fert_Cons_tN_RESOURCESTAT_archv.csv:# Title: FAO fertilizer consumption by country, year
aglu/FAO/FAO_Fert_Cons_tN_RESOURCESTAT_archv.csv:countries,country codes,item,item codes,element,element codes,1961,1962,1963,1964,1965,1966,1967,1968,1969,1970,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002
aglu/FAO/FAO_For_Prod_m3_FORESTAT.csv:# Title: FAO forestry production (roundwood total) by country.year
aglu/FAO/FAO_For_Prod_m3_FORESTAT.csv:countries,country codes,item,item codes,element,element codes,1961,1962,1963,1964,1965,1966,1967,1968,1969,1970,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017,2018,2019
aglu/FAO/FAO_ag_Food_t_SUA.csv:# Title: FAO agricultural food consumption by country, item, year
aglu/FAO/FAO_ag_Food_t_SUA.csv:countries,country.codes,item,item.codes,element,element.codes,1961,1962,1963,1964,1965,1966,1967,1968,1969,1970,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015
aglu/FAO/FAO_USA_ag_an_P_USDt_PRICESTAT.csv:country.codes,countries,item.codes,item,element.codes,element,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017
aglu/FAO/FAO_an_Exp_t_SUA.csv:# Title: FAO animal exports by country, item, year
aglu/FAO/FAO_an_Exp_t_SUA.csv:countries,country.codes,item,item.codes,element,element.codes,1961,1962,1963,1964,1965,1966,1967,1968,1969,1970,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013
aglu/FAO/FAO_an_Stocks.csv:# Title: FAO animal stocks by country, item, year
aglu/FAO/FAO_an_Stocks.csv:country codes,countries,item codes,item,element codes,element,units,1961,1962,1963,1964,1965,1966,1967,1968,1969,1970,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017
aglu/FAO/FAO_ag_CROSIT.csv:# Description: FAO projections to 2050 by country and commodity
aglu/FAO/FAO_ag_CROSIT.csv:country_ID,crop_ID,year,HA_kha_rainfed,Yield_kgHa_rainfed,Prod_kt_rainfed,HA_kha_irrigated,Yield_kgHa_irrigated,Prod_kt_irrigated,HA_kha,Yield_kgHa,Prod_kt,lookup
aglu/FAO/FAO_Fert_Prod_tN_RESOURCESTAT.csv:# Title: FAO fertilizer production by country, year
aglu/FAO/FAO_Fert_Prod_tN_RESOURCESTAT.csv:countries,country codes,item,item codes,element,element codes,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017
aglu/FAO/FAO_ag_HA_ha_PRODSTAT.csv:# Title: FAO agricultural harvested area by country, item,year
aglu/FAO/FAO_ag_HA_ha_PRODSTAT.csv:countries,country.codes,item.codes,item,element,element.codes,1961,1962,1963,1964,1965,1966,1967,1968,1969,1970,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017
common/iso_GCAM_regID.csv:iso,country_name,region_GCAM3,GCAM_region_ID
common/FAO_GDP_Deflators.csv:# Title: FAO GDP deflators by country (2015 = 100)
common/FAO_GDP_Deflators.csv:countries,country codes,item,element,1970,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017,2018,2019
emissions/EDGAR/EDGAR_CF4.csv:# Title: CF4 Emissions by country and main source category
emissions/EDGAR/EDGAR_SF6.csv:# Title: SF6 Emissions by country and main source category
emissions/EDGAR/EDGAR_C2F6.csv:# Title: C2F6 Emissions by country and main source category
emissions/GFED_Deforest_OC.csv:# Title: OC Emissions from deforestation by country 
emissions/CEDS/GFED-CMIP6_LUC_emissions.csv:# Description: CMIP6 emissions data for agr fires processed by in-house processing code to match GCAM country-sector breakdown
emissions/GFED_ForestFire_BC.csv:# Title: OC Emissions from forest fires by country 
emissions/EPA_country_map.csv:# File: EPA_country_map.csv
emissions/EPA_country_map.csv:# Description: Maps EPA country names to iso country names and iso country codes and GCAM regions and Used to prepare EPA emissions data from the global emissions projections
emissions/EPA_country_map.csv:EPA_country,iso,GCAM_region_ID
emissions/EPA_MACC_2030_MtCO2e.csv:# Title: EPA MACC curves - 2030 by sector/region/country
emissions/EPA/EPA_SF6_Magn.csv:country,1990,1995,2000,2005,2010,2015,2020,2025,2030
emissions/EPA/EPA_PFC_PV.csv:country,1990,1995,2000,2005,2010,2015,2020,2025,2030
emissions/EPA/EPA_ODSS_RefAC.csv:country,1990,1995,2000,2005,2010,2015,2020,2025,2030
emissions/EPA/EPA_ODSS_Aerosols.csv:country,1990,1995,2000,2005,2010,2015,2020,2025,2030
emissions/EPA/EPA_ODSS_FireExt.csv:country,1990,1995,2000,2005,2010,2015,2020,2025,2030
emissions/EPA/EPA_SF6_Semi.csv:country,1990,1995,2000,2005,2010,2015,2020,2025,2030
emissions/EPA/EPA_ODSS_Solvents.csv:country,1990,1995,2000,2005,2010,2015,2020,2025,2030
emissions/EPA/EPA_SF6_FPD.csv:country,1990,1995,2000,2005,2010,2015,2020,2025,2030
emissions/EPA/EPA_PFC_Semi.csv:country,1990,1995,2000,2005,2010,2015,2020,2025,2030
emissions/EPA/EPA_SF6_EPS.csv:country,1990,1995,2000,2005,2010,2015,2020,2025,2030
emissions/EPA/EPA_ODSS_Foams.csv:country,1990,1995,2000,2005,2010,2015,2020,2025,2030
emissions/EPA/EPA_2019_MACC_Ag_updated_baseline.csv:source,country,year,gas,unit,value
emissions/EPA/EPA_Semi_HFCs.csv:country,1990,1995,2000,2005,2010,2015,2020,2025,2030
emissions/EPA/EPA_PFC_FPD.csv:country,1990,1995,2000,2005,2010,2015,2020,2025,2030
emissions/EPA/EPA_FPD_HFCs.csv:country,1990,1995,2000,2005,2010,2015,2020,2025,2030
emissions/EPA/EPA_HCFC22.csv:country,1990,1995,2000,2005,2010,2015,2020,2025,2030
emissions/EPA/EPA_PFC_Al.csv:country,1990,1995,2000,2005,2010,2015,2020,2025,2030
emissions/GFED_ForestFire_OC.csv:# Title: OC Emissions from forest fires by country 
emissions/GFED/GFED_ForestFire_NOx.csv:# Title: NOx Emissions from forest fires by country
emissions/GFED/GFED_Deforest_CO.csv:# Title: CO Emissions from deforestation by country
emissions/GFED/GFED_Deforest_SO2.csv:# Title: SO2 Emissions from deforestation by country
emissions/GFED/GFED_ForestFire_SO2.csv:# Title: SO2 Emissions from forest fires by country
emissions/GFED/GFED_ForestFire_CO.csv:# Title: CO Emissions from forest fires by country
emissions/GFED/GFED_Deforest_NOx.csv:# Title: NOx Emissions from deforestation by country
emissions/EPA_MACC_2020_MtCO2e.csv:# Title: EPA MACC curves - 2020 by sector/region/country
emissions/GFED_Deforest_BC.csv:# Title: BC Emissions from deforestation by country 
emissions/mappings/CDIAC_nation_iso.csv:# Title: Mapping between CDIAC country names, UN country codes, and 3-digit ISO codes
emissions/EPA_MACC_baselines_MtCO2e.csv:# Title: EPA MACC curves - baseline by sector/region/country
energy/rsrc_unconv_oil_prod_bbld.csv:# Title: Historical unconventional oil production by country where applicable
energy/steel_prod.csv:# Title: Crude steel production by country and sector
energy/CEDB_ResFloorspace_chn.csv:country,iso,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015
energy/NREL_wind_energy_distance_range.csv:# Source: Eurek et. al (2017); https://doi.org/10.1016/j.eneco.2016.11.015; raw data at https://catalog.data.gov/dataset/global-cfdda-based-onshore-and-offshore-wind-potential-supply-curves-by-country-class-and-
energy/A20.wind_class_CFs.csv:# Source: Eurek et. al (2017); https://doi.org/10.1016/j.eneco.2016.11.015; raw data at https://catalog.data.gov/dataset/global-cfdda-based-onshore-and-offshore-wind-potential-supply-curves-by-country-class-and-
energy/IEA_PCResFloorspace.csv:country,iso,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004
energy/NREL_onshore_energy.csv:# Source: Eurek et. al (2017); https://doi.org/10.1016/j.eneco.2016.11.015; raw data at https://catalog.data.gov/dataset/global-cfdda-based-onshore-and-offshore-wind-potential-supply-curves-by-country-class-and--912c6
energy/NREL_onshore_energy.csv:IAM_country,distance,c1,c2,c3,c4,c5,c6,c7,c8,c9,total
energy/offshore_wind_potential_scaler.csv:# Source:  assumption, informed by fact that Alaksa resource is not accounted in the USA estimate in the original data set (https://catalog.data.gov/dataset/global-cfdda-based-onshore-and-offshore-wind-potential-supply-curves-by-country-class-and--912c6)
energy/GIS/population_weighted_CDD_HadCM3_A2.csv:country,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017,2018,2019,2020,2021,2022,2023,2024,2025,2026,2027,2028,2029,2030,2031,2032,2033,2034,2035,2036,2037,2038,2039,2040,2041,2042,2043,2044,2045,2046,2047,2048,2049,2050,2051,2052,2053,2054,2055,2056,2057,2058,2059,2060,2061,2062,2063,2064,2065,2066,2067,2068,2069,2070,2071,2072,2073,2074,2075,2076,2077,2078,2079,2080,2081,2082,2083,2084,2085,2086,2087,2088,2089,2090,2091,2092,2093,2094,2095,2096,2097,2098,2099
energy/GIS/population_weighted_HDD_CCSM3x_A2.csv:country,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017,2018,2019,2020,2021,2022,2023,2024,2025,2026,2027,2028,2029,2030,2031,2032,2033,2034,2035,2036,2037,2038,2039,2040,2041,2042,2043,2044,2045,2046,2047,2048,2049,2050,2051,2052,2053,2054,2055,2056,2057,2058,2059,2060,2061,2062,2063,2064,2065,2066,2067,2068,2069,2070,2071,2072,2073,2074,2075,2076,2077,2078,2079,2080,2081,2082,2083,2084,2085,2086,2087,2088,2089,2090,2091,2092,2093,2094,2095,2096,2097,2098,2099
energy/GIS/population_weighted_CDD_no_GCM_constdd.csv:country,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017,2018,2019,2020,2021,2022,2023,2024,2025,2026,2027,2028,2029,2030,2031,2032,2033,2034,2035,2036,2037,2038,2039,2040,2041,2042,2043,2044,2045,2046,2047,2048,2049,2050,2051,2052,2053,2054,2055,2056,2057,2058,2059,2060,2061,2062,2063,2064,2065,2066,2067,2068,2069,2070,2071,2072,2073,2074,2075,2076,2077,2078,2079,2080,2081,2082,2083,2084,2085,2086,2087,2088,2089,2090,2091,2092,2093,2094,2095,2096,2097,2098,2099
energy/GIS/population_weighted_CDD_CCSM3x_B1.csv:country,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017,2018,2019,2020,2021,2022,2023,2024,2025,2026,2027,2028,2029,2030,2031,2032,2033,2034,2035,2036,2037,2038,2039,2040,2041,2042,2043,2044,2045,2046,2047,2048,2049,2050,2051,2052,2053,2054,2055,2056,2057,2058,2059,2060,2061,2062,2063,2064,2065,2066,2067,2068,2069,2070,2071,2072,2073,2074,2075,2076,2077,2078,2079,2080,2081,2082,2083,2084,2085,2086,2087,2088,2089,2090,2091,2092,2093,2094,2095,2096,2097,2098,2099
energy/GIS/population_weighted_CDD_HadCM3_B1.csv:country,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017,2018,2019,2020,2021,2022,2023,2024,2025,2026,2027,2028,2029,2030,2031,2032,2033,2034,2035,2036,2037,2038,2039,2040,2041,2042,2043,2044,2045,2046,2047,2048,2049,2050,2051,2052,2053,2054,2055,2056,2057,2058,2059,2060,2061,2062,2063,2064,2065,2066,2067,2068,2069,2070,2071,2072,2073,2074,2075,2076,2077,2078,2079,2080,2081,2082,2083,2084,2085,2086,2087,2088,2089,2090,2091,2092,2093,2094,2095,2096,2097,2098,2099
energy/GIS/population_weighted_HDD_CCSM3x_B1.csv:country,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017,2018,2019,2020,2021,2022,2023,2024,2025,2026,2027,2028,2029,2030,2031,2032,2033,2034,2035,2036,2037,2038,2039,2040,2041,2042,2043,2044,2045,2046,2047,2048,2049,2050,2051,2052,2053,2054,2055,2056,2057,2058,2059,2060,2061,2062,2063,2064,2065,2066,2067,2068,2069,2070,2071,2072,2073,2074,2075,2076,2077,2078,2079,2080,2081,2082,2083,2084,2085,2086,2087,2088,2089,2090,2091,2092,2093,2094,2095,2096,2097,2098,2099
energy/GIS/population_weighted_HDD_no_GCM_constdd.csv:country,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017,2018,2019,2020,2021,2022,2023,2024,2025,2026,2027,2028,2029,2030,2031,2032,2033,2034,2035,2036,2037,2038,2039,2040,2041,2042,2043,2044,2045,2046,2047,2048,2049,2050,2051,2052,2053,2054,2055,2056,2057,2058,2059,2060,2061,2062,2063,2064,2065,2066,2067,2068,2069,2070,2071,2072,2073,2074,2075,2076,2077,2078,2079,2080,2081,2082,2083,2084,2085,2086,2087,2088,2089,2090,2091,2092,2093,2094,2095,2096,2097,2098,2099
energy/GIS/population_weighted_HDD_HadCM3_A2.csv:country,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017,2018,2019,2020,2021,2022,2023,2024,2025,2026,2027,2028,2029,2030,2031,2032,2033,2034,2035,2036,2037,2038,2039,2040,2041,2042,2043,2044,2045,2046,2047,2048,2049,2050,2051,2052,2053,2054,2055,2056,2057,2058,2059,2060,2061,2062,2063,2064,2065,2066,2067,2068,2069,2070,2071,2072,2073,2074,2075,2076,2077,2078,2079,2080,2081,2082,2083,2084,2085,2086,2087,2088,2089,2090,2091,2092,2093,2094,2095,2096,2097,2098,2099
energy/GIS/population_weighted_HDD_HadCM3_B1.csv:country,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017,2018,2019,2020,2021,2022,2023,2024,2025,2026,2027,2028,2029,2030,2031,2032,2033,2034,2035,2036,2037,2038,2039,2040,2041,2042,2043,2044,2045,2046,2047,2048,2049,2050,2051,2052,2053,2054,2055,2056,2057,2058,2059,2060,2061,2062,2063,2064,2065,2066,2067,2068,2069,2070,2071,2072,2073,2074,2075,2076,2077,2078,2079,2080,2081,2082,2083,2084,2085,2086,2087,2088,2089,2090,2091,2092,2093,2094,2095,2096,2097,2098,2099
energy/GIS/population_weighted_CDD_CCSM3x_A2.csv:country,1971,1972,1973,1974,1975,1976,1977,1978,1979,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017,2018,2019,2020,2021,2022,2023,2024,2025,2026,2027,2028,2029,2030,2031,2032,2033,2034,2035,2036,2037,2038,2039,2040,2041,2042,2043,2044,2045,2046,2047,2048,2049,2050,2051,2052,2053,2054,2055,2056,2057,2058,2059,2060,2061,2062,2063,2064,2065,2066,2067,2068,2069,2070,2071,2072,2073,2074,2075,2076,2077,2078,2079,2080,2081,2082,2083,2084,2085,2086,2087,2088,2089,2090,2091,2092,2093,2094,2095,2096,2097,2098,2099
energy/IEA_cement_TPE_GJt.csv:# Description: GJ of energy used to produce a tonne of cement by country
energy/GIS_ctry.csv:# Title: GIS country iso codes,
energy/GIS_ctry.csv:# Description: Mapping between iso codes and country names in GIS data,
energy/GIS_ctry.csv:country,iso
energy/Other_pcflsp_m2_ctry_Yh.csv:country,iso,gcam.consumer,2004,2005,source
energy/mappings/EIA_ctry.csv:# Title: Mapping between iso codes and country names in EIA data
energy/mappings/comtrade_countrycode_ISO.csv:# File: comtrade_countrycode_ISO.csv
energy/mappings/comtrade_countrycode_ISO.csv:# Title: Comtrade country code to ISO mapping
energy/mappings/IEA_ctry.csv:# Title: Mapping between iso codes and country names in IEA databases
energy/mappings/UCD_ctry.csv:# Title:Mapping iso code to country to GCAM3 region
energy/mappings/UCD_ctry.csv:iso,country_name,UCD_region
energy/aluminum_prod_USGS.csv:# Title: Aluminum production by country
energy/aluminum_prod_USGS.csv:country,iso,year,value,unit
energy/Hydropower_potential.csv:# Title: World Hydro Potential and Development: hydropower potential by country
energy/NREL_offshore_energy.csv:# Source: Eurek et. al (2017); https://doi.org/10.1016/j.eneco.2016.11.015; raw data at https://catalog.data.gov/dataset/global-cfdda-based-onshore-and-offshore-wind-potential-supply-curves-by-country-class-and--912c6
energy/NREL_offshore_energy.csv:IAM_country,depth_class,distance_to_shore,c1,c2,c3,c4,c5,c6,c7,c8,c9,total
gcam-usa/Dooley_CCS_USA.csv:# Dahowski et al. Comparing large scale CCS deployment potential in the USA and China: A detailed analysis based on country-specific CO2 transport & storage cost curves. Energy Procedia 4; 2732-2739. doi:DOI: 10.1016/j.egypro.2011.02.175 (2011).
gcam-usa/USGS_mining_water_shares.csv:state,year,state.to.country.share,fresh.share,saline.share
socioeconomics/socioeconomics_ctry.csv:# Title: Mapping (iso-standard country name-Maddisson country name - aggregate region name)
socioeconomics/socioeconomics_ctry.csv:# Description: Maps country names and codes  in socioeconomics databases to three digit iso codes
socioeconomics/socioeconomics_ctry.csv:iso,country_name,Maddison_ctry,Downscale_from
socioeconomics/IMF_GDP_growth.csv:# Description: GDP real growth by country 1980-2024
socioeconomics/UN_popTot.csv:# Description: UN population data by country historical and projected through 2100
socioeconomics/USDA_GDP_MER.csv:# Title: GDP MER by country 1969-2017
water/FAO_desal_AQUASTAT.csv:# Title: Desalinated water production by country and year
water/aquastat_ctry.csv:# Title: Mapping between aquastat database country names and iso codes
water/aquastat_ctry.csv:aquastat_ctry,iso,AusNWC_reg,Superwell_country
water/IBNET_municipal_water_cost_USDm3.csv:# Comments: missing data for Japan and Taiwan are filled in from municipal.water.demand.32.csv. Note some country names were changed to match FAO
water/IBNET_municipal_water_cost_USDm3.csv:year,country,cost,GDPPC,consumption
water/FAO_desal_missing_AQUASTAT.csv:# Title: Desalinated water production by country and year, for selected countries and years missing in the FAO AQUASTAT database
water/nonirrigation_consumption.csv:# Title: Nonirrigation consumption by country
water/nonirrigation_consumption.csv:# Description: 2005 water demands by iso country from gridded data downscaled from the 14 region GCAM water demands.  This should be revised with new gridded data from the 32 region GCAM water demands.
water/nonirrigation_withdrawal.csv:# Title: Nonirrigation withdrawal by country
water/nonirrigation_withdrawal.csv:# Description: 2005 water demands by iso country from gridded data downscaled from the 14 region GCAM water demands.  This should be revised with new gridded data from the 32 region GCAM water demands.
water/basin_to_country_mapping.csv:# File: basin_to_country_mapping.csv
water/basin_to_country_mapping.csv:# Comments: Basin assigned to the country which has the largest basin area.
water/Liu_EFW_inventory.csv:# column 1    ,country,country name
water/Liu_EFW_inventory.csv:# country information,,,water withdrawal,,,,,power irrigation,share between surface water and groundwater,,,,,,variables for estimating process-level water-use amount,,,,adjusting coefficient for EI ,,E4W by water sources,,,E4W by processes,,,,,,E4W by sectors
water/Liu_EFW_inventory.csv:country,iso,GCAM_R14,ag,irr,ind,muni,trtww,pow_irr,pow_irr_gw,pow_irr_sf,sf_shr2,gw_shr2,irr_gw_shr,irr_sf_shr,pct_ind_sf_cooling,pct_ind_sf_manu,ratio_sw_ind,trtshr,gw_sc_EI_coef,muni_sply_EI_coef,E4W_sf,E4W_gw,E4W_non-fresh,E4W_sc,E4W_trt,E4W_dist,E4W_ww_clt,E4W_ww_trt,E4W_ww_dis,E4W_ag,E4W_ind,E4W_mun

giving me this file list

aglu/Rohwer_2007_IrrigationEff.csv
aglu/AGLU_ctry.csv
aglu/LDS/LDS_ag_prod_t.csv
aglu/LDS/MIRCA_irrHA_ha.csv
aglu/LDS/MIRCA_rfdHA_ha.csv
aglu/LDS/LDS_value_milUSD.csv
aglu/LDS/Land_type_area_ha.csv
aglu/LDS/Ref_veg_carbon_Mg_per_ha.csv
aglu/LDS/Water_footprint_m3.csv
aglu/LDS/Mueller_yield_levels.csv
aglu/LDS/L123.LC_bm2_R_MgdFor_Yh_GLU_beforeadjust.csv
aglu/LDS/LDS_ag_HA_ha.csv
aglu/FAO/FAO_an_Dairy_Stocks.csv
aglu/FAO/FAO_ag_Prod_t_PRODSTAT.csv
aglu/FAO/FAO_ag_Imp_t_SUA.csv
aglu/FAO/FAO_harv_CL_kha_RESOURCESTAT.csv
aglu/FAO/FAO_an_Food_t_SUA.csv
aglu/FAO/FAO_CL_kha_RESOURCESTAT.csv
aglu/FAO/FAO_Fert_Cons_tN_RESOURCESTAT.csv
aglu/FAO/FAO_an_Prod_t_SUA.csv
aglu/FAO/FAO_USA_an_Prod_t_PRODSTAT.csv
aglu/FAO/FAO_For_Exp_m3_USD_FORESTAT.csv
aglu/FAO/FAO_USA_For_Exp_t_USD_FORESTAT.csv
aglu/FAO/FAO_fallowland_kha_RESOURCESTAT.csv
aglu/FAO/FAO_For_Exp_m3_FORESTAT.csv
aglu/FAO/FAO_an_Imp_t_SUA.csv
aglu/FAO/FAO_Fert_Prod_tN_RESOURCESTAT_archv.csv
aglu/FAO/FAO_ag_Exp_t_SUA.csv
aglu/FAO/FAO_ag_Feed_t_SUA.csv
aglu/FAO/FAO_an_Prod_t_PRODSTAT.csv
aglu/FAO/FAO_For_Imp_m3_FORESTAT.csv
aglu/FAO/FAO_Fert_Cons_tN_RESOURCESTAT_archv.csv
aglu/FAO/FAO_For_Prod_m3_FORESTAT.csv
aglu/FAO/FAO_ag_Food_t_SUA.csv
aglu/FAO/FAO_USA_ag_an_P_USDt_PRICESTAT.csv
aglu/FAO/FAO_an_Exp_t_SUA.csv
aglu/FAO/FAO_an_Stocks.csv
aglu/FAO/FAO_ag_CROSIT.csv
aglu/FAO/FAO_Fert_Prod_tN_RESOURCESTAT.csv
aglu/FAO/FAO_ag_HA_ha_PRODSTAT.csv
common/iso_GCAM_regID.csv
common/FAO_GDP_Deflators.csv
emissions/EDGAR/EDGAR_CF4.csv
emissions/EDGAR/EDGAR_SF6.csv
emissions/EDGAR/EDGAR_C2F6.csv
emissions/GFED_Deforest_OC.csv
emissions/CEDS/GFED-CMIP6_LUC_emissions.csv
emissions/GFED_ForestFire_BC.csv
emissions/EPA_country_map.csv
emissions/EPA_MACC_2030_MtCO2e.csv
emissions/EPA/EPA_SF6_Magn.csv
emissions/EPA/EPA_PFC_PV.csv
emissions/EPA/EPA_ODSS_RefAC.csv
emissions/EPA/EPA_ODSS_Aerosols.csv
emissions/EPA/EPA_ODSS_FireExt.csv
emissions/EPA/EPA_SF6_Semi.csv
emissions/EPA/EPA_ODSS_Solvents.csv
emissions/EPA/EPA_SF6_FPD.csv
emissions/EPA/EPA_PFC_Semi.csv
emissions/EPA/EPA_SF6_EPS.csv
emissions/EPA/EPA_ODSS_Foams.csv
emissions/EPA/EPA_2019_MACC_Ag_updated_baseline.csv
emissions/EPA/EPA_Semi_HFCs.csv
emissions/EPA/EPA_PFC_FPD.csv
emissions/EPA/EPA_FPD_HFCs.csv
emissions/EPA/EPA_HCFC22.csv
emissions/EPA/EPA_PFC_Al.csv
emissions/GFED_ForestFire_OC.csv
emissions/GFED/GFED_ForestFire_NOx.csv
emissions/GFED/GFED_Deforest_CO.csv
emissions/GFED/GFED_Deforest_SO2.csv
emissions/GFED/GFED_ForestFire_SO2.csv
emissions/GFED/GFED_ForestFire_CO.csv
emissions/GFED/GFED_Deforest_NOx.csv
emissions/EPA_MACC_2020_MtCO2e.csv
emissions/GFED_Deforest_BC.csv
emissions/mappings/CDIAC_nation_iso.csv
emissions/EPA_MACC_baselines_MtCO2e.csv
energy/rsrc_unconv_oil_prod_bbld.csv
energy/steel_prod.csv
energy/CEDB_ResFloorspace_chn.csv
energy/NREL_wind_energy_distance_range.csv
energy/A20.wind_class_CFs.csv
energy/IEA_PCResFloorspace.csv
energy/NREL_onshore_energy.csv
energy/offshore_wind_potential_scaler.csv
energy/GIS/population_weighted_CDD_HadCM3_A2.csv
energy/GIS/population_weighted_HDD_CCSM3x_A2.csv
energy/GIS/population_weighted_CDD_no_GCM_constdd.csv
energy/GIS/population_weighted_CDD_CCSM3x_B1.csv
energy/GIS/population_weighted_CDD_HadCM3_B1.csv
energy/GIS/population_weighted_HDD_CCSM3x_B1.csv
energy/GIS/population_weighted_HDD_no_GCM_constdd.csv
energy/GIS/population_weighted_HDD_HadCM3_A2.csv
energy/GIS/population_weighted_HDD_HadCM3_B1.csv
energy/GIS/population_weighted_CDD_CCSM3x_A2.csv
energy/IEA_cement_TPE_GJt.csv
energy/GIS_ctry.csv
energy/Other_pcflsp_m2_ctry_Yh.csv
energy/mappings/EIA_ctry.csv
energy/mappings/comtrade_countrycode_ISO.csv
energy/mappings/IEA_ctry.csv
energy/mappings/UCD_ctry.csv
energy/aluminum_prod_USGS.csv
energy/Hydropower_potential.csv
energy/NREL_offshore_energy.csv
gcam-usa/Dooley_CCS_USA.csv
gcam-usa/USGS_mining_water_shares.csv
socioeconomics/socioeconomics_ctry.csv
socioeconomics/IMF_GDP_growth.csv
socioeconomics/UN_popTot.csv
socioeconomics/USDA_GDP_MER.csv
water/FAO_desal_AQUASTAT.csv
water/aquastat_ctry.csv
water/IBNET_municipal_water_cost_USDm3.csv
water/FAO_desal_missing_AQUASTAT.csv
water/nonirrigation_consumption.csv
water/basin_to_country_mapping.csv
water/Liu_EFW_inventory.csv
ecwood commented 1 year ago

After going through all of those files, make xml went a lot farther. Here is the error message now:

ubuntu@ip-172-31-57-208:~/gcam-core$ make xml
cd input/gcamdata && Rscript -e "devtools::load_all('.')" -e "driver_drake(write_xml=TRUE)"
Loading gcamdata
Loading required namespace: drake
GCAM Data System v5.1
Found 420 chunks
Found 4265 chunk data requirements
Found 2416 chunk data products
1452 chunk data input(s) not accounted for
▶ target xml.modeltime.xml
▶ target water.basin_to_country_mapping
▶ target emissions.RCP_OC_2000
▶ target emissions.RCP_BC_2000
▶ target socioeconomics.socioeconomics_ctry
▶ target socioeconomics.UN_popTot
▶ target water.IBNET_municipal_water_cost_USDm3
▶ target water.aquastat_ctry
▶ target water.mfg_water_mapping
▶ target xml.disable_climate_model.xml
▶ target aglu.Rohwer_2007_IrrigationEff
▶ target water.Liu_EFW_inventory
▶ target aglu.AGLU_ctry
▶ target socioeconomics.USDA_GDP_MER
▶ target water.nonirrigation_consumption
▶ target aglu.FAO.FAO_ag_CROSIT
▶ target emissions.CEDS.GFED.CMIP6_LUC_emissions
▶ target xml.hector.xml
▶ target emissions.CEDS.gains_iso_fuel_emissions
▶ target emissions.CEDS.gains_iso_sector_emissions
▶ target emissions.EPA.EPA_2019_raw
▶ target aglu.FAO.FAO_harv_CL_kha_RESOURCESTAT
▶ target aglu.FAO.FAO_ag_Food_t_SUA
▶ target aglu.FAO.FAO_Fert_Cons_tN_RESOURCESTAT
▶ target aglu.FAO.FAO_Fert_Cons_tN_RESOURCESTAT_archv
▶ target aglu.FAO.FAO_an_Dairy_Stocks
▶ target aglu.FAO.FAO_fallowland_kha_RESOURCESTAT
▶ target aglu.FAO.FAO_Fert_Prod_tN_RESOURCESTAT
▶ target aglu.FAO.FAO_an_Prod_t_SUA
▶ target aglu.FAO.FAO_For_Prod_m3_FORESTAT
▶ target aglu.FAO.FAO_ag_Feed_t_SUA
▶ target aglu.FAO.FAO_CL_kha_RESOURCESTAT
▶ target aglu.FAO.FAO_an_Stocks
▶ target aglu.FAO.FAO_an_Imp_t_SUA
▶ target aglu.FAO.FAO_an_Exp_t_SUA
▶ target aglu.FAO.FAO_ag_Imp_t_SUA
▶ target aglu.FAO.FAO_ag_Exp_t_SUA
▶ target aglu.FAO.FAO_For_Exp_m3_FORESTAT
▶ target aglu.FAO.FAO_an_Food_t_SUA
▶ target aglu.FAO.FAO_ag_Prod_t_PRODSTAT
▶ target aglu.FAO.FAO_ag_HA_ha_PRODSTAT
▶ target aglu.FAO.FAO_For_Imp_m3_FORESTAT
▶ target aglu.FAO.FAO_Fert_Prod_tN_RESOURCESTAT_archv
▶ target energy.NREL_onshore_energy
▶ target energy.A23.subsector_shrwt_nuc_R
▶ target energy.GIS.population_weighted_CDD_HadCM3_B1
▶ target energy.GIS.population_weighted_HDD_HadCM3_A2
▶ target energy.GIS.population_weighted_CDD_no_GCM_constdd
▶ target energy.GIS.population_weighted_HDD_CCSM3x_B1
▶ target energy.GIS_ctry
▶ target energy.GIS.population_weighted_CDD_CCSM3x_A2
▶ target energy.GIS.population_weighted_HDD_HadCM3_B1
▶ target energy.GIS.population_weighted_CDD_HadCM3_A2
▶ target energy.GIS.population_weighted_HDD_no_GCM_constdd
▶ target energy.GIS.population_weighted_HDD_CCSM3x_A2
▶ target energy.GIS.population_weighted_CDD_CCSM3x_B1
▶ target energy.NREL_offshore_energy
▶ target aglu.LDS.Land_type_area_ha
▶ target emissions.EPA.EPA_Semi_HFCs
▶ target emissions.EPA.EPA_FPD_HFCs
▶ target emissions.EPA.EPA_ODSS_RefAC
▶ target emissions.EPA.EPA_HCFC22
▶ target emissions.EPA.EPA_ODSS_Aerosols
▶ target emissions.EPA.EPA_ODSS_FireExt
▶ target emissions.EPA.EPA_ODSS_Solvents
▶ target emissions.EPA.EPA_ODSS_Foams
▶ target energy.mappings.UCD_ctry
▶ target energy.Smith_irradiance_ctry_kwh
▶ target module_energy_LA162.dac
▶ target energy.Zhou_wind_supply_ctry_EJ
▶ target xml.no_climate_model.xml
▶ target energy.steel_prod
▶ target emissions.EPA.EPA_2019_MACC_Ag_updated_baseline
▶ target emissions.EPA.EPA_2019_MACC_raw
▶ target energy.mappings.EIA_ctry
▶ target xml.nuclear_USA.xml
▶ target emissions.EPA.EPA_SF6_FPD
▶ target emissions.EPA.EPA_PFC_FPD
▶ target emissions.EPA.EPA_PFC_Al
▶ target emissions.EPA.EPA_SF6_Semi
▶ target emissions.EPA.EPA_PFC_PV
▶ target emissions.EPA.EPA_SF6_Magn
▶ target emissions.EPA.EPA_SF6_EPS
▶ target emissions.EPA.EPA_PFC_Semi
▶ target emissions.EDGAR.EDGAR_C2F6
▶ target emissions.EDGAR.EDGAR_SF6
▶ target module_sample_sample
▶ target aglu.LDS.LDS_ag_HA_ha
▶ target aglu.LDS.Water_footprint_m3
▶ target aglu.LDS.Mueller_yield_levels
▶ target aglu.LDS.Ref_veg_carbon_Mg_per_ha
▶ target aglu.LDS.LDS_ag_prod_t
▶ target aglu.LDS.MIRCA_rfdHA_ha
▶ target aglu.LDS.MIRCA_irrHA_ha
▶ target aglu.LDS.LDS_value_milUSD
▶ target xml.magicc.xml
▶ target energy.mappings.cement_regions
▶ target common.FAO_GDP_Deflators
▶ target aglu.FAO.FAO_For_Exp_m3_USD_FORESTAT
▶ target aglu.FAO.FAO_an_Prod_t_PRODSTAT
▶ target energy.Hydropower_potential
▶ target energy.mappings.comtrade_countrycode_ISO
▶ target emissions.mappings.CDIAC_nation_iso
▶ target socioeconomics.IMF_GDP_growth
▶ target module_gcamusa_L103.water_mapping_USA
▶ target module_water_L102.water_supply_unlimited
▶ target module_socioeconomics_L100.Population_downscale_ctry
▶ target module_water_L101.water_supply_groundwater
✖ fail module_water_L101.water_supply_groundwater
Error: target module_water_L101.water_supply_groundwater failed.
diagnose(module_water_L101.water_supply_groundwater)$error$message:
  left_join_no_match: NA values in new data columns
diagnose(module_water_L101.water_supply_groundwater)$error$calls:
  gcamdata:::module_water_L101.water_supply_groundwater("MAKE", 
    c(common.iso_GCAM_regID, water.aquastat_ctry, water.groundwater_uniform, 
        water.groundwater_trend_gleeson, water.groundwater_trend_watergap, 
        water.superwell_groundwater_cost_elec))
  superwell_groundwater_cost_elec %>% filter(!is.na(Country)) %>% 
    mutate(Country = if_else(GCAM_basin_ID == 103, "Taiwan", 
        Country)) %>% left_join_error_no_match(superwell_ctry, 
    by = "Country") %>% left_join_error_no_match(select(iso_GCAM_regID, 
    iso, GCAM_region_ID), by = "iso") %>% mutate(cost_bin = if_else(cost_bin == 
    "> 5", "(5, 5]", cost_bin))
  mutate(., cost_bin = if_else(cost_bin == "> 5", "(5, 5]", cost_bin))
  left_join_error_no_match(., s
Execution halted
make: *** [Makefile:5: xml] Error 1
ecwood commented 1 year ago

After running

ubuntu@ip-172-31-57-208:~/gcam-core$ grep -r "module_water_L101.water_supply_groundwater" input/gcamdata/R/*
input/gcamdata/R/zchunk_L101.water_supply_groundwater.R:#' module_water_L101.water_supply_groundwater
input/gcamdata/R/zchunk_L101.water_supply_groundwater.R:module_water_L101.water_supply_groundwater <- function(command, ...) {

I got this list of files:

  if(command == driver.DECLARE_INPUTS) {
    return(c(FILE = "common/iso_GCAM_regID",
             FILE = "water/aquastat_ctry",
             FILE = "water/groundwater_uniform",
             FILE = "water/groundwater_trend_gleeson",
             FILE = "water/groundwater_trend_watergap",
             FILE = "water/superwell_groundwater_cost_elec"))
ecwood commented 1 year ago

I addressed those and was back to this:

ubuntu@ip-172-31-57-208:~/gcam-core$ make xml
cd input/gcamdata && Rscript -e "devtools::load_all('.')" -e "driver_drake(write_xml=TRUE)"
Loading gcamdata
Loading required namespace: drake
GCAM Data System v5.1
Found 420 chunks
Found 4265 chunk data requirements
Found 2416 chunk data products
1452 chunk data input(s) not accounted for
▶ target module_energy_LA161.Cstorage
▶ target module_emissions_L152.MACC
✖ fail module_emissions_L152.MACC
Error: target module_emissions_L152.MACC failed.
diagnose(module_emissions_L152.MACC)$error$message:
  left_join_no_match: NA values in new data columns
diagnose(module_emissions_L152.MACC)$error$calls:
  gcamdata:::module_emissions_L152.MACC("MAKE", c(emissions.EPA.EPA_2019_raw, 
    emissions.EPA.EPA_2019_MACC_Ag_updated_baseline, emissions.EPA.EPA_2019_MACC_raw, 
    emissions.EPA_MACC_mapping, emissions.EPA_MACC_control_mapping, 
    emissions.EPA_MAC_missing_region, emissions.EPA_country_map))
  EPA_ag %>% left_join_error_no_match(EPA_MACC_control_mapping, 
    by = "source") %>% left_join_error_no_match(EPA_country_map %>% 
    select(-iso) %>% rename(country = EPA_country), by = "country") %>% 
    group_by(GCAM_region_ID, Sector, Process, year) %>% summarise(value = sum(value)) %>% 
    ungroup() %>% filter(year %in% emissions.EPA_MACC_YEAR)
  filter(., year %in% emissions.EPA_MACC_YEAR)
  ungroup(.)
  summarise(., value = sum(value))
  group_by(., GCAM_region_ID, Sector, Proc
Execution halted
make: *** [Makefile:5: xml] Error 1

which has these corresponding fields:

    return(c(FILE = "emissions/EPA/EPA_2019_raw",
             FILE = "emissions/EPA/EPA_2019_MACC_Ag_updated_baseline",
             FILE = "emissions/EPA/EPA_2019_MACC_raw",
             FILE = "emissions/EPA_MACC_mapping",
             FILE = "emissions/EPA_MACC_control_mapping",
             FILE = "emissions/EPA_MAC_missing_region",
             FILE = "emissions/EPA_country_map"))
ecwood commented 1 year ago

I addressed that error. The new error:

ubuntu@ip-172-31-57-208:~/gcam-core$ make xml
cd input/gcamdata && Rscript -e "devtools::load_all('.')" -e "driver_drake(write_xml=TRUE)"
Loading gcamdata
Loading required namespace: drake
GCAM Data System v5.1
Found 420 chunks
Found 4265 chunk data requirements
Found 2416 chunk data products
1452 chunk data input(s) not accounted for
ℹ Consider drake::r_make() to improve robustness.
▶ target module_energy_LA100.CDIAC_downscale_ctry
✖ fail module_energy_LA100.CDIAC_downscale_ctry
Error: target module_energy_LA100.CDIAC_downscale_ctry failed.
diagnose(module_energy_LA100.CDIAC_downscale_ctry)$error$message:
  left_join_no_match: NA values in new data columns
diagnose(module_energy_LA100.CDIAC_downscale_ctry)$error$calls:
  gcamdata:::module_energy_LA100.CDIAC_downscale_ctry("MAKE", c(emissions.CDIAC_CO2_by_nation, 
    emissions.CDIAC_Cseq_by_nation, emissions.mappings.CDIAC_nation_iso))
  CDIAC_nation_iso %>% select(nation, UN_code) %>% distinct(UN_code, 
    .keep_all = TRUE) %>% left_join_error_no_match(CDIAC_Cseq_by_nation, 
    ., by = "UN_code") %>% mutate(liquids.sequestration = abs(liquids.sequestration)) %>% 
    select(nation, year, liquids.sequestration) %>% right_join(CDIAC_CO2_by_nation, 
    by = c("nation", "year")) %>% replace_na(list(liquids.sequestration = 0)) %>% 
    filter(year %in% energy.CDIAC_CO2_HISTORICAL_YEARS)
  filter(., year %in% energy.CDIAC_CO2_HISTORICAL_YEARS)
  replace_na(., list(liquids.sequestration = 0))
  right_join(., CDI
Execution halted
make: *** [Makefile:5: xml] Error 1

which has these files

    return(c(FILE = "emissions/CDIAC_CO2_by_nation",
             FILE = "emissions/CDIAC_Cseq_by_nation",
             FILE = "emissions/mappings/CDIAC_nation_iso"))
ecwood commented 1 year ago

Fixing those issues gave me this error:

ubuntu@ip-172-31-57-208:~/gcam-core$ make xml
cd input/gcamdata && Rscript -e "devtools::load_all('.')" -e "driver_drake(write_xml=TRUE)"
Loading gcamdata
Loading required namespace: drake
GCAM Data System v5.1
Found 420 chunks
Found 4265 chunk data requirements
Found 2416 chunk data products
1452 chunk data input(s) not accounted for
ℹ Consider drake::r_make() to improve robustness.
▶ target module_energy_LA120.offshore_wind
▶ target module_aglu_LA100.0_LDS_preprocessing
▶ target module_aglu_LA100.FAO_downscale_ctry
▶ target module_aglu_LA100.IMAGE_downscale_ctry_yr
▶ target L102.unlimited_nonmapped_water_price_R_W_Y_75USDm3
▶ target L102.unlimited_mapped_water_price_B_W_Y_75USDm3
▶ target module_aglu_LB132.ag_an_For_Prices_USA_C_2005
▶ target L100.Pop_thous_ctry_Yh
▶ target L162.out_Mt_R_dac_Yh
▶ target module_energy_LA119.solar
▶ target module_energy_LB1011.ff_GrossTrade
▶ target water.groundwater_trend_watergap
▶ target water.groundwater_trend_gleeson
▶ target water.superwell_groundwater_cost_elec
▶ target water.groundwater_uniform
▶ target module_energy_LA114.wind
▶ target emissions.EPA.EPA_2019_MACC_Ag_updated_baseline
▶ target emissions.EPA.EPA_2019_MACC_raw
Error: (converted from warning) One or more parsing issues, see `problems()` for details
Execution halted
make: *** [Makefile:5: xml] Error 1
ecwood commented 1 year ago
ubuntu@ip-172-31-57-208:~/gcam-core/input/gcamdata$ Rscript -e "devtools::load_all('.')" -e "driver_drake(write_xml=TRUE)"
Loading gcamdata
Loading required namespace: drake
GCAM Data System v5.1
Found 420 chunks
Found 4265 chunk data requirements
Found 2416 chunk data products
1452 chunk data input(s) not accounted for
ℹ Consider drake::r_make() to improve robustness.
▶ target module_energy_L262.dac
▶ target L100.LDS_ag_prod_t
▶ target L100.LDS_ag_HA_ha
▶ target L100.FAO_ag_Prod_t
▶ target L120.TechChange_offshore_wind
▶ target L120.RsrcCurves_EJ_R_offshore_wind
▶ target L114.RsrcCurves_EJ_R_wind
▶ target L100.FAO_ag_Feed_t
▶ target L100.Water_footprint_m3
▶ target L100.FAO_fallowland_kha
▶ target L100.FAO_harv_CL_kha
▶ target L100.FAO_CL_kha
▶ target module_aglu_LB1321.regional_ag_prices
▶ target L100.FAO_ag_HA_ha
▶ target module_water_L101.water_supply_groundwater
▶ target L120.GridCost_offshore_wind
▶ target L119.Irradiance_rel_R
▶ target L120.RegCapFactor_offshore_wind
▶ target L100.FAO_For_Exp_m3
▶ target L100.Land_type_area_ha
▶ target L100.FAO_ag_Food_t
▶ target module_water_L145.water_demand_municipal
✖ fail module_water_L145.water_demand_municipal
Error: target module_water_L145.water_demand_municipal failed.
diagnose(module_water_L145.water_demand_municipal)$error$message:
  left_join_no_match: NA values in new data columns
diagnose(module_water_L145.water_demand_municipal)$error$calls:
  gcamdata:::module_water_L145.water_demand_municipal("MAKE", c(common.iso_GCAM_regID, 
    water.aquastat_ctry, water.FAO_municipal_water_AQUASTAT, 
    water.IBNET_municipal_water_cost_USDm3, water.municipal_water_use_efficiency, 
    water.mfg_water_mapping, L100.Pop_thous_ctry_Yh))
  FAO_municipal_water_AQUASTAT %>% left_join_error_no_match(aquastat_ctry[c("iso", 
    "aquastat_ctry")], by = c(Area = "aquastat_ctry")) %>% mutate(Year = if_else(Year > 
    max(HISTORICAL_YEARS), max(HISTORICAL_YEARS), Year)) %>% 
    select(iso, Year, Value) %>% complete(iso = unique(iso), 
    Year = HISTORICAL_YEARS) %>% group_by(iso, Year) %>% summarise(Value = sum(Value)) %>% 
    ungroup() %>% rename(year = Year, value = Value)
  rename(., year = Year, 
Execution halted
    return(c(FILE = "common/iso_GCAM_regID",
             FILE = "water/aquastat_ctry",
             FILE = "water/FAO_municipal_water_AQUASTAT",
             FILE = "water/IBNET_municipal_water_cost_USDm3",
             FILE = "water/municipal_water_use_efficiency",
             FILE = "water/mfg_water_mapping",
             "L100.Pop_thous_ctry_Yh"))
ecwood commented 1 year ago

I fixed the problems() issue by unzipping them and running a script to do the comments instead. (Note: I had to change the filename to not include .gz in it). Here are some of the file lists I've come across working on this:

    return(c(FILE = "common/iso_GCAM_regID",
             FILE = "aglu/LDS/Land_type_area_ha",
             FILE = "water/A71.globaltech_coef",
             FILE = "water/AusNWC_desal_techs",
             FILE = "water/EFW_mapping",
             FILE = "water/aquastat_ctry",
             FILE = "water/basin_to_country_mapping",
             FILE = "water/DesalData_capacity_basin",
             FILE = "water/FAO_desal_AQUASTAT",
             FILE = "water/FAO_desal_missing_AQUASTAT",
             FILE = "water/nonirrigation_withdrawal",
             "L1011.en_bal_EJ_R_Si_Fi_Yh"))
    return(c(FILE = "common/iso_GCAM_regID",
             FILE = "water/aquastat_ctry",
             FILE = "water/FAO_industrial_water_AQUASTAT",
             FILE = "water/mfg_water_ratios",
             FILE = "water/mfg_water_mapping",
             FILE = "water/Vassolo_mfg_water",
             "L101.en_bal_EJ_ctry_Si_Fi_Yh_full"))
    return(c(FILE = "common/iso_GCAM_regID",
             FILE = "aglu/LDS/Land_type_area_ha",
             FILE = "water/A71.globaltech_coef",
             FILE = "water/AusNWC_desal_techs",
             FILE = "water/EFW_mapping",
             FILE = "water/aquastat_ctry",
             FILE = "water/basin_to_country_mapping",
             FILE = "water/DesalData_capacity_basin",
             FILE = "water/FAO_desal_AQUASTAT",
             FILE = "water/FAO_desal_missing_AQUASTAT",
             FILE = "water/nonirrigation_withdrawal",
             "L1011.en_bal_EJ_R_Si_Fi_Yh"))
    return(c(FILE = "common/iso_GCAM_regID",
             FILE = "aglu/A_defaultYieldRate",
             FILE = "aglu/AGLU_ctry",
             FILE = "aglu/FAO/FAO_ag_items_PRODSTAT",
             FILE = "aglu/FAO/FAO_ag_CROSIT",
             "L100.LDS_ag_HA_ha",
             "L101.ag_Prod_Mt_R_C_Y_GLU"))

input/gcamdata/R/zchunk_L132.water_demand_manufacturing.R

    return(c(FILE = "common/iso_GCAM_regID",
             FILE = "water/aquastat_ctry",
             FILE = "water/FAO_industrial_water_AQUASTAT",
             FILE = "water/mfg_water_ratios",
             FILE = "water/mfg_water_mapping",
             FILE = "water/Vassolo_mfg_water",
             "L101.en_bal_EJ_ctry_Si_Fi_Yh_full"))
cd input/gcamdata && Rscript -e "devtools::load_all('.')" -e "driver_drake(write_xml=TRUE)"
Loading gcamdata
Loading required namespace: drake
GCAM Data System v5.1
Found 420 chunks
Found 4265 chunk data requirements
Found 2416 chunk data products
1452 chunk data input(s) not accounted for
▶ target module_emissions_L152.MACC
▶ target module_water_L171.desalination
▶ target module_water_batch_water_demand_municipal_xml
▶ target module_water_L132.water_demand_manufacturing
✖ fail module_water_L132.water_demand_manufacturing
Error: target module_water_L132.water_demand_manufacturing failed.
diagnose(module_water_L132.water_demand_manufacturing)$error$message:
  left_join_no_match: NA values in new data columns
diagnose(module_water_L132.water_demand_manufacturing)$error$calls:
  gcamdata:::module_water_L132.water_demand_manufacturing("MAKE", 
    c(common.iso_GCAM_regID, water.aquastat_ctry, water.FAO_industrial_water_AQUASTAT, 
        water.mfg_water_ratios, water.mfg_water_mapping, water.Vassolo_mfg_water, 
        L101.en_bal_EJ_ctry_Si_Fi_Yh_full))
  subset(L101.en_bal_EJ_ctry_Si_Fi_Yh_full, sector %in% water.GCAM_MFG_SECTORS_VASSOLO & 
    fuel %in% water.GCAM_MFG_FUELS_EFW & year == 1995) %>% group_by(iso) %>% 
    summarise(energy_EJ = sum(value)) %>% ungroup() %>% left_join_error_no_match(mfg_water_mapping, 
    by = "iso") %>% group_by(continent) %>% summarise(energy_EJ = sum(energy_EJ)) %>% 
    ungroup()
  ungroup(.)
  summarise(., energy_EJ = sum(energy_EJ))
  group_by(., continent)
  left_jo
Execution halted
make: *** [Makefile:5: xml] Error 1
ecwood commented 1 year ago

After talking with Celina, we've decided to stop working on this. Instead, we will use a larger instance. We think it is better to have all of the data in the long run anyways, to fully utilize the model.