search

RMI-PACTA / pacta.scenario.data.preparation

The goal of {pacta.scenario.data.preparation} is to prepare and format all scenario input datasets required to run the {pacta.portfolio.allocate} tool.
https://rmi-pacta.github.io/pacta.scenario.data.preparation/
Other
1 stars 0 forks source link

deal with the malformed CSV text from `IEA-EV-dataEVsalesCarsProjection-APS.csv` #46

Closed cjyetman closed 4 months ago

cjyetman commented 4 months ago

This causes a breaking change in workflow.scenario.preparation that will be fixed by https://github.com/RMI-PACTA/workflow.scenario.preparation/pull/49

devtools::load_all()
#> ℹ Loading pacta.scenario.data.preparation

scenario_preparation_inputs_path <- "~/data/pactarawdata/scenario-sources"
scenario_preparation_outputs_path <- "~/data/pactadatadev/workflow-scenario-preparation-outputs"

config <- list(
  weo_2022_raw_path = "00Imported/WEO2022 extended data/WEO2022 extended data",
  weo_2022_ext_data_regions_raw_filepath = "raw_data_from_provider/used_in_pacta.scenario_preparation/WEO2022_Extended_Data_Regions.csv",
  weo_2022_ext_data_world_raw_filepath = "raw_data_from_provider/used_in_pacta.scenario_preparation/WEO2022_Extended_Data_World.csv",
  weo_2022_fossil_fuels_raw_filepath = "light_process/weo2022_fossilfuel_demand_supply.csv",
  weo_2022_nze_auto_raw_filepath = "processed_data/nze_may_2021_report_old_data/hard_process_auto/used/NZE2021_RawData_2050.xlsx",
  weo_2022_nze_steel_raw_filepath = "raw_data_from_provider/used_in_pacta.scenario_preparation/WEO2022_NZE_SteelData.csv",
  weo_2022_sales_aps_auto_raw_filepath = "raw_data_from_provider/used_in_pacta.scenario_preparation/SalesAPS_rawdata.csv",
  weo_2022_electric_sales_aps_auto_raw_filename = "raw_data_from_provider/used_in_pacta.scenario_preparation/IEA-EV-dataEV salesCarsProjection-APS.csv"
)

source("../workflow.scenario.preparation/process_scenario_weo_2022.R")

pacta.data.validation::validate_intermediate_scenario_output(weo_2022)

waldo::compare(
  dplyr::arrange(pacta.scenario.preparation::weo_2022, source, scenario, scenario_geography, sector, technology),
  weo_2022,
  tolerance = 1e-15
)
#> ✔ No differences

tail(weo_2022_electric_sales_aps_auto_raw_text)
#> [1] "World,Projection-APS,EV sales,Cars,BEV,2021,sales,4700000" 
#> [2] "World,Projection-APS,EV sales,Cars,PHEV,2021,sales,1900000"
#> [3] "World,Projection-APS,EV sales,Cars,BEV,2025,sales,14000000"
#> [4] "World,Projection-APS,EV sales,Cars,PHEV,2025,sales,4700000"
#> [5] "World,Projection-APS,EV sales,Cars,BEV,2030,sales,34000000"
#> [6] "World,Projection-APS,EV sales,Cars,PHEV,2030,sales,7000000"
length(weo_2022_electric_sales_aps_auto_raw_text)
#> [1] 49
cjyetman commented 4 months ago

The goal is to keep file i/o in workflows, and data munging in R pkgs, so this more accurately follows that strategy.

jdhoffa commented 4 months ago

@AlexAxthelm @cjyetman am I the only person that feels like this is an absurdly huge hoop to jump through just to reformat a file that was misformatted due to our manual error to begin with...?! lol

like just update the file!?

cjyetman commented 4 months ago

@AlexAxthelm @cjyetman am I the only person that feels like this is an absurdly huge hoop to jump through just to reformat a file that was misformatted due to manual error to begin with...?! lol

like just update the file!?

no... that was also my preferred solution

AlexAxthelm commented 4 months ago

If you both feel strongly that editing the file is the better call here, then we can do that. I'll make the changes and upload to AFS.

jdhoffa commented 4 months ago

Thanks @AlexAxthelm

Closing this PR then.