search

ClimateCompatibleGrowth / gui_workflow

The snakemake workflow for the Gulf UnderSea Interconnector feasibility study
MIT License
0 stars 0 forks source link

Added scripts to aggregate results for hourly demand and generation #17

Closed abhishek0208 closed 3 years ago

abhishek0208 commented 3 years ago

The workflow now processes aggregated results for hourly demand and generation as hourly_demand.csv and hourly_generation.csv.

The above files, and related rules, require Demand.csv and ProductionByTechnology.csv to be included in agg_results.csv to generate annual_demand.csv and annual_generation.csv.

willu47 commented 3 years ago

Working fine with the following lines in agg_results.csv:

resultfile,indices,filename
Demand,"GLOBAL",annual_demand
ProductionByTechnology,"GLOBAL",annual_generation
abhishek0208 commented 3 years ago

@willu47 I agree that we need an alternative to storing results as .csv files. I hadn't heard of feather and pyarrow (which seems to often be referred to parquet) so thanks for the pointers. I did a quick search of performance benchmarks between them: According to this , both pyarrow (parquet) writes out smaller files while taking longer to read files in. So maybe we go with parquet since file size is the more pressing concern?

abhishek0208 commented 3 years ago

And yes, we could aggregate hourly_generation by commodity. In addition, could we add an entry with wildcards in the list of indexes? e.g. **P****** to select all powerplant technologies.

willu47 commented 3 years ago

And yes, we could aggregate hourly_generation by commodity. In addition, could we add an entry with wildcards in the list of indexes? e.g. **P****** to select all powerplant technologies.

This is a good idea. I'll create a new issue.