Move away from excel

[kamilakrych]

Excel files are an integral part of the BuildME framework. However, version control of such files is not optimal, as the whole file needs to be replaced instead of just a single line or so. @nheeren suggested moving away from excel files and substituting them with more version-control-friendly formats such as yaml, json etc.

[nheeren]

Thanks for creating this issue and your continued support!!

Another thing to consider may be accessibility. We want to keep the entry barrier as low as possible for students, etc. SInce CSV opens readily in most spreadsheet software, it may be another candidate.

[CBreton026]

Hi everyone! I started working on a run_config.yml file to make it easier to set the BuildME run parameters each time, without directly modifying the settings.py file. For now, here are my thoughts:

• The paths and 'permanent' choices (archetype proxies, climate stations) should stay in settings.py or a similar file;
• The selection of run-specific parameters such as included regions should be made in the config file - this would remove the need for having 'debug_combinations' and 'combinations' in settings.py.

For example, the user could simply comment/uncomment each region in the config file:

---
# Included regions: This tells BuildME which regions to use from the settings.py file
incl_regions:
  - "USA"
  # - "DE"
  # - "CN"
  # - "JP"
  # - "IT"
  # - "FR"
  # - "PL"
  - "CA"
  # - "R32EU12-M"
  # - "IN"
  # - "ES"
  # - "UK"
  # - "Oth-R32EU15"
  # - "Oth-R32EU12-H"
  # - "Oth-Asia"
  # - "Oth-LAM"
  # - "Oth-MAF"
  # - "Oth-MAF-Sub-Sahara"
  # - "Oth-OECD"
  # - "Oth-REF"

Then, the incl_regions dictionary would contain the regions to use for a given run. BuildME could then fetch the selected regions from the settings.combinations.

Now, I was wondering if other choices should be made similarly. Should 'occupation', 'energy standard', 'RES', 'climate_scenario' or 'cooling' be selected at the project level? Is it preferable to keep this selection on a per-region basis (as it is now), or can some choices be made for all regions simultaneously? For example, is there a use-case where the user would need to model different combinations of 'RES' scenarios for different regions in a single run (e.g. only 'RES0' for USA, 'RES0' and 'RES2.1' for CA, etc.)?

@shnkn , I am curious to hear your thoughts about this development, especially regarding the IDF converter upgrades you are working on.

[nheeren]

Thanks yet again. This looks great, but I don't fully grasp your idea yet. What would go into settings.py and what would go into the yaml file? I think it is better to keep things that belong together in one central place even if the file becomes a bit crowded. In my experience in working with BuildME one will often make quick changes and try specific combinations for debugging and experimenting.

[CBreton026]

I guess I was looking for an easier way to change the settings without directly modifying the code.

Let's say I want to run a subset of about 50% of "combinations" -- then, I would have to either copy/paste and modify them in debug_combinations, or I could delete or comment out parts of "combinations". However, this modifies my local "settings.py" -- I can always save a backup or reinstall the GitHub file, but it's a bit cumbersome.

If some parameters are always set at the project level (e.g. if we always run the same RES scenarios for all included regions), then it might be easier to keep a list of full possible combinations in settings.py, but then to select only the relevant ones for each run based on an input file. Setting this file to e.g. scenarios = ['RES0'] would run only RES0 for all regions in a given run.

I hope this helps clarify my idea. I think it also links to the discussion we had about moving away from excel, and possibly moving the odym_materials out of settings.py, and changing the replace.xlsx and materials.xlsx files. It seems like the settings.py contains a lot of "user-input" as source code - maybe it would be better to convert such information as yaml input files?

EDIT - Maybe the settings.py could be use to generate initial user-config files if they are absent? This would mean that one could modify the input files for a run, then these user-config files could be copied in the results folder and (optional) deleted. Each 'run' could thus start from either a clean config file or a previous user-config, without modifying the source code.