Input data validation checks - Process-related

[ahawkes]

Here is an initial list of input data validation checks for process-related input files, organised by input data CSV file.

PROCESSES

• No duplicate ID (already implemented)

PROCESS_PARAMETERS

• no duplicate process_id - only one set of parameters for each agent (BUG: #137)
• process_id must exist in processes.csv file (already implemented)
• lifetime must be positive (#163)
• discount_rate must be positive. Warning if it is >1 to say discount rate for the process is more than 100%. (#163)
• cap2act must be positive (#163)

PROCESS_FLOWS

• process_id must exist in processes.csv file (duplicates are allowed) (already implemented)
• commodity_id must exist in commodities.csv (#166)
• flow_type can be "fixed" or "flexible" (defaults to "fixed" if not supplied) only (already implemented)

PROCESS_PACS

• process_id must exist in processes.csv file (duplicates are allowed) (already implemented)
• Any identified PAC must exist in commodities.csv (i.e. PAC is the commodity_id) (#164)
• every process must have at least one PAC (already implemented - empty CSV files are not allowed)
• if more than one PAC is identified for one process, all PACs must be either all output flows or all input flows (i.e. can't be a combination of inputs and outputs). Important not to skip this check! (#165)

PROCESS_AVAILABILITIES

• process_id must exist in processes.csv file (duplicates are allowed) (already implemented)
• limit_type must be "fx", "lo" or "up" (already implemented)
• time_slice must be a time slice (a season.time_of_day, whole season, or annual level)
• value must be between 0 and 1 (already implemented)

PROCESS_REGIONS

• process_id must exist in processes.csv file (duplicates are allowed as process can be available in several regions) (already implemented)
• region_id must exist in regions.csv (#138)

ASSET (putting this here as it's mainly process-related) (all should be done with #63)

• process_id must exist in processes.csv file (duplicates are allowed as assets of the same process type can be in different regions, or in the same region with different commission_year or different owner agents)
• region_id must exist in regions.csv file and process_regions.csv must have an entry with that process_id / region_id
• agent_id must exist in agents.csv file
• commission_year must be on or before the 1st milestone year
• decommission_year must be after the 1st milestone year

GENERAL PROCESS-RELATED CONSISTENCY CHECKS

• [ ] #167
• [ ] #168
• [ ] #169

[alexdewar]

Thanks for this @ahawkes. This is a really useful todo list.

I've crossed off one thing that's already implemented and I've opened an issue for another. We can keep your text updated as we open issues for the remaining bits.

[alexdewar]

Hi @ahawkes, I've got a couple of questions about the validation for assets here because I'm working on #63.

1. For the commission year, you say "commission_year must be on or before the 1st milestone year". Does that mean that assets are never commissioned partway through the simulation? It also sounds like they could be commissioned after the simulation ends. Would it be better if we prohibited that entirely or should we issue a warning if users set commission_year to a time after the simulation ends (as it will never be used)?
2. You mention a decommission year, but there is no decommission_year column in assets.csv yet. Is this a feature we should add? If it's not required right now, I'll open an issue to do it later so we don't forget.