OSeMOSYS / otoole

OSeMOSYS Tools for Energy
https://otoole.readthedocs.io
MIT License
23 stars 17 forks source link

Config file not checking data defenitions from MathProg format #151

Closed trevorb1 closed 1 year ago

trevorb1 commented 1 year ago

If I have a MathProg data file that has a parameter which is not defined in the config.yaml file, otoole breaks. For example, running the command:

otoole convert datafile excel data.txt data.xlsx config.yaml

Will traceback to an amply error along the lines of:

packages\amply\amply.py", line 308, in eval
    assert len(self.records[0]) == 1
TypeError: object of type 'TabularRecord' has no len()

We should add a check that all definitions in the config file match the definitions once the data has been read in. The error is thrown in the following function. Specifically while the context manager is open and the file is being loaded in.

https://github.com/OSeMOSYS/otoole/blob/544508dce5c0365e86903c1d5e0523a25fff3a7f/src/otoole/read_strategies.py#L311-L325

trevorb1 commented 1 year ago

I confirmed where this becomes an issue. With a parameter removed from the config file, the following table can be generated. Note that the datafile format is otoole, not momani.

from format to format Works Error
csv datafile Runs, but does not copy missing data none
csv excel Runs, but does not copy missing data none
excel csv no OtooleExcelNameMismatchError: AccumulatedAnnualDemand -> Excel tab name not found in config file
excel datafile no OtooleExcelNameMismatchError: AccumulatedAnnualDemand -> Excel tab name not found in config file
datafile csv no AmplyError: Param AccumulatedAnnualDemand not previously defined
datafile excel no AmplyError: Param AccumulatedAnnualDemand not previously defined

The from format excel is the functionality I think we want. While the from format datafile works, I think this issue is only with long formatted data. When the data is wide foratted (momani), then the exception in the issue description in shown. When the from format is csv, no warning is given to the user which isnt great either.

trevorb1 commented 1 year ago

Replaced by issue #160 and PR #157