Headers for a parameters in an Excel file like this:
REGION | TECHNOLOGY | value
are not detected and another error is caused:
DEBUG:otoole.input:Expected indices for OperationalLife are ['REGION', 'TECHNOLOGY']
Traceback (most recent call last):
File "/Users/wusher/miniconda3/envs/otoole38/bin/otoole", line 33, in <module>
sys.exit(load_entry_point('otoole', 'console_scripts', 'otoole')())
File "/Users/wusher/repository/otoole/src/otoole/cli.py", line 470, in main
args.func(args)
File "/Users/wusher/repository/otoole/src/otoole/cli.py", line 220, in conversion_matrix
input_data, _ = read_strategy.read(args.from_path)
File "/Users/wusher/repository/otoole/src/otoole/read_strategies.py", line 158, in read
input_data = self._check_index(input_data)
File "/Users/wusher/repository/otoole/src/otoole/input.py", line 388, in _check_index
self._check_param_index_names(name=name, config=details, df=df)
File "/Users/wusher/repository/otoole/src/otoole/input.py", line 435, in _check_param_index_names
raise OtooleIndexError(
otoole.exceptions.OtooleIndexError: OperationalLife -> Indices inconsistent between config and data. Config indices are ['REGION', 'TECHNOLOGY']. Data indices are ['REGION', 'TECHNOLOGY', 'value', 'YEAR'].
Expected Behavior
An error should be raised stating that the index does not match the available sets, or that value should be upper case.
Steps To Reproduce
In an Excel workbook, change the headers of a parameter from VALUE to value.
Run otoole convert excel csv <workbook.xlsx> <test_folder> config.yaml
Log output
INFO:otoole.cli:Reading config from config.yaml
INFO:otoole.cli:Validating config from config.yaml
INFO:otoole.read_strategies:Checking set EMISSION
INFO:otoole.read_strategies:Checking set MODE_OF_OPERATION
INFO:otoole.read_strategies:Checking set REGION
INFO:otoole.read_strategies:Checking set FUEL
INFO:otoole.read_strategies:Checking set TIMESLICE
INFO:otoole.read_strategies:Checking set TECHNOLOGY
INFO:otoole.read_strategies:Checking set YEAR
INFO:otoole.read_strategies:AccumulatedAnnualDemand reshaped from wide to narrow format
INFO:otoole.read_strategies:AnnualEmissionLimit reshaped from wide to narrow format
INFO:otoole.read_strategies:AnnualExogenousEmission reshaped from wide to narrow format
INFO:otoole.read_strategies:AvailabilityFactor reshaped from wide to narrow format
INFO:otoole.read_strategies:CapacityFactor reshaped from wide to narrow format
INFO:otoole.read_strategies:CapacityToActivityUnit is already in narrow form with headers ['REGION', 'TECHNOLOGY', 'VALUE']
INFO:otoole.read_strategies:CapitalCost reshaped from wide to narrow format
INFO:otoole.read_strategies:DiscountRate is already in narrow form with headers ['REGION', 'VALUE']
INFO:otoole.read_strategies:DepreciationMethod is already in narrow form with headers ['REGION', 'VALUE']
INFO:otoole.read_strategies:EmissionActivityRatio reshaped from wide to narrow format
INFO:otoole.read_strategies:EmissionsPenalty reshaped from wide to narrow format
INFO:otoole.read_strategies:FixedCost reshaped from wide to narrow format
INFO:otoole.read_strategies:InputActivityRatio reshaped from wide to narrow format
INFO:otoole.read_strategies:OutputActivityRatio reshaped from wide to narrow format
INFO:otoole.read_strategies:ModelPeriodEmissionLimit is already in narrow form with headers ['REGION', 'EMISSION', 'VALUE']
INFO:otoole.read_strategies:ModelPeriodExogenousEmission is already in narrow form with headers ['REGION', 'EMISSION', 'VALUE']
INFO:otoole.read_strategies:OperationalLife reshaped from wide to narrow format
INFO:otoole.read_strategies:REMinProductionTarget reshaped from wide to narrow format
INFO:otoole.read_strategies:RETagFuel reshaped from wide to narrow format
INFO:otoole.read_strategies:RETagTechnology reshaped from wide to narrow format
INFO:otoole.read_strategies:ReserveMargin reshaped from wide to narrow format
INFO:otoole.read_strategies:ReserveMarginTagFuel reshaped from wide to narrow format
INFO:otoole.read_strategies:ReserveMarginTagTechnology reshaped from wide to narrow format
INFO:otoole.read_strategies:ResidualCapacity reshaped from wide to narrow format
INFO:otoole.read_strategies:SpecifiedAnnualDemand reshaped from wide to narrow format
INFO:otoole.read_strategies:SpecifiedDemandProfile reshaped from wide to narrow format
INFO:otoole.read_strategies:TotalAnnualMaxCapacity reshaped from wide to narrow format
INFO:otoole.read_strategies:TotalAnnualMaxCapacityInvestment reshaped from wide to narrow format
INFO:otoole.read_strategies:TotalAnnualMinCapacity reshaped from wide to narrow format
INFO:otoole.read_strategies:TotalAnnualMinCapacityInvestment reshaped from wide to narrow format
INFO:otoole.read_strategies:TotalTechnologyAnnualActivityLowerLimit reshaped from wide to narrow format
INFO:otoole.read_strategies:TotalTechnologyAnnualActivityUpperLimit reshaped from wide to narrow format
INFO:otoole.read_strategies:TotalTechnologyModelPeriodActivityLowerLimit is already in narrow form with headers ['TECHNOLOGY', 'VALUE']
INFO:otoole.read_strategies:TotalTechnologyModelPeriodActivityUpperLimit is already in narrow form with headers ['TECHNOLOGY', 'VALUE']
INFO:otoole.read_strategies:TradeRoute reshaped from wide to narrow format
INFO:otoole.read_strategies:VariableCost reshaped from wide to narrow format
INFO:otoole.read_strategies:YearSplit reshaped from wide to narrow format
DEBUG:otoole.input:Identified EMISSION as a set
DEBUG:otoole.input:Identified MODE_OF_OPERATION as a set
DEBUG:otoole.input:Identified REGION as a set
DEBUG:otoole.input:Identified FUEL as a set
DEBUG:otoole.input:Identified TIMESLICE as a set
DEBUG:otoole.input:Identified TECHNOLOGY as a set
DEBUG:otoole.input:Identified YEAR as a set
DEBUG:otoole.input:Actual indices for AccumulatedAnnualDemand are ['REGION', 'FUEL', 'YEAR']
DEBUG:otoole.input:Expected indices for AccumulatedAnnualDemand are ['REGION', 'FUEL', 'YEAR']
DEBUG:otoole.input:Identified AccumulatedAnnualDemand as a parameter
DEBUG:otoole.input:Unable to set index on AccumulatedAnnualDemand
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'FUEL': 'str', 'YEAR': 'int', 'VALUE': 'float'}
DEBUG:otoole.input: VALUE
REGION FUEL YEAR
NaN BJFW3X 2015 76.263780
BJHE3X 2015 0.000000
BJHF3X 2015 0.690638
BJLF3X 2015 68.373182
BJCH3X 2015 0.000000
DEBUG:otoole.input:Actual indices for AnnualEmissionLimit are ['REGION', 'EMISSION', 'YEAR']
DEBUG:otoole.input:Expected indices for AnnualEmissionLimit are ['REGION', 'EMISSION', 'YEAR']
DEBUG:otoole.input:Identified AnnualEmissionLimit as a parameter
DEBUG:otoole.input:Unable to set index on AnnualEmissionLimit
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'EMISSION': 'str', 'YEAR': 'int', 'VALUE': 'float'}
DEBUG:otoole.input: VALUE
REGION EMISSION YEAR
NaN BJREN 2015 0.0
BFREN 2015 0.0
CIREN 2015 0.0
GMREN 2015 0.0
GHREN 2015 0.0
DEBUG:otoole.input:Actual indices for AnnualExogenousEmission are ['REGION', 'EMISSION', 'YEAR']
DEBUG:otoole.input:Expected indices for AnnualExogenousEmission are ['REGION', 'EMISSION', 'YEAR']
DEBUG:otoole.input:Identified AnnualExogenousEmission as a parameter
DEBUG:otoole.input:Unable to set index on AnnualExogenousEmission
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'EMISSION': 'str', 'YEAR': 'int', 'VALUE': 'float'}
DEBUG:otoole.input:Empty DataFrame
Columns: [VALUE]
Index: []
DEBUG:otoole.input:Actual indices for AvailabilityFactor are ['REGION', 'TECHNOLOGY', 'YEAR']
DEBUG:otoole.input:Expected indices for AvailabilityFactor are ['REGION', 'TECHNOLOGY', 'YEAR']
DEBUG:otoole.input:Identified AvailabilityFactor as a parameter
DEBUG:otoole.input:Unable to set index on AvailabilityFactor
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'TECHNOLOGY': 'str', 'YEAR': 'int', 'VALUE': 'float'}
DEBUG:otoole.input: VALUE
REGION TECHNOLOGY YEAR
NaN BJBMCHC01N 2015 0.93
BJBMCHC02N 2015 0.93
BJBMCHC03N 2015 0.93
BJBMCHC04N 2015 0.93
BJBMCHP01O 2015 0.93
DEBUG:otoole.input:Actual indices for CapacityFactor are ['REGION', 'TECHNOLOGY', 'TIMESLICE', 'YEAR']
DEBUG:otoole.input:Expected indices for CapacityFactor are ['REGION', 'TECHNOLOGY', 'TIMESLICE', 'YEAR']
DEBUG:otoole.input:Identified CapacityFactor as a parameter
DEBUG:otoole.input:Unable to set index on CapacityFactor
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'TECHNOLOGY': 'str', 'TIMESLICE': 'str', 'YEAR': 'int', 'VALUE': 'float'}
DEBUG:otoole.input: VALUE
REGION TECHNOLOGY TIMESLICE YEAR
NaN BJBMCHC01N S1D1 2015 0.54
S1D2 2015 0.54
S2D1 2015 0.54
S2D2 2015 0.54
S3D1 2015 0.54
DEBUG:otoole.input:Actual indices for CapacityToActivityUnit are ['REGION', 'TECHNOLOGY']
DEBUG:otoole.input:Expected indices for CapacityToActivityUnit are ['REGION', 'TECHNOLOGY']
DEBUG:otoole.input:Identified CapacityToActivityUnit as a parameter
DEBUG:otoole.input:Unable to set index on CapacityToActivityUnit
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'TECHNOLOGY': 'str', 'VALUE': 'float'}
DEBUG:otoole.input: VALUE
REGION TECHNOLOGY
NaN BJBACKSTOP 31.536
BJBMCHC01N 31.536
BJBMCHC02N 31.536
BJBMCHC03N 31.536
BJBMCHC04N 31.536
DEBUG:otoole.input:Actual indices for CapitalCost are ['REGION', 'TECHNOLOGY', 'YEAR']
DEBUG:otoole.input:Expected indices for CapitalCost are ['REGION', 'TECHNOLOGY', 'YEAR']
DEBUG:otoole.input:Identified CapitalCost as a parameter
DEBUG:otoole.input:Unable to set index on CapitalCost
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'TECHNOLOGY': 'str', 'YEAR': 'int', 'VALUE': 'float'}
DEBUG:otoole.input: VALUE
REGION TECHNOLOGY YEAR
NaN BJBACKSTOP 2015 99999.00000
BJCRUDPROX 2015 24.12682
BJNG00ILGX 2015 0.85500
BJNG00ELGX 2015 1.52000
BJBMCHC01N 2015 7808.00000
DEBUG:otoole.input:Actual indices for DiscountRate are ['REGION']
DEBUG:otoole.input:Expected indices for DiscountRate are ['REGION']
DEBUG:otoole.input:Identified DiscountRate as a parameter
DEBUG:otoole.input:Unable to set index on DiscountRate
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'VALUE': 'float'}
DEBUG:otoole.input: VALUE
REGION
NaN 0.1
DEBUG:otoole.input:Actual indices for DepreciationMethod are ['REGION']
DEBUG:otoole.input:Expected indices for DepreciationMethod are ['REGION']
DEBUG:otoole.input:Identified DepreciationMethod as a parameter
DEBUG:otoole.input:Unable to set index on DepreciationMethod
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'VALUE': 'float'}
DEBUG:otoole.input: VALUE
REGION
NaN 1
DEBUG:otoole.input:Actual indices for EmissionActivityRatio are ['REGION', 'TECHNOLOGY', 'EMISSION', 'MODE_OF_OPERATION', 'YEAR']
DEBUG:otoole.input:Expected indices for EmissionActivityRatio are ['REGION', 'TECHNOLOGY', 'EMISSION', 'MODE_OF_OPERATION', 'YEAR']
DEBUG:otoole.input:Identified EmissionActivityRatio as a parameter
DEBUG:otoole.input:Unable to set index on EmissionActivityRatio
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'TECHNOLOGY': 'str', 'EMISSION': 'str', 'MODE_OF_OPERATION': 'int', 'YEAR': 'int', 'VALUE': 'float'}
DEBUG:otoole.input: VALUE
REGION TECHNOLOGY EMISSION MODE_OF_OPERATION YEAR
NaN BJCO00I00X BJCO2 1 2015 0.090374
BJCO00X00X BJCO2 1 2015 0.090374
BJLF00I00X BJCO2 1 2015 0.069333
BJHF00I00X BJCO2 1 2015 0.074688
BJCR00I00X BJCO2 1 2015 0.070650
DEBUG:otoole.input:Actual indices for EmissionsPenalty are ['REGION', 'EMISSION', 'YEAR']
DEBUG:otoole.input:Expected indices for EmissionsPenalty are ['REGION', 'EMISSION', 'YEAR']
DEBUG:otoole.input:Identified EmissionsPenalty as a parameter
DEBUG:otoole.input:Unable to set index on EmissionsPenalty
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'EMISSION': 'str', 'YEAR': 'int', 'VALUE': 'float'}
DEBUG:otoole.input: VALUE
REGION EMISSION YEAR
NaN BJCH4 2015 0.0
BJN20 2015 0.0
BJFGA 2015 0.0
BJCO2 2015 0.0
BFCH4 2015 0.0
DEBUG:otoole.input:Actual indices for FixedCost are ['REGION', 'TECHNOLOGY', 'YEAR']
DEBUG:otoole.input:Expected indices for FixedCost are ['REGION', 'TECHNOLOGY', 'YEAR']
DEBUG:otoole.input:Identified FixedCost as a parameter
DEBUG:otoole.input:Unable to set index on FixedCost
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'TECHNOLOGY': 'str', 'YEAR': 'int', 'VALUE': 'float'}
DEBUG:otoole.input: VALUE
REGION TECHNOLOGY YEAR
NaN BJBACKSTOP 2015 999999.00
BJETHANOLX 2015 0.00
BJBMCHC01N 2015 91.28
BJBMCHC02N 2015 91.28
BJBMCHC03N 2015 91.28
DEBUG:otoole.input:Actual indices for InputActivityRatio are ['REGION', 'TECHNOLOGY', 'FUEL', 'MODE_OF_OPERATION', 'YEAR']
DEBUG:otoole.input:Expected indices for InputActivityRatio are ['REGION', 'TECHNOLOGY', 'FUEL', 'MODE_OF_OPERATION', 'YEAR']
DEBUG:otoole.input:Identified InputActivityRatio as a parameter
DEBUG:otoole.input:Unable to set index on InputActivityRatio
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'TECHNOLOGY': 'str', 'FUEL': 'str', 'MODE_OF_OPERATION': 'int', 'YEAR': 'int', 'VALUE': 'float'}
DEBUG:otoole.input: VALUE
REGION TECHNOLOGY FUEL MODE_OF_OPERATION YEAR
NaN BJUR00X00X BJWAT1 1 2015 0.0012
BJFW00P00X BJBIOM 1 2015 1.0000
BJCO00P00X BJCOAL 1 2015 1.0000
BJLF00P00X BJLFOI 1 2015 1.0000
BJHF00P00X BJHFOI 1 2015 1.0000
DEBUG:otoole.input:Actual indices for OutputActivityRatio are ['REGION', 'TECHNOLOGY', 'FUEL', 'MODE_OF_OPERATION', 'YEAR']
DEBUG:otoole.input:Expected indices for OutputActivityRatio are ['REGION', 'TECHNOLOGY', 'FUEL', 'MODE_OF_OPERATION', 'YEAR']
DEBUG:otoole.input:Identified OutputActivityRatio as a parameter
DEBUG:otoole.input:Unable to set index on OutputActivityRatio
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'TECHNOLOGY': 'str', 'FUEL': 'str', 'MODE_OF_OPERATION': 'int', 'YEAR': 'int', 'VALUE': 'float'}
DEBUG:otoole.input: VALUE
REGION TECHNOLOGY FUEL MODE_OF_OPERATION YEAR
NaN BJBACKSTOP BJEL01 1 2015 1.00
BJWA00000X BJWAT1 1 2015 1.00
BJFW00P00X BJFW3X 1 2015 0.95
BJCO00P00X BJCO3X 1 2015 0.95
BJLF00P00X BJLF3X 1 2015 0.95
DEBUG:otoole.input:Actual indices for ModelPeriodEmissionLimit are ['REGION', 'EMISSION']
DEBUG:otoole.input:Expected indices for ModelPeriodEmissionLimit are ['REGION', 'EMISSION']
DEBUG:otoole.input:Identified ModelPeriodEmissionLimit as a parameter
DEBUG:otoole.input:Unable to set index on ModelPeriodEmissionLimit
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'EMISSION': 'str', 'VALUE': 'float'}
DEBUG:otoole.input:Empty DataFrame
Columns: [VALUE]
Index: []
DEBUG:otoole.input:Actual indices for ModelPeriodExogenousEmission are ['REGION', 'EMISSION']
DEBUG:otoole.input:Expected indices for ModelPeriodExogenousEmission are ['REGION', 'EMISSION']
DEBUG:otoole.input:Identified ModelPeriodExogenousEmission as a parameter
DEBUG:otoole.input:Unable to set index on ModelPeriodExogenousEmission
DEBUG:otoole.input:Column dtypes identified: {'REGION': 'str', 'EMISSION': 'str', 'VALUE': 'float'}
DEBUG:otoole.input:Empty DataFrame
Columns: [VALUE]
Index: []
DEBUG:otoole.input:Actual indices for OperationalLife are ['REGION', 'TECHNOLOGY', 'value', 'YEAR']
DEBUG:otoole.input:Expected indices for OperationalLife are ['REGION', 'TECHNOLOGY']
Traceback (most recent call last):
File "/Users/wusher/miniconda3/envs/otoole38/bin/otoole", line 33, in <module>
sys.exit(load_entry_point('otoole', 'console_scripts', 'otoole')())
File "/Users/wusher/repository/otoole/src/otoole/cli.py", line 470, in main
args.func(args)
File "/Users/wusher/repository/otoole/src/otoole/cli.py", line 220, in conversion_matrix
input_data, _ = read_strategy.read(args.from_path)
File "/Users/wusher/repository/otoole/src/otoole/read_strategies.py", line 158, in read
input_data = self._check_index(input_data)
File "/Users/wusher/repository/otoole/src/otoole/input.py", line 388, in _check_index
self._check_param_index_names(name=name, config=details, df=df)
File "/Users/wusher/repository/otoole/src/otoole/input.py", line 435, in _check_param_index_names
raise OtooleIndexError(
otoole.exceptions.OtooleIndexError: OperationalLife -> Indices inconsistent between config and data. Config indices are ['REGION', 'TECHNOLOGY']. Data indices are ['REGION', 'TECHNOLOGY', 'value', 'YEAR'].
Operating System
MacOS
What version of otoole are you running?
v1.01
Possible Solution
This check in input.py could be modified, or an additional check for case could be added.
The Issue
Headers for a parameters in an Excel file like this:
are not detected and another error is caused:
Expected Behavior
An error should be raised stating that the index does not match the available sets, or that value should be upper case.
Steps To Reproduce
VALUE
tovalue
.otoole convert excel csv <workbook.xlsx> <test_folder> config.yaml
Log output
Operating System
MacOS
What version of otoole are you running?
v1.01
Possible Solution
This check in
input.py
could be modified, or an additional check for case could be added.Anything else?
No response