YearSplit parameter not converted

[wogandavid]

Hi @willu47 , I encountered some unexpected behavior when converting an Excel file of OSeMOSYS sets and parameters to a datafile using otoole.

The expected behavior is to read in the Excel file and produce the datafile for use by GLPK/CBC. I follow the normal command to convert:

otoole convert excel datafile combined_inputs.xlsx datafile_from_python.txt

The resulting datafile does not have the data for YearSplit, Conversionld, Conversionlh, and Conversionls.

I am using Python 3.7 and otoole 0.10.0.

I was able to produce the expected behavior by downgrading otoole to 0.8.6 then reinstalling 0.10.0. I wonder if there is a dependency that is causing an issue?

The combined_inputs.xlsx is attached: combined_inputs.xlsx

[willu47]

I've started working on a fix for this.

[willu47]

Hi @wogandavid - thanks again for the clear bug report. I've pushed a fix which you can install using pip install otoole==0.10.1b2. Could you let me know if this sorts your problem?

[wogandavid]

Hi @willu47, b2 fixes the YearSplit, Conversionld, Conversionlh, and Conversionls issue, but a new issue appears for me. When converting the same Excel file to a datafile, DepreciationMethod is written as a set rather than a parameter.

I am using otoole 0.10.1b2, pandas 1.2.0, amply 0.1.4

[willu47]

Okay, sorry about that - v0.10.1b3 should fix that now... I've tried a solve with glpsol and osemosys_fast and it works.

[wogandavid]

Thanks @willu47 . Do you want me to test b3? I didn't see the package on PyPI.

[willu47]

@wogandavid - apologies - v0.10.1.b3 is now available on pypi.

[wogandavid]

@willu47 no problem. b3 is working as expected. Thanks!

[willu47]

Great, thanks for checking - I'll merge this into the current version of otoole soon.

[wogandavid]

@willu47 one more potential issue. Using v0.10.1b3, when running otoole's results command to go from a cbc solution file to csv, I receive the following errors after running:

otoole results cbc csv ./tmp/results.sol ./results --input_datafile ./tmp/datafile_from_python.txt

/mnt/c/Users/david/GitHub/aperc-osemosys/env/lib/python3.7/site-packages/otoole-0.10.1b3-py3.7.egg/otoole/results/results.py:263: FutureWarning: The default value of regex will change from True to False in a future version.
  .str.replace(r"^\*\*", "")
/mnt/c/Users/david/GitHub/aperc-osemosys/env/lib/python3.7/site-packages/otoole-0.10.1b3-py3.7.egg/otoole/results/results.py:267: FutureWarning: The default value of regex will change from True to False in a future version. In addition, single character regular expressions will*not* be treated as literal strings when regex=True.
  df["Index"] = df["Index"].str.replace(")", "")
/mnt/c/Users/david/GitHub/aperc-osemosys/env/lib/python3.7/site-packages/pandas/core/frame.py:3188: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  self[k1] = value[k2]
Calculation returned empty dataframe for parameter 'AccumulatedNewCapacity'
/mnt/c/Users/david/GitHub/aperc-osemosys/env/lib/python3.7/site-packages/pandas/core/indexes/multi.py:3585: RuntimeWarning: The values in the array are unorderable. Pass `sort=False` to suppress this warning.
  uniq_tuples = lib.fast_unique_multiple([self._values, other._values], sort=sort)
Calculation returned empty dataframe for parameter 'AnnualEmissions'
/mnt/c/Users/david/GitHub/aperc-osemosys/env/lib/python3.7/site-packages/pandas/core/indexes/multi.py:3585: RuntimeWarning: The values in the array are unorderable. Pass `sort=False` to suppress this warning.
  uniq_tuples = lib.fast_unique_multiple([self._values, other._values], sort=sort)
Calculation returned empty dataframe for parameter 'AnnualTechnologyEmission'
Calculation returned empty dataframe for parameter 'AnnualTechnologyEmissionByMode'
Calculation returned empty dataframe for parameter 'AnnualVariableOperatingCost'
Calculation returned empty dataframe for parameter 'CapitalInvestment'
Calculation returned empty dataframe for parameter 'DiscountedTechnologyEmissionsPenalty'
Calculation returned empty dataframe for parameter 'NewStorageCapacity'
Calculation returned empty dataframe for parameter 'NumberOfNewTechnologyUnits'
Calculation returned empty dataframe for parameter 'ProductionByTechnology'
Calculation returned empty dataframe for parameter 'ProductionByTechnologyAnnual'
Calculation returned empty dataframe for parameter 'RateOfProductionByTechnology'
Calculation returned empty dataframe for parameter 'RateOfProductionByTechnologyByMode'
Calculation returned empty dataframe for parameter 'RateOfUseByTechnology'
Calculation returned empty dataframe for parameter 'RateOfUseByTechnologyByMode'
Calculation returned empty dataframe for parameter 'SalvageValueStorage'
Calculation returned empty dataframe for parameter 'Trade'
Calculation returned empty dataframe for parameter 'UseByTechnology'
Error in sys.excepthook:
Traceback (most recent call last):
  File "/mnt/c/Users/david/GitHub/aperc-osemosys/env/lib/python3.7/site-packages/otoole-0.10.1b3-py3.7.egg/otoole/cli.py", line 282, in exception_handler
    print("{}: {}".format(exception_type.__name__, exception.message))
AttributeError: 'KeyError' object has no attribute 'message'

Original exception was:
Traceback (most recent call last):
  File "/mnt/c/Users/david/GitHub/aperc-osemosys/env/bin/otoole", line 33, in <module>
    sys.exit(load_entry_point('otoole', 'console_scripts', 'otoole')())
  File "/mnt/c/Users/david/GitHub/aperc-osemosys/env/lib/python3.7/site-packages/otoole-0.10.1b3-py3.7.egg/otoole/cli.py", line 287, in main
    args.func(args)
  File "/mnt/c/Users/david/GitHub/aperc-osemosys/env/lib/python3.7/site-packages/otoole-0.10.1b3-py3.7.egg/otoole/cli.py", line 118, in result_matrix
    context.convert(args.from_path, args.to_path, input_data=input_data)
  File "/mnt/c/Users/david/GitHub/aperc-osemosys/env/lib/python3.7/site-packages/otoole-0.10.1b3-py3.7.egg/otoole/input.py", line 123, in convert
    self._write(inputs, output_filepath, default_values)
  File "/mnt/c/Users/david/GitHub/aperc-osemosys/env/lib/python3.7/site-packages/otoole-0.10.1b3-py3.7.egg/otoole/input.py", line 112, in _write
    self._write_strategy.write(inputs, filepath, default_values)
  File "/mnt/c/Users/david/GitHub/aperc-osemosys/env/lib/python3.7
/site-packages/otoole-0.10.1b3-py3.7.egg/otoole/input.py", line 252, in write
    entity_type = self.input_config[name]["type"]
KeyError: 'AnnualFixedOperatingCost'

I tested the same solution file and datafile in a separate environment using v0.10.0 and it performed as expected. I cannot attach the solution file but the model file and datafile are attached. I used the fast version of the code.

model.txt

datafile_from_python.txt

FYI the output from v0.10.0:

otoole results cbc csv results.sol . --input_datafile datafile_from_python.txt

/mnt/c/Users/david/GitHub/otoole-test/env/lib/python3.7/site-packages/otoole/results/results.py:263: FutureWarning: The default value of regex will change from True to False in a future version.
  .str.replace(r"^\*\*", "")
/mnt/c/Users/david/GitHub/otoole-test/env/lib/python3.7/site-packages/otoole/results/results.py:267: FutureWarning: The default value of regex will change from True to False in a future version. In addition, single character regular expressions will*not* be treated as literal strings when regex=True.
  df["Index"] = df["Index"].str.replace(")", "")
/mnt/c/Users/david/GitHub/otoole-test/env/lib/python3.7/site-packages/pandas/core/frame.py:3188: SettingWithCopyWarning:

A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  self[k1] = value[k2]
Calculation returned empty dataframe for parameter 'AccumulatedNewCapacity'
/mnt/c/Users/david/GitHub/otoole-test/env/lib/python3.7/site-packages/pandas/core/indexes/multi.py:3585: RuntimeWarning: The values in the array are unorderable. Pass `sort=False` to suppress this warning.
  uniq_tuples = lib.fast_unique_multiple([self._values, other._values], sort=sort)
Calculation returned empty dataframe for parameter 'AnnualEmissions'
/mnt/c/Users/david/GitHub/otoole-test/env/lib/python3.7/site-packages/pandas/core/indexes/multi.py:3585: RuntimeWarning: The values in the array are unorderable. Pass `sort=False` to suppress this warning.
  uniq_tuples = lib.fast_unique_multiple([self._values, other._values], sort=sort)
Calculation returned empty dataframe for parameter 'AnnualTechnologyEmission'
Calculation returned empty dataframe for parameter 'AnnualTechnologyEmissionByMode'
Calculation returned empty dataframe for parameter 'AnnualVariableOperatingCost'
Calculation returned empty dataframe for parameter 'CapitalInvestment'
Calculation returned empty dataframe for parameter 'DiscountedTechnologyEmissionsPenalty'
Calculation returned empty dataframe for parameter 'NewStorageCapacity'
Calculation returned empty dataframe for parameter 'NumberOfNewTechnologyUnits'
Calculation returned empty dataframe for parameter 'ProductionByTechnology'
Calculation returned empty dataframe for parameter 'ProductionByTechnologyAnnual'
Calculation returned empty dataframe for parameter 'RateOfProductionByTechnology'
Calculation returned empty dataframe for parameter 'RateOfProductionByTechnologyByMode'
Calculation returned empty dataframe for parameter 'RateOfUseByTechnology'
Calculation returned empty dataframe for parameter 'RateOfUseByTechnologyByMode'
Calculation returned empty dataframe for parameter 'SalvageValueStorage'
Calculation returned empty dataframe for parameter 'Trade'
Calculation returned empty dataframe for parameter 'UseByTechnology'

[willu47]

Closed by #102