Open HauHe opened 5 months ago
The bug occurs when running otoole as part of OSeMOSYS_step
Hey @HauHe!
On your note of:
In UTOPIA the
DiscountRateIdv
is not defined. Hence, otoole should take theDiscountRate
is this is the actual expected behaviour? If DiscountRateIdv is not defined in the configuration file, then otoole wont know what value to use as default; regardless of how its defined in the model file. As a temporary solution, adding the definition to the otoole configuration file may fix the issue.
DiscountRateIdv:
indices: [REGION, TECHNOLOGY]
type: param
dtype: float
default: 0.05
This does not get around the issue of this error not being correctly handled by otoole. Below is the log output if DiscountRateIdv
is not in the configuration file:
otoole -v results cplex csv model.sol results datafile data.txt config.yaml
INFO:otoole.results.results:Looking for CapitalInvestment
Traceback (most recent call last):
File "/home/trevorb1/miniconda3/envs/step/lib/python3.11/site-packages/pandas/core/internals/blocks.py", line 1429, in setitem
values[indexer] = casted
~~~~~~^^^^^^^^^
ValueError: could not broadcast input array from shape (0,) into shape (15,1)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/trevorb1/miniconda3/envs/step/bin/otoole", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/trevorb1/repos/otoole/src/otoole/cli.py", line 318, in main
args.func(args)
File "/home/trevorb1/repos/otoole/src/otoole/cli.py", line 74, in _result_matrix
convert_results(
File "/home/trevorb1/repos/otoole/src/otoole/convert.py", line 139, in convert_results
context.convert(from_path, to_path, input_data=input_data)
File "/home/trevorb1/repos/otoole/src/otoole/input.py", line 113, in convert
inputs, default_values = self._read(input_filepath, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/trevorb1/repos/otoole/src/otoole/input.py", line 95, in _read
return self._read_strategy.read(filepath, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/trevorb1/repos/otoole/src/otoole/results/results.py", line 44, in read
results = self.calculate_results(
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/trevorb1/repos/otoole/src/otoole/results/results.py", line 69, in calculate_results
results[name] = results_package[name]
~~~~~~~~~~~~~~~^^^^^^
File "/home/trevorb1/repos/otoole/src/otoole/results/result_package.py", line 96, in __getitem__
results = self.result_mapper[name]()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/trevorb1/repos/otoole/src/otoole/results/result_package.py", line 328, in capital_investment
crf = capital_recovery_factor(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/trevorb1/repos/otoole/src/otoole/results/result_package.py", line 798, in capital_recovery_factor
crf[:] = values
~~~^^^
File "/home/trevorb1/miniconda3/envs/step/lib/python3.11/site-packages/pandas/core/frame.py", line 4282, in __setitem__
return self._setitem_slice(slc, value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/trevorb1/miniconda3/envs/step/lib/python3.11/site-packages/pandas/core/frame.py", line 4306, in _setitem_slice
self.iloc[key] = value
~~~~~~~~~^^^^^
File "/home/trevorb1/miniconda3/envs/step/lib/python3.11/site-packages/pandas/core/indexing.py", line 911, in __setitem__
iloc._setitem_with_indexer(indexer, value, self.name)
File "/home/trevorb1/miniconda3/envs/step/lib/python3.11/site-packages/pandas/core/indexing.py", line 1944, in _setitem_with_indexer
self._setitem_single_block(indexer, value, name)
File "/home/trevorb1/miniconda3/envs/step/lib/python3.11/site-packages/pandas/core/indexing.py", line 2218, in _setitem_single_block
self.obj._mgr = self.obj._mgr.setitem(indexer=indexer, value=value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/trevorb1/miniconda3/envs/step/lib/python3.11/site-packages/pandas/core/internals/managers.py", line 415, in setitem
return self.apply("setitem", indexer=indexer, value=value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/trevorb1/miniconda3/envs/step/lib/python3.11/site-packages/pandas/core/internals/managers.py", line 363, in apply
applied = getattr(b, f)(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/trevorb1/miniconda3/envs/step/lib/python3.11/site-packages/pandas/core/internals/blocks.py", line 1432, in setitem
raise ValueError(
ValueError: setting an array element with a sequence.
Thanks @trevorb1 , indeed when adding the definition of DiscountRateIdv
to the config file AND removing the default definition in the model file it runs.
I wonder if the issue is actually that in Utopia
the DiscountRate
is only defined via the setting the default to 0.5?
Is otoole reading the default values from the text file? Or is it taking the default from the config file?
Because in the function for calculating the capital_investment there is an if else condition, first checking if DiscountRateIdv
is defined and if not it should take the DiscountRate
but how come the DiscountRate
df is empty when the default is defined in the config file?
adding the definition of DiscountRateIdv to the config file AND removing the default definition in the model file it runs.
When updating the configuration file, I left normal definition for discount rate and it ran fine:
param DiscountRate{r in REGION};
param DiscountRateIdv{r in REGION, t in TECHNOLOGY}, default DiscountRate[r];
However printed out these warning:
/home/trevorb1/repos/otoole/src/otoole/results/result_package.py:643: RuntimeWarning: The values in the array are unorderable. Pass `sort=False` to suppress this warning.
discounted_total_costs = discounted_operational_costs.add(
/home/trevorb1/repos/otoole/src/otoole/results/result_package.py:647: RuntimeWarning: The values in the array are unorderable. Pass `sort=False` to suppress this warning.
discounted_total_costs = discounted_total_costs.add(
/home/trevorb1/repos/otoole/src/otoole/results/result_package.py:651: RuntimeWarning: The values in the array are unorderable. Pass `sort=False` to suppress this warning.
discounted_total_costs = discounted_total_costs.sub(
When updating the configuration file, I left normal definition for discount rate and it ran fine:
Interesting, that gave me an error.
(ose_step_dev) HauHe/OSeMOSYS_step > step --step_length 1 --step_length 5 --input_data data/utopia.txt --solver cbc Building and Solving Models: 0%| | 0/5 [00:00<?, ?it/s] Traceback (most recent call last): File "/Users/haukeh/opt/miniconda3/envs/ose_step_dev/bin/step", line 8, in <module> sys.exit(main()) ^^^^^^ File "/Users/haukeh/opt/miniconda3/envs/ose_step_dev/lib/python3.11/site-packages/click/core.py", line 1130, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/haukeh/opt/miniconda3/envs/ose_step_dev/lib/python3.11/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/Users/haukeh/opt/miniconda3/envs/ose_step_dev/lib/python3.11/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/haukeh/opt/miniconda3/envs/ose_step_dev/lib/python3.11/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/haukeh/GitHub/HauHe/OSeMOSYS_step/src/osemosys_step/main.py", line 281, in main shutil.rmtree(item) File "/Users/haukeh/opt/miniconda3/envs/ose_step_dev/lib/python3.11/shutil.py", line 732, in rmtree _rmtree_safe_fd(fd, path, onerror) File "/Users/haukeh/opt/miniconda3/envs/ose_step_dev/lib/python3.11/shutil.py", line 635, in _rmtree_safe_fd onerror(os.scandir, path, sys.exc_info()) File "/Users/haukeh/opt/miniconda3/envs/ose_step_dev/lib/python3.11/shutil.py", line 631, in _rmtree_safe_fd with os.scandir(topfd) as scandir_it: ^^^^^^^^^^^^^^^^^ NotADirectoryError: [Errno 20] Not a directory: PosixPath('results/.DS_Store')
Is this an otoole error?
Not sure, but when removing the default from DiscountRateIdv
in the model file it didn't occur again
Hmm... can you drop in the model/data files you are using, in addition to the otoole commands please? Seems weird that updating the DiscountRate
all the sudden throws a NotADirectoryError
. So just want to isolate the otoole issue from an osemosys_step issue!
See also issue #217
The Issue
When running otoole to convert UTOPIA results calculated by CBC I get an error. When putting a print statement in
result_package.py
in line 315 for theDiscountRate
the df seems to be empty.Expected Behavior
In UTOPIA the
DiscountRateIdv
is not defined. Hence, otoole should take theDiscountRate
.Steps To Reproduce
No response
Log output
No response
Operating System
MacOS
What version of otoole are you running?
1.1.2.post1.dev25+gc3331f2
Possible Solution
No response
Anything else?
No response