search

calliope-project / calliope

A multi-scale energy systems modelling framework
https://www.callio.pe
Apache License 2.0
299 stars 94 forks source link

Assertion Error in operation mode #277

Closed FLomb closed 4 years ago

FLomb commented 5 years ago

Problem description

When running a model I am working on in operation mode, an "Assertion Error" appears after the start of model run. According to some debugging, this might be related to supply_plus resource constraints (in this case applied to PV and Wind techs) causing problems due to unrecognised values in the related timeseries. If inspecting those constraints through _backend_model, some "nan" values indeed appear where 0 values would be expected, whilst the rest is Inf (which is what normally happens). Nonetheless, if trying to run the same model in plan mode, everything seems to work fine, and results are based on properly processed VRES timeseries; yet, inspecting again the same constraints, major errors appear, e.g. : 

ERROR: evaluating object as numeric value:
    resource_con[R10::pv_rooftop,2015-01-07 23:00:00]
        (object: <class 'pyomo.core.base.var._GeneralVarData'>)
    No value for uninitialized NumericValue object
    resource_con[R10::pv_rooftop,2015-01-07 23:00:00]

The strange thing is that this clear issue with the reading of timeseries produces a crash only in operation mode, whilst in plan mode it generates trobules with the constraint but those are not reflected at all in the results, which are just fine as expected.

The association between the "Assertion Error" and the timeseries arises from the following warning raised by Calliope if run in operation mode:

  * loc_techs_supply_plus  (loc_techs_supply_plus) object 'R7::pv_farm' ... 'R7::wind'
[2019-09-12 11:07:29] WARNING  d:\github\calliope\calliope\backend\checks.py:104: RuntimeWarning:

invalid value encountered in greater

Which ultimately leads to:

Traceback (most recent call last):

  File "<ipython-input-23-640ff0f5af65>", line 1, in <module>
    model.run()

  File "d:\github\calliope\calliope\core\model.py", line 244, in run
    self._model_data, self._timings, **kwargs

  File "d:\github\calliope\calliope\backend\run.py", line 56, in run
    backend=BACKEND[run_config.backend], build_only=build_only

  File "d:\github\calliope\calliope\backend\run.py", line 306, in run_operate
    solver_options=solver_options, save_logs=save_logs, warmstart=warmstart,

  File "d:\github\calliope\calliope\backend\pyomo\model.py", line 196, in solve_model
    results = opt.solve(backend_model, tee=True, **solve_kwargs)

  File "D:\Ananconda\envs\calliope_dev\lib\site-packages\pyomo\opt\base\solvers.py", line 573, in solve
    self._presolve(*args, **kwds)

  File "D:\Ananconda\envs\calliope_dev\lib\site-packages\pyomo\solvers\plugins\solvers\GUROBI.py", line 223, in _presolve
    ILMLicensedSystemCallSolver._presolve(self, *args, **kwds)

  File "D:\Ananconda\envs\calliope_dev\lib\site-packages\pyomo\opt\solver\shellcmd.py", line 198, in _presolve
    OptSolver._presolve(self, *args, **kwds)

  File "D:\Ananconda\envs\calliope_dev\lib\site-packages\pyomo\opt\base\solvers.py", line 673, in _presolve
    **kwds)

  File "D:\Ananconda\envs\calliope_dev\lib\site-packages\pyomo\opt\base\solvers.py", line 744, in _convert_problem
    **kwds)

  File "D:\Ananconda\envs\calliope_dev\lib\site-packages\pyomo\opt\base\convert.py", line 105, in convert_problem
    problem_files, symbol_map = converter.apply(*tmp, **tmpkw)

  File "D:\Ananconda\envs\calliope_dev\lib\site-packages\pyomo\solvers\plugins\converter\model.py", line 88, in apply
    io_options=io_options)

  File "D:\Ananconda\envs\calliope_dev\lib\site-packages\pyomo\core\base\block.py", line 1716, in write
    io_options)

  File "D:\Ananconda\envs\calliope_dev\lib\site-packages\pyomo\repn\plugins\cpxlp.py", line 175, in __call__
    include_all_variable_bounds=include_all_variable_bounds)

  File "D:\Ananconda\envs\calliope_dev\lib\site-packages\pyomo\repn\plugins\cpxlp.py", line 646, in _print_model_LP
    value(constraint_data.upper)

AssertionError

Steps to reproduce the problem

The test model used to investigate the issue can be found at: https://github.com/FLomb/errorFix_Calliope_operation

Just run it in operation mode to produce the Error.

Calliope version

0.6.5-dev

brynpickering commented 5 years ago

So this is caused by technologies having an energy_cap_max/equals with an infinite value. In plan mode, energy_cap is a decision variable; in operate mode it is a parameter (with value equal to energy_cap_max/equals).

So there are instances for non-dispatchable technologies where you get available_resource = 0 * inf (e.g. PV at night). This will lead to NaN responses.

This used to be picked up by checks in the backend, which would not allow the optimisation to continue because non-allowed inf values were found. It seems there is some issue in these checks, which is causing the raised warning: invalid value encountered in greater.

Your best bet is to only construct operation mode problems where all relevant parameters are set to finite values. If you've done this, and issues are still arising, then we'll need to dig into what's happening in the backend checks.

FLomb commented 5 years ago

Considering that I have no loc::tech with inf values set for energy_cap_equals/max, I am wondering if this could be related to some of the following warnings, related to the force_resource option:

[2019-09-17 16:20:39] WARNING  Warning: Possible issues found during model processing:
 * Energy capacity constraint removed from AT::el_export as force_resource is applied
 * Energy capacity constraint removed from AT::el_import as force_resource is applied
 * Energy capacity constraint removed from CH::el_export as force_resource is applied
 * Energy capacity constraint removed from CH::el_import as force_resource is applied
 * Energy capacity constraint removed from CNOR::demand_power as force_resource is applied
 * Energy capacity constraint removed from CNOR::hydro_dam as force_resource is applied
 * Energy capacity constraint removed from CNOR::hydro_ror as force_resource is applied
 * Energy capacity constraint removed from CSUD::demand_power as force_resource is applied
 * Energy capacity constraint removed from CSUD::hydro_dam as force_resource is applied
 * Energy capacity constraint removed from CSUD::hydro_ror as force_resource is applied
 * Energy capacity constraint removed from FR::el_export as force_resource is applied
 * Energy capacity constraint removed from FR::el_import as force_resource is applied
 * Energy capacity constraint removed from GR::el_export as force_resource is applied
 * Energy capacity constraint removed from GR::el_import as force_resource is applied
 * Energy capacity constraint removed from NORD::demand_power as force_resource is applied
 * Energy capacity constraint removed from NORD::hydro_dam as force_resource is applied
 * Energy capacity constraint removed from NORD::hydro_ror as force_resource is applied
 * Energy capacity constraint removed from R10::el_curtailment as force_resource is applied
 * Energy capacity constraint removed from R10::pv_farm as force_resource is applied
 * Energy capacity constraint removed from R10::pv_rooftop as force_resource is applied
 * Energy capacity constraint removed from R10::wind as force_resource is applied
 * Energy capacity constraint removed from R11::el_curtailment as force_resource is applied
 * Energy capacity constraint removed from R11::pv_farm as force_resource is applied
 * Energy capacity constraint removed from R11::pv_rooftop as force_resource is applied
 * Energy capacity constraint removed from R11::wind as force_resource is applied
 * Energy capacity constraint removed from R12::el_curtailment as force_resource is applied
 * Energy capacity constraint removed from R12::pv_farm as force_resource is applied
 * Energy capacity constraint removed from R12::pv_rooftop as force_resource is applied
 * Energy capacity constraint removed from R12::wind as force_resource is applied
 * Energy capacity constraint removed from R13::el_curtailment as force_resource is applied
 * Energy capacity constraint removed from R13::pv_farm as force_resource is applied
 * Energy capacity constraint removed from R13::pv_rooftop as force_resource is applied
 * Energy capacity constraint removed from R13::wind as force_resource is applied
 * Energy capacity constraint removed from R14::el_curtailment as force_resource is applied
 * Energy capacity constraint removed from R14::pv_farm as force_resource is applied
 * Energy capacity constraint removed from R14::pv_rooftop as force_resource is applied
 * Energy capacity constraint removed from R14::wind as force_resource is applied
 * Energy capacity constraint removed from R15::el_curtailment as force_resource is applied
 * Energy capacity constraint removed from R15::pv_farm as force_resource is applied
 * Energy capacity constraint removed from R15::pv_rooftop as force_resource is applied
 * Energy capacity constraint removed from R15::wind as force_resource is applied
 * Energy capacity constraint removed from R16::el_curtailment as force_resource is applied
 * Energy capacity constraint removed from R16::pv_farm as force_resource is applied
 * Energy capacity constraint removed from R16::pv_rooftop as force_resource is applied
 * Energy capacity constraint removed from R16::wind as force_resource is applied
 * Energy capacity constraint removed from R17::el_curtailment as force_resource is applied
 * Energy capacity constraint removed from R17::pv_farm as force_resource is applied
 * Energy capacity constraint removed from R17::pv_rooftop as force_resource is applied
 * Energy capacity constraint removed from R17::wind as force_resource is applied
 * Energy capacity constraint removed from R18::el_curtailment as force_resource is applied
 * Energy capacity constraint removed from R18::pv_farm as force_resource is applied
 * Energy capacity constraint removed from R18::pv_rooftop as force_resource is applied
 * Energy capacity constraint removed from R18::wind as force_resource is applied
 * Energy capacity constraint removed from R1::el_curtailment as force_resource is applied
 * Energy capacity constraint removed from R1::pv_farm as force_resource is applied
 * Energy capacity constraint removed from R1::pv_rooftop as force_resource is applied
 * Energy capacity constraint removed from R1::wind as force_resource is applied
 * Energy capacity constraint removed from R2::el_curtailment as force_resource is applied
 * Energy capacity constraint removed from R2::pv_farm as force_resource is applied
 * Energy capacity constraint removed from R2::pv_rooftop as force_resource is applied
 * Energy capacity constraint removed from R2::wind as force_resource is applied
 * Energy capacity constraint removed from R3::el_curtailment as force_resource is applied
 * Energy capacity constraint removed from R3::pv_farm as force_resource is applied
 * Energy capacity constraint removed from R3::pv_rooftop as force_resource is applied
 * Energy capacity constraint removed from R3::wind as force_resource is applied
 * Energy capacity constraint removed from R4::el_curtailment as force_resource is applied
 * Energy capacity constraint removed from R4::pv_farm as force_resource is applied
 * Energy capacity constraint removed from R4::pv_rooftop as force_resource is applied
 * Energy capacity constraint removed from R4::wind as force_resource is applied
 * Energy capacity constraint removed from R5::el_curtailment as force_resource is applied
 * Energy capacity constraint removed from R5::pv_farm as force_resource is applied
 * Energy capacity constraint removed from R5::pv_rooftop as force_resource is applied
 * Energy capacity constraint removed from R5::wind as force_resource is applied
 * Energy capacity constraint removed from R6::el_curtailment as force_resource is applied
 * Energy capacity constraint removed from R6::pv_farm as force_resource is applied
 * Energy capacity constraint removed from R6::pv_rooftop as force_resource is applied
 * Energy capacity constraint removed from R6::wind as force_resource is applied
 * Energy capacity constraint removed from R7::el_curtailment as force_resource is applied
 * Energy capacity constraint removed from R7::pv_farm as force_resource is applied
 * Energy capacity constraint removed from R7::pv_rooftop as force_resource is applied
 * Energy capacity constraint removed from R7::wind as force_resource is applied
 * Energy capacity constraint removed from R8::el_curtailment as force_resource is applied
 * Energy capacity constraint removed from R8::pv_farm as force_resource is applied
 * Energy capacity constraint removed from R8::pv_rooftop as force_resource is applied
 * Energy capacity constraint removed from R8::wind as force_resource is applied
 * Energy capacity constraint removed from R9::el_curtailment as force_resource is applied
 * Energy capacity constraint removed from R9::geothermal as force_resource is applied
 * Energy capacity constraint removed from R9::pv_farm as force_resource is applied
 * Energy capacity constraint removed from R9::pv_rooftop as force_resource is applied
 * Energy capacity constraint removed from R9::wind as force_resource is applied
 * Energy capacity constraint removed from SARD::demand_power as force_resource is applied
 * Energy capacity constraint removed from SARD::el_curtailment as force_resource is applied
 * Energy capacity constraint removed from SARD::hydro_dam as force_resource is applied
 * Energy capacity constraint removed from SARD::hydro_ror as force_resource is applied
 * Energy capacity constraint removed from SARD::pv_farm as force_resource is applied
 * Energy capacity constraint removed from SARD::pv_rooftop as force_resource is applied
 * Energy capacity constraint removed from SARD::wind as force_resource is applied
 * Energy capacity constraint removed from SI::el_export as force_resource is applied
 * Energy capacity constraint removed from SI::el_import as force_resource is applied
 * Energy capacity constraint removed from SICI::demand_power as force_resource is applied
 * Energy capacity constraint removed from SICI::el_curtailment as force_resource is applied
 * Energy capacity constraint removed from SICI::hydro_dam as force_resource is applied
 * Energy capacity constraint removed from SICI::hydro_ror as force_resource is applied
 * Energy capacity constraint removed from SICI::pv_farm as force_resource is applied
 * Energy capacity constraint removed from SICI::pv_rooftop as force_resource is applied
 * Energy capacity constraint removed from SICI::wind as force_resource is applied
 * Energy capacity constraint removed from SUD::demand_power as force_resource is applied
 * Energy capacity constraint removed from SUD::hydro_dam as force_resource is applied
 * Energy capacity constraint removed from SUD::hydro_ror as force_resource is applied
 * Resource capacity constraint removed from AT::el_import as force_resource is applied
 * Resource capacity constraint removed from CH::el_import as force_resource is applied
 * Resource capacity constraint removed from CNOR::hydro_dam as force_resource is applied
 * Resource capacity constraint removed from CNOR::hydro_ror as force_resource is applied
 * Resource capacity constraint removed from CSUD::hydro_dam as force_resource is applied
 * Resource capacity constraint removed from CSUD::hydro_ror as force_resource is applied
 * Resource capacity constraint removed from FR::el_import as force_resource is applied
 * Resource capacity constraint removed from GR::el_import as force_resource is applied
 * Resource capacity constraint removed from NORD::hydro_dam as force_resource is applied
 * Resource capacity constraint removed from NORD::hydro_ror as force_resource is applied
 * Resource capacity constraint removed from R10::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R10::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R10::wind as force_resource is applied
 * Resource capacity constraint removed from R11::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R11::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R11::wind as force_resource is applied
 * Resource capacity constraint removed from R12::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R12::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R12::wind as force_resource is applied
 * Resource capacity constraint removed from R13::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R13::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R13::wind as force_resource is applied
 * Resource capacity constraint removed from R14::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R14::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R14::wind as force_resource is applied
 * Resource capacity constraint removed from R15::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R15::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R15::wind as force_resource is applied
 * Resource capacity constraint removed from R16::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R16::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R16::wind as force_resource is applied
 * Resource capacity constraint removed from R17::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R17::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R17::wind as force_resource is applied
 * Resource capacity constraint removed from R18::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R18::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R18::wind as force_resource is applied
 * Resource capacity constraint removed from R1::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R1::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R1::wind as force_resource is applied
 * Resource capacity constraint removed from R2::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R2::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R2::wind as force_resource is applied
 * Resource capacity constraint removed from R3::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R3::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R3::wind as force_resource is applied
 * Resource capacity constraint removed from R4::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R4::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R4::wind as force_resource is applied
 * Resource capacity constraint removed from R5::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R5::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R5::wind as force_resource is applied
 * Resource capacity constraint removed from R6::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R6::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R6::wind as force_resource is applied
 * Resource capacity constraint removed from R7::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R7::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R7::wind as force_resource is applied
 * Resource capacity constraint removed from R8::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R8::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R8::wind as force_resource is applied
 * Resource capacity constraint removed from R9::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R9::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R9::wind as force_resource is applied
 * Resource capacity constraint removed from SARD::hydro_dam as force_resource is applied
 * Resource capacity constraint removed from SARD::hydro_ror as force_resource is applied
 * Resource capacity constraint removed from SARD::pv_farm as force_resource is applied
 * Resource capacity constraint removed from SARD::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from SARD::wind as force_resource is applied
 * Resource capacity constraint removed from SI::el_import as force_resource is applied
 * Resource capacity constraint removed from SICI::hydro_dam as force_resource is applied
 * Resource capacity constraint removed from SICI::hydro_ror as force_resource is applied
 * Resource capacity constraint removed from SICI::pv_farm as force_resource is applied
 * Resource capacity constraint removed from SICI::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from SICI::wind as force_resource is applied
 * Resource capacity constraint removed from SUD::hydro_dam as force_resource is applied
 * Resource capacity constraint removed from SUD::hydro_ror as force_resource is applied
 * Storage cannot be cyclic in operate run mode, setting `run.cyclic_storage` to False for this run
FLomb commented 5 years ago

The strange thing is that for some of these technologies (e.g. hydro_ror) the force_resource option is explicitly set to "False" in the .yaml

FLomb commented 5 years ago

P.S. : and all such warnings only appear in operation mode

brynpickering commented 5 years ago

Those warnings are from checks only triggered in operate mode. If 'force_resource' is applied then 'resource_cap' is a useless parameter. It used to be that the only useful parameter was 'resource_area' (resource_per_area), but we've now extended that to include 'energy_cap' (resource_per_cap), so obviously the check needs updating to look for this and not remove parameters from the optimisation if they will be used in a constraint.

Additionally, a comparison in the checks isn't working as expected (back to invalid value encountered in greater), which is likely leading to the removal of parameters even when 'force_resource' is False.

brynpickering commented 5 years ago

@FLomb Could you check the version on the branch associated with pull request #285, to see if it sorts this problem out for your particular case?

FLomb commented 5 years ago

@brynpickering unfortunately not. The 'Assertion Error' has disappeared, so the problem is constructed by pyomo and subsequently solved. Nonetheless, it results infeasible, apparently due to something still not being properly handled with resource/capacity constrains. All those warnings I was reporting, in fact, still appear.

brynpickering commented 5 years ago

@FLomb I think I know where that latter issue arose - have a go now!

FLomb commented 5 years ago

Ok, now the warnings changed a bit, but still some are present. In fact, I get now a Model Error. I paste here the new warnings:

`[2019-11-14 17:54:55] WARNING  d:\github\calliope\calliope\backend\checks.py:142: RuntimeWarning:

invalid value encountered in greater

[2019-11-14 17:55:05] WARNING  Warning: Possible issues found during model processing:
 * Energy capacity constraint removed from CNOR::demand_power as force_resource is applied and resource is not linked to energy flow (resource_unit = `energy`)
 * Energy capacity constraint removed from CSUD::demand_power as force_resource is applied and resource is not linked to energy flow (resource_unit = `energy`)
 * Energy capacity constraint removed from NORD::demand_power as force_resource is applied and resource is not linked to energy flow (resource_unit = `energy`)
 * Energy capacity constraint removed from SARD::demand_power as force_resource is applied and resource is not linked to energy flow (resource_unit = `energy`)
 * Energy capacity constraint removed from SICI::demand_power as force_resource is applied and resource is not linked to energy flow (resource_unit = `energy`)
 * Energy capacity constraint removed from SUD::demand_power as force_resource is applied and resource is not linked to energy flow (resource_unit = `energy`)
 * Resource capacity constraint removed from R10::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R10::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R10::wind as force_resource is applied
 * Resource capacity constraint removed from R11::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R11::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R11::wind as force_resource is applied
 * Resource capacity constraint removed from R12::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R12::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R12::wind as force_resource is applied
 * Resource capacity constraint removed from R13::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R13::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R13::wind as force_resource is applied
 * Resource capacity constraint removed from R14::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R14::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R14::wind as force_resource is applied
 * Resource capacity constraint removed from R15::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R15::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R15::wind as force_resource is applied
 * Resource capacity constraint removed from R16::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R16::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R16::wind as force_resource is applied
 * Resource capacity constraint removed from R17::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R17::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R17::wind as force_resource is applied
 * Resource capacity constraint removed from R18::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R18::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R18::wind as force_resource is applied
 * Resource capacity constraint removed from R1::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R1::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R1::wind as force_resource is applied
 * Resource capacity constraint removed from R2::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R2::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R2::wind as force_resource is applied
 * Resource capacity constraint removed from R3::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R3::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R3::wind as force_resource is applied
 * Resource capacity constraint removed from R4::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R4::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R4::wind as force_resource is applied
 * Resource capacity constraint removed from R5::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R5::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R5::wind as force_resource is applied
 * Resource capacity constraint removed from R6::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R6::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R6::wind as force_resource is applied
 * Resource capacity constraint removed from R7::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R7::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R7::wind as force_resource is applied
 * Resource capacity constraint removed from R8::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R8::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R8::wind as force_resource is applied
 * Resource capacity constraint removed from R9::pv_farm as force_resource is applied
 * Resource capacity constraint removed from R9::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from R9::wind as force_resource is applied
 * Resource capacity constraint removed from SARD::pv_farm as force_resource is applied
 * Resource capacity constraint removed from SARD::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from SARD::wind as force_resource is applied
 * Resource capacity constraint removed from SICI::pv_farm as force_resource is applied
 * Resource capacity constraint removed from SICI::pv_rooftop as force_resource is applied
 * Resource capacity constraint removed from SICI::wind as force_resource is applied
 * Storage cannot be cyclic in operate run mode, setting `run.cyclic_storage` to False for this run`

And the error:

`ModelError: Errors during model processing:
 * Operate mode: User must define a finite energy_cap (via energy_cap_equals or energy_cap_max) for AT::el_export
 * Operate mode: User must define a finite energy_cap (via energy_cap_equals or energy_cap_max) for CH::el_export
 * Operate mode: User must define a finite energy_cap (via energy_cap_equals or energy_cap_max) for FR::el_export
 * Operate mode: User must define a finite energy_cap (via energy_cap_equals or energy_cap_max) for GR::el_export
 * Operate mode: User must define a finite energy_cap (via energy_cap_equals or energy_cap_max) for R10::el_curtailment
 * Operate mode: User must define a finite energy_cap (via energy_cap_equals or energy_cap_max) for R11::el_curtailment
 * Operate mode: User must define a finite energy_cap (via energy_cap_equals or energy_cap_max) for R12::el_curtailment
 * Operate mode: User must define a finite energy_cap (via energy_cap_equals or energy_cap_max) for R13::el_curtailment
 * Operate mode: User must define a finite energy_cap (via energy_cap_equals or energy_cap_max) for R14::el_curtailment
 * Operate mode: User must define a finite energy_cap (via energy_cap_equals or energy_cap_max) for R15::el_curtailment
 * Operate mode: User must define a finite energy_cap (via energy_cap_equals or energy_cap_max) for R16::el_curtailment
 * Operate mode: User must define a finite energy_cap (via energy_cap_equals or energy_cap_max) for R17::el_curtailment
 * Operate mode: User must define a finite energy_cap (via energy_cap_equals or energy_cap_max) for R18::el_curtailment
 * Operate mode: User must define a finite energy_cap (via energy_cap_equals or energy_cap_max) for R1::el_curtailment
 * Operate mode: User must define a finite energy_cap (via energy_cap_equals or energy_cap_max) for R2::el_curtailment
 * Operate mode: User must define a finite energy_cap (via energy_cap_equals or energy_cap_max) for R3::el_curtailment
 * Operate mode: User must define a finite energy_cap (via energy_cap_equals or energy_cap_max) for R4::el_curtailment
 * Operate mode: User must define a finite energy_cap (via energy_cap_equals or energy_cap_max) for R5::el_curtailment
 * Operate mode: User must define a finite energy_cap (via energy_cap_equals or energy_cap_max) for R6::el_curtailment
 * Operate mode: User must define a finite energy_cap (via energy_cap_equals or energy_cap_max) for R7::el_curtailment
 * Operate mode: User must define a finite energy_cap (via energy_cap_equals or energy_cap_max) for R8::el_curtailment
 * Operate mode: User must define a finite energy_cap (via energy_cap_equals or energy_cap_max) for R9::el_curtailment
 * Operate mode: User must define a finite energy_cap (via energy_cap_equals or energy_cap_max) for SARD::el_curtailment
 * Operate mode: User must define a finite energy_cap (via energy_cap_equals or energy_cap_max) for SI::el_export
 * Operate mode: User must define a finite energy_cap (via energy_cap_equals or energy_cap_max) for SICI::el_curtailment`

Now, the error is strange because it refers to 'demand' techs, which are not conceived to define any energy_cap but rather a resource, which is of course defined and working perfectly in plan mode. Interestingly, the code is fine with the main demand tech whose resource is defined via a timseries, conversely to these other two, which are demand sinks defined by single resource values.

brynpickering commented 5 years ago

They all seem to be expected warnings / errors now. Demand technologies do sometimes have an energy capacity associated with them, certainly as a result of planning mode. So it's not really a problem. Your errors are caused by not setting a maximum capacity for those curtailment and export technologies, is it correct that you haven't defined an energy capacity for these?

FLomb commented 5 years ago

Of course I didn't define an 'energy_cap' for demand techs: that's not allowed as a constraint in the 'defaults.yaml'. What you are expected to define is a cap via the 'resource' constraint. Which I did define, not region by region but in the 'techs.yaml' file, and which works perfectly in plan mode. Why not in operate mode?

brynpickering commented 5 years ago

OK, I've cleaned up the checks to not comment on situations where the information is already NaN (i.e. for demand techs it will silently do NaN -> np.inf). But, technically, you could provide energy_cap information for demand techs, by starting with the output of a planning mode model, whereby the model_data has the completed decision variables, inc. energy_cap, and setting run.operation.use_cap_results: true.

As to the errors, although I think it would be good practice to always define an energy_cap for every technology, I have removed the check from situations where it being inf doesn't violate any constraints. How does it work now?

FLomb commented 4 years ago

Ok, so the model now runs without raising errors also in operate mode. But, unlike the plan mode, it becomes infeasible after a couple of iterations of the receding horizon. Based on my backend inspection, I am quite sure that this is still related to the things which seemed to go wrong in the first place. That is, resource constraints. In fact, all resource consumption constraints for techs which have resources defined through timeseries are characterised by 'None' values in the constraint limits, whilst the correct values appear when the same model is run in 'plan' mode. But resources themselves do show the correct values in the backend. So, somewhere between resource data handling and resource constraints construction there's a conditional statement that makes things different for the two run modes. I've tried to look at core/preprocess scripts, but I wasn't able to detect where this issue arises. If you have any more precise idea of where that could hide, I'm happy to further look into that. Thanks

brynpickering commented 4 years ago

Your new issue is probably not a Calliope problem, but rather a problem of how you set your solver options. Try removing all solver options and seeing if it runs with no infeasibility.

I'm closing this issue in light of the original issue being solved. If you continue having infeasibility issues @FLomb, just open a new issue!