MACRO runs after MESSAGE scenario solves "optimal with unscaled inf."

[OFR-IIASA]

After CPLEX finishes optimization of a scenario, with a status "optimal with unscaled infeasibilities", which with MESSAGE standalone would return an error (?), MACRO will still run and possibly even "converge". This will return an error after macro completes "successfully".

--- MESSAGE-MACRO_run.gms(5072) 631 Mb
--- GDX File c:\repo\message_ix\message_ix\model\output\MsgOutput_ENGAGE_SSP2_v4_EN_NPi2020_1200_step1.gdx
--- MESSAGE-MACRO_run.gms(5080) 631 Mb
+++ End of MESSAGEix-MACRO run - have a nice day! +++
*** Status: Normal completion
--- Job MESSAGE-MACRO_run.gms Stop 04/18/20 09:47:15 elapsed 0:20:53.408
2020-04-18 09:47:15,442 DEBUG at.ac.iiasa.ixmp.database.DbDAO:2041 - loading elements for set 'map_temporal_hierarchy' from database...
2020-04-18 09:47:15,446 DEBUG at.ac.iiasa.ixmp.database.DbDAO:2041 - loading elements for parameter 'duration_time' from database...
2020-04-18 09:47:15,486 DEBUG at.ac.iiasa.ixmp.objects.TimeSeries:109 - Cleaning cache...
2020-04-18 09:47:15,486 DEBUG at.ac.iiasa.ixmp.objects.TimeSeries:109 - Cleaning cache...
--- Warning: The GAMS version [24.9.1] differs from the API version [24.8.3].
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\Scripts\mix-data-script.py", line 11, in <module>
load_entry_point('message-data', 'console_scripts', 'mix-data')()
File "C:\ProgramData\Anaconda3\lib\site-packages\click\core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\click\core.py", line 717, in main
rv = self.invoke(ctx)
File "C:\ProgramData\Anaconda3\lib\site-packages\click\core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\ProgramData\Anaconda3\lib\site-packages\click\core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\ProgramData\Anaconda3\lib\site-packages\click\core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\ProgramData\Anaconda3\lib\site-packages\click\core.py", line 555, in invoke
return callback(*args, **kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\click\decorators.py", line 27, in new_func
return f(get_current_context().obj, *args, **kwargs)
File "c:\repo\message_data_clean_test_rep\message_data\projects\engage\__init__.py", line 73, in run
main(context)
File "c:\repo\message_data_clean_test_rep\message_data\projects\engage\runscript_main.py", line 120, in main
low_dem_scen='npi_low_dem_scen')
File "c:\repo\message_data_clean_test_rep\message_data\projects\engage\scenario_runner.py", line 342, in run
self.solve(rep_scen_name, copy_demands=True)
File "c:\repo\message_data_clean_test_rep\message_data\projects\engage\scenario_runner.py", line 247, in solve
var_list=['I', 'C', 'GDP'])
File "c:\repo\message_ix\message_ix\core.py", line 437, in solve
super().solve(model=model, solve_options=solve_options, **kwargs)
File "c:\repo\ixmp\ixmp\core.py", line 1490, in solve
model.run(self)
File "c:\repo\message_ix\message_ix\models.py", line 100, in run
result = super().run(scenario)
File "c:\repo\ixmp\ixmp\model\gams.py", line 154, in run
var_list=as_str_list(self.var_list) or [],
File "c:\repo\ixmp\ixmp\backend\jdbc.py", line 353, in read_file
self.jindex[ts].readSolutionFromGDX(*args)
jpype._jclass.IxException: at.ac.iiasa.ixmp.exceptions.IxException: This Scenario is not checked out - no changes to discard!
2020-04-18 09:47:15,634 INFO at.ac.iiasa.ixmp.Platform:157 - closed the connection to database 'jdbc:oracle:thin:@gp3.iiasa.ac.at:1521:GP3'

[khaeru]

@behnam-zakeri @volker-krey FYI—this seems to be a consequence of #223, and might possibly be taken up and addressed as part of #327.

The second line gives a clue as to which (model, scenario) combination produced this problem for @OFR-IIASA: MsgOutput_ENGAGE_SSP2_v4_EN_NPi2020_1200_step1.gdx—this may help in trying to reproduce, diagnose, and resolve the issue.

[OFR-IIASA]

One solution that has worked in the past was to use the cplex lpmethod 2 and then use the advanced bases. This has worked in the past, but the issue that I encountered now, is that solving the initial scenarios takes +10 hours (I stopped the solving process after 10 hours). This doesn't solve the actual problem described in the issue, but has served as a workaround to avoid unscaled infeasibilities.

[khaeru]

This doesn't solve the actual problem described in the issue, but has served as a workaround

Right.

Can you clarify what the desired behaviour is here? If MESSAGE solves to optimality with unscaled feasibilities, should MACRO run? Or should it stop with an error?

@zikolach is there any reason this should happen?

self.jindex[ts].readSolutionFromGDX(*args)
jpype._jclass.IxException: at.ac.iiasa.ixmp.exceptions.IxException: This Scenario is not checked out - no changes to discard!

[OFR-IIASA]

I believe that a scenario being optimal with unscaled infeasibilities returns an error for message standalone - @danielhuppmann is this true? If this is the case, then I believe that macro shouldnt be run if encountering unscaled infeasibilities, although I can be mistaken, as macro did converge in this case - possibly @volker-krey can also comment on the desired behavior.

[zikolach]

@khaeru There should some more information about error to be logged out (see here). But it also seems like an issue in the code - it should probably be checkOut(false); instead of db.setStatus(runId, LOCKED_IN_DB);

[OFR-IIASA]

In discussion with @volker-krey , @behnam-zakeri and @khaeru it was decided that the default behavior in such cases should be, that if MESSAGE is solved optimally with unscaled infeasibilities, then macro should NOT continue.

[Jihoon]

I believe that a scenario being optimal with unscaled infeasibilities returns an error for message standalone - @danielhuppmann is this true? If this is the case, then I believe that macro shouldnt be run if encountering unscaled infeasibilities, although I can be mistaken, as macro did converge in this case - possibly @volker-krey can also comment on the desired behavior.

I can confirm the MESSAGE standalone run does provide a java error.

Optimal solution found, but with infeasibilities after unscaling.
Simplex iterations after crossover:  319
​
Objective :     3702040.755052
​
--- Restarting execution
--- MESSAGE_run.gms(4419) 912 Mb
--- Reading solution for model MESSAGE_LP
--- MESSAGE_run.gms(4612) 1062 Mb
--- GDX File c:\users\min\message\message_ix\message_ix\model\output\MsgOutput_MESSAGEix-GLOBIOM_1.1-M-R12-NGFS_baseline_add_material.gdx
--- MESSAGE_run.gms(4614) 1062 Mb
    +++ End of MESSAGEix (stand-alone) run - have a nice day! +++
*** Status: Normal completion
--- Job MESSAGE_run.gms Stop 04/28/22 16:29:17 elapsed 0:02:56.998
--- Warning: The GAMS version [26.1.0] differs from the API version [24.8.3].
Traceback (most recent call last):
  File "MsgScenario.java", line 1126, in at.ac.iiasa.ixmp.objects.MsgScenario.readSolutionFromGDX
  File "MsgScenario.java", line 1240, in at.ac.iiasa.ixmp.objects.MsgScenario.readSolutionFromGDX
  File "TimeSeries.java", line 1017, in at.ac.iiasa.ixmp.objects.TimeSeries.discardChanges
  File "Scenario.java", line 2241, in at.ac.iiasa.ixmp.objects.Scenario.discardChanges
Exception: Java Exception
​
The above exception was the direct cause of the following exception:
​
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\envs\local_mix\Scripts\mix-models-script.py", line 33, in <module>
    sys.exit(load_entry_point('message-ix-models', 'console_scripts', 'mix-models')())
  File "C:\ProgramData\Anaconda3\envs\local_mix\lib\site-packages\click\core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\local_mix\lib\site-packages\click\core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "C:\ProgramData\Anaconda3\envs\local_mix\lib\site-packages\click\core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\ProgramData\Anaconda3\envs\local_mix\lib\site-packages\click\core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\ProgramData\Anaconda3\envs\local_mix\lib\site-packages\click\core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\ProgramData\Anaconda3\envs\local_mix\lib\site-packages\click\core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\local_mix\lib\site-packages\click\decorators.py", line 38, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "\\hdrive\home$\u045\min\mydocuments\message\message_data\message_data\projects\ngfs\__init__.py", line 118, in run
    main(context)
  File "\\hdrive\home$\u045\min\mydocuments\message\message_data\message_data\projects\ngfs\runscript_main.py", line 165, in main
    sc_macro = macro_COVID(
  File "\\hdrive\home$\u045\min\mydocuments\message\message_data\message_data\projects\ngfs\runscript_main.py", line 75, in macro_COVID
    sc.solve(model="MESSAGE", solve_options={'lpmethod': '4'})
  File "c:\users\min\message\message_ix\message_ix\core.py", line 566, in solve
    super().solve(model=model, solve_options=solve_options, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\local_mix\lib\site-packages\ixmp\core\scenario.py", line 836, in solve
    model_obj.run(self)
  File "c:\users\min\message\message_ix\message_ix\models.py", line 366, in run
    result = super().run(scenario)
  File "C:\ProgramData\Anaconda3\envs\local_mix\lib\site-packages\ixmp\model\gams.py", line 299, in run
    scenario.platform._backend.read_file(
  File "C:\ProgramData\Anaconda3\envs\local_mix\lib\site-packages\ixmp\backend\jdbc.py", line 524, in read_file
    self.jindex[ts].readSolutionFromGDX(*args)
at.ac.iiasa.ixmp.exceptions.at.ac.iiasa.ixmp.exceptions.IxException: at.ac.iiasa.ixmp.exceptions.IxException: This Scenario is not checked out - no changes to discard!

[adrivinca]

suggested solutions, which I have not tested

• using solve_options={'barcrossalg': '2'} to avoid unscaled infeasibilities

• adding scaind=1 in the 'cplex.opt' file, may allow a faster solution. see the CPLEX documentation

[gidden]

Quick note - you should be able to invoke scaid in the same way via solve_options

[khaeru]

you should be able to invoke scaid in the same way via solve_options

I confirm you can.

Basically I think the issue is here: https://github.com/iiasa/message_ix/blob/51ccdfb631e815bc4142df45650f908900c9dbf6/message_ix/model/MESSAGE/model_solve.gms#L29-L40

The conditional is not catching whatever attributes of MESSAGE_LP reflect this "optimal with unscaled infeasibilities", and so it does not abort.

[OFR-IIASA]

After playing with some alternatives, I have the following solution to counteract the issue. I have added a few lines to generate a second "cplex.opt" file as "cplex.op2" in "models.py", which is copy of the first with the addition of the argument "barcrossalg = 2". In "MESSAGE-MACRO_run.gms", there is now an if-statement that checks for the solve-status of MESSAGELP. Unscaled infeasibilities have the status 5; upon encountering this, the optfile used by MESSAGELP will be the latter of the two, which includes the barcossalg option. I have tried other solutions, but cplex parameters cannot be directly edited in a loop, hence the only option I have found is to switch to a second cplex options file. The changes will be added to PR #645 .