FFiand commented 1 year ago

1) Preliminaries

[x] Create point files for (internal) set of test instances
[x] Adjust code to allow model generation without solving
- [x] Model attribute .JustScrDir=1
- [x] execute_loadpoint of point file
[x] Test Automation: Batch script to run the (internal) set of test instances with the following features enabled
- [x] profile
- [x] procTreeMemMonitor
- [x] reference
- [x] write combined profile summary to single file
- [x] write combined memory high water mark report to single file

2) Evaluation of baseline runs

[x] Identification of slow assignments
[x] Identification of slow equations
[x] evaluation of solvelink impact. This is not only relevant for communication with the solver. The solve statement can be seen as a big operation against the internal database. E.g. solvelink 0 vs 5 can have an impact on scattered memory which will

3) Investigation of potential Improvements

[x] Performance
- [x] If 2) revealed slow assignments, make them fast
- [x] If 2) revealed slow equations, make them fast
- [x] impact of solvelink: not only relevant for communication with the solver but can also have an impact on execution after solve
[x] Memory
- [x] investigate potential to reduce memory high water mark by clearing symbols which are no longer needed at execution time

JBroihan commented 9 months ago

Evaluation of Base Line / Ground Truth runs

Instance	Model Status	Elapsed Time [s]	Total Seconds [s]	Compilation Time [s]	Execution Time [s]	Generation Time [s]	Solver Time [s]	Startup Time [s]	Closedown Time [s]	Highwater RSS [GB]	Highwater VSS [GB]	# Rows	# Columns	# Non Zeros	# Non Zeros per s	RHS [min, max]	Bound [min, max]	Matrix [min, max]
1p5C_OS_SSP2	FEASIBLE SOLUTION	75.11	70.16	1.75	40.36	13.39	28.03	0.02	0.00	2.85	3.20	624,436	863,666	4,650,991	347,322.16	9.222E-05, 3.258E+06	7.001E-06, 6.612E+07	6.977E-09, 2.535E+06
BAU_IND_V2	OPTIMAL	7,753.52	7,740.36	2.52	166.44	113.83	7,571.39	0.02	0.00	22.81	26.23	6,750,106	6,399,364	40,479,629	355,621.02	1.566E-14, 2.285E+05	3.516E-09, 5.000E+04	1.623E-07, 3.352E+06
BEAM-ME_Instances/672_11_8	OPTIMAL	13,005.10	12,979.09	0.56	363.55	251.36	12,614.98	0.00	0.00	62.67	70.11	21,108,711	15,111,159	124,095,043	493,694.47	4.097E-06, 5.940E+05	2.680E-02, 2.676E+05	1.882E-04, 1.663E+04
BEAM-ME_Instances/672_22_8	INTERMEDIATE INFEASIBLE	43,946.87	43,891.17	0.97	653.31	432.31	43,236.88	0.01	0.00	112.88	163.87	37,645,140	27,331,345	218,532,379	505,495.74	1.405E-06, 5.940E+05	8.298E-03, 2.676E+05	2.675E-04, 1.663E+04
BEAM-ME_Instances/2016_22_8	ERROR NO SOLUTION	3,330.59	1,969.62	2.09	1,930.45	1,752.70	1,201.00	17.39	19.69	115.10	215.29	112,857,882	81,947,383	658,167,482	375,515.69	2.460E-07, 5.940E+05	8.298E-03, 2.676E+05	1.946E-06, 1.663E+04
E4SMA	FEASIBLE SOLUTION	1,379.17	1,360.83	1.89	140.36	56.80	1,218.56	0.02	0.00	17.91	19.95	4,229,763	4,437,740	32,382,036	570,136.38	7.402E-16, 4.407E+06	9.557E-06, 1.000E+09	1.258E-09, 1.610E+06
GAZNAT_V1_Instance	FEASIBLE SOLUTION	15,483.52	15,470.69	3.59	137.92	93.22	15,329.16	0.02	0.00	19.17	20.62	5,336,684	4,555,729	34,188,921	366,759.15	1.566E-14, 2.291E+05	3.516E-09, 3.000E+06	1.623E-07, 3.352E+06
POLIZERO_V1_Instance	FEASIBLE SOLUTION	2,866.62	2,845.72	4.83	236.00	114.53	2,604.88	0.01	0.00	22.65	24.69	5,640,942	6,991,820	48,190,465	420,763.51	7.918E-19, 5.440E+05	3.837E-19, 1.000E+07	7.309E-08, 4.178E+06
POLIZERO_V2_Instance	FEASIBLE SOLUTION	5,111.24	5,064.34	4.27	420.73	192.95	4,639.33	0.01	0.00	36.28	40.14	10,540,301	13,847,195	69,064,648	357,935.08	1.918E-18, 8.825E+06	1.823E-19, 2.919E+07	9.620E-19, 2.269E+06
TIMES-DK-Instance	OPTIMAL	149.12	147.76	2.12	16.14	7.94	129.50	0.00	0.00	2.65	4.01	867,042	1,031,037	4,740,351	597,247.20	3.577E-08, 7.453E+04	4.606E-05, 4.500E+04	1.909E-09, 4.282E+06
UK-TIMES	OPTIMAL	101.42	99.08	0.53	20.06	11.23	78.48	0.00	0.00	3.03	3.29	875,780	932,609	6,883,510	612,684.47	4.659E-05, 6.697E+05	6.000E-04, 5.867E+05	2.793E-11, 3.746E+05

JBroihan commented 9 months ago

Identification of Slow Equations and Assignments

The following table shows the sum of time spend in an equation/assignment across all benchmarked instances (Total Time [s]) as well as the average percentage of time spend in an equation/assignment relative to the execution time. Based on the amount of time GAMS spends in specific equations/assignments bottlenecks can be identified and reviewed with higher priority.

$s \in S$: set of statements (equations and assignments) $i \in I$: set of benchmarked instances $StatementTime_{s,i}$: time spend in statement $s$ for instance $i$ $ExecutionTime_i$: execution time of instance $i$

$Total Times = \sum{i \in I} StatementTime_{s,i}$ $RelativeTimes = (\sum{i \in I} \frac{StatementTime_{s,i}}{ExecutionTime_i}$ * 100}) / |I|

Description	Total Time [s]	relative to Execution Time [%]
Equation EQL_ACTUPC	268.93	27.44
Equation EQE_UCRS	28.44	18.87
Equation EQE_COMBAL	87.74	12.30
Equation EQE_UCT	28.56	12.10
Equation EQ_ACTRAMP	83.19	7.71
Equation EQE_UCRTS	23.78	7.69
Equation EQG_COMBAL	8.09	3.51
Equation EQ_PEAK	21.63	2.84
Equation EQ_OBJVAR	4.60	1.99
Equation EQL_CAPACT	11.24	1.97
Equation EQE_ACTEFF	9.17	1.90

JBroihan commented 9 months ago

Evaluation of `SolveLink` Impact

The SolveLink option specifies what solver linking conventions are used when GAMS executes a solve statement.

SolveLink=2 (default): Model instance saved to scratch directory, the solver is called with a spawn (if possible) or a shell (if spawn is not possible) while GAMS remains open - If this is not supported by the selected solver, it gets reset to 1 automatically.

SolveLink=0: Model instance and entire GAMS state saved to scratch directory, GAMS exits (and vacates memory), and the solver script is called. After the solver terminates, GAMS restarts from the saved state and continues to executing. Is often beneficial for memory intensive models.

SolveLink=5: The model instance is passed to the solver in-memory - If this is not supported by the selected solver, it gets reset to 2 automatically. Is memory intensive but beneficial if multiple solves are executed.

While the results show differences for the time spend in different parts of the solve process (e.g. compilation, execution, startup, etc.) depending on the SolveLink value, we do not observe an impact on the total elapsed time for both SolveLink=0 and SolveLink=5 compared to the default SolveLink=2. However, we see a significant reduction in memory consumption (Highwater RSS and VSS) of ~22% by setting SolveLink=0.

Relative Difference obtained by `SolveLink=0`

Instance	Elapsed Time [%]	Total Seconds [%]	Compilation Time [%]	Execution Time [%]	Generation Time [%]	Solver Time [%]	Startup Time [%]	Closedown Time [%]	Highwater RSS [%]	Highwater VSS [%]	# Non Zeros per s [%]
1p5C_OS_SSP2	-0.13	-43.21	-9.83	-10.22	-0.46	2.34	7,425.00	inf	-47.02	-45.94	0.47
BAU_IND_V2	-1.02	-98.08	27.36	-15.17	0.71	-0.94	13,868.75	inf	-18.06	-15.75	-0.71
BEAM-ME_Instances/672_11_8	-0.80	-97.75	25.09	-21.64	-0.29	-0.58	inf	inf	-15.59	-15.28	0.29
BEAM-ME_Instances/672_22_8	-0.16	-98.85	1.65	-25.29	2.74	-0.01	48,960.00	inf	1.40	-11.61	-2.67
E4SMA	0.07	-91.21	-28.93	-18.23	-0.61	1.13	12,006.25	inf	-21.44	-19.35	0.61
GAZNAT_V1_Instance	-1.13	-99.20	-0.03	-14.99	1.24	-1.10	11,518.75	inf	-17.21	-16.00	-1.22
POLIZERO_V1_Instance	-0.13	-92.47	12.30	-14.61	1.28	0.32	28,440.00	inf	-29.32	-26.97	-1.27
POLIZERO_V2_Instance	-0.55	-92.72	1.45	-15.91	-0.48	0.08	39,273.33	inf	-28.83	-26.21	0.48
TIMES-DK-Instance	-0.63	-89.33	-8.09	-17.81	0.00	0.05	inf	inf	-23.40	-15.21	0.00
UK-TIMES	-1.62	-82.71	-5.84	-20.17	-2.65	-0.24	inf	inf	-26.73	-24.62	2.72
mean	-0.61	-88.55	1.51	-17.40	0.15	0.10	inf	inf	-22.62	-21.69	-0.13

Relative Difference obtained by `SolveLink=5`

Instance	Elapsed Time [%]	Total Seconds [%]	Compilation Time [%]	Execution Time [%]	Generation Time [%]	Solver Time [%]	Startup Time [%]	Closedown Time [%]	Highwater RSS [%]	Highwater VSS [%]	# Non Zeros per s [%]
1p5C_OS_SSP2	0.11	1.49	0.00	-2.24	-0.23	7.02	-100.00	nan	0.35	1.56	0.23
BAU_IND_V2	-0.76	-0.76	25.49	-3.57	1.63	-0.71	-6.25	nan	-0.13	0.00	-1.61
BEAM-ME_Instances/672_11_8	-1.34	-1.36	27.94	-7.41	0.19	-1.18	inf	nan	2.74	1.18	-0.19
BEAM-ME_Instances/672_22_8	-0.08	-0.09	0.00	-6.13	2.74	0.00	-100.00	nan	1.53	0.51	-2.67
E4SMA	0.41	0.40	22.32	-5.74	-1.95	1.07	93.75	nan	-0.73	-0.70	1.99
GAZNAT_V1_Instance	-0.74	-0.74	0.45	-4.73	1.19	-0.71	0.00	nan	-0.21	-0.15	-1.18
POLIZERO_V1_Instance	-1.59	-1.62	8.41	-2.26	0.38	-1.58	0.00	nan	-1.02	-0.93	-0.38
POLIZERO_V2_Instance	-0.01	-0.06	1.10	-2.99	1.08	0.21	6.67	nan	0.61	-0.60	-1.07
TIMES-DK-Instance	-0.08	-0.09	-7.34	-4.06	0.79	0.51	inf	nan	-0.75	-0.25	-0.79
UK-TIMES	-1.26	-1.26	64.78	-7.86	-2.65	-0.02	nan	nan	-0.66	-0.61	2.72
mean	-0.53	-0.41	14.31	-4.70	0.32	0.46	inf	nan	0.17	0.00	-0.30

New `SolveLink=5`

Instance	Model Status	Elapsed Time [s]	Total Seconds [s]	Compilation Time [s]	Execution Time [s]	Generation Time [s]	Solver Time [s]	Startup Time [s]	Highwater RSS [GB]	Highwater VSS [GB]	# Rows	# Columns	# Non Zeros	# Non Zeros per s	RHS [min, max]	Bound [min, max]	Matrix [min, max]
1p5C_OS_SSP2	FEASIBLE SOLUTION	75.19	71.20	1.75	39.45	13.36	30.00	0.00	2.86	3.25	624,436	863,666	4,650,991	348,128.07	9.222E-05, 3.258E+06	7.001E-06, 6.612E+07	6.977E-09, 2.535E+06
BAU_IND_V2	OPTIMAL	7,694.53	7,681.22	3.16	160.50	115.69	7,517.55	0.01	22.78	26.23	6,750,106	6,399,364	40,479,629	349,906.46	1.566E-14, 2.285E+05	3.516E-09, 5.000E+04	1.623E-07, 3.352E+06
BEAM-ME_Instances/672_11_8	OPTIMAL	12,831.06	12,803.19	0.72	336.61	251.84	12,465.84	0.02	64.39	70.94	21,108,711	15,111,159	124,095,043	492,747.64	4.097E-06, 5.940E+05	2.680E-02, 2.676E+05	1.882E-04, 1.663E+04
BEAM-ME_Instances/672_22_8	INTERMEDIATE INFEASIBLE	43,913.10	43,852.42	0.97	613.30	444.17	43,238.16	0.00	114.61	164.71	37,645,140	27,331,345	218,532,379	491,999.45	1.405E-06, 5.940E+05	8.298E-03, 2.676E+05	2.675E-04, 1.663E+04
E4SMA	FEASIBLE SOLUTION	1,384.83	1,366.25	2.31	132.30	55.69	1,231.61	0.03	17.78	19.81	4,229,763	4,437,740	32,382,036	581,500.82	7.402E-16, 4.407E+06	9.557E-06, 1.000E+09	1.258E-09, 1.610E+06
GAZNAT_V1_Instance	FEASIBLE SOLUTION	15,368.92	15,355.77	3.61	131.40	94.33	15,220.74	0.02	19.13	20.59	5,336,684	4,555,729	34,188,921	362,447.22	1.566E-14, 2.291E+05	3.516E-09, 3.000E+06	1.623E-07, 3.352E+06
POLIZERO_V1_Instance	FEASIBLE SOLUTION	2,821.12	2,799.70	5.23	230.66	114.97	2,563.80	0.01	22.42	24.46	5,640,942	6,991,820	48,190,465	419,160.51	7.918E-19, 5.440E+05	3.837E-19, 1.000E+07	7.309E-08, 4.178E+06
POLIZERO_V2_Instance	FEASIBLE SOLUTION	5,110.54	5,061.44	4.31	408.17	195.03	4,648.94	0.02	36.50	39.90	10,540,301	13,847,195	69,064,648	354,121.39	1.918E-18, 8.825E+06	1.823E-19, 2.919E+07	9.620E-19, 2.269E+06
TIMES-DK-Instance	OPTIMAL	149.00	147.62	1.97	15.48	8.00	130.16	0.02	2.63	4.00	867,042	1,031,037	4,740,351	592,543.88	3.577E-08, 7.453E+04	4.606E-05, 4.500E+04	1.909E-09, 4.282E+06
UK-TIMES	OPTIMAL	100.14	97.83	0.88	18.49	10.94	78.47	0.00	3.01	3.27	875,780	932,609	6,883,510	629,378.26	4.659E-05, 6.697E+05	6.000E-04, 5.867E+05	2.793E-11, 3.746E+05

New `SolveLink=0`

Instance	Model Status	Elapsed Time [s]	Total Seconds [s]	Compilation Time [s]	Execution Time [s]	Generation Time [s]	Solver Time [s]	Startup Time [s]	Closedown Time [s]	Highwater RSS [GB]	Highwater VSS [GB]	# Rows	# Columns	# Non Zeros	# Non Zeros per s	RHS [min, max]	Bound [min, max]	Matrix [min, max]
1p5C_OS_SSP2	FEASIBLE SOLUTION	75.01	39.84	1.58	36.23	13.33	28.69	1.20	0.83	1.51	1.73	624,436	863,666	4,650,991	348,937.73	9.222E-05, 3.258E+06	7.001E-06, 6.612E+07	6.977E-09, 2.535E+06
BAU_IND_V2	OPTIMAL	7,674.73	148.41	3.20	141.19	114.64	7,500.16	2.23	1.78	18.69	22.10	6,750,106	6,399,364	40,479,629	353,099.06	1.566E-14, 2.285E+05	3.516E-09, 5.000E+04	1.623E-07, 3.352E+06
BEAM-ME_Instances/672_11_8	OPTIMAL	12,900.51	292.37	0.70	284.89	250.64	12,542.28	3.20	3.58	52.90	59.40	21,108,711	15,111,159	124,095,043	495,112.68	4.097E-06, 5.940E+05	2.680E-02, 2.676E+05	1.882E-04, 1.663E+04
BEAM-ME_Instances/672_22_8	INTERMEDIATE INFEASIBLE	43,875.57	503.06	0.98	488.11	444.16	43,231.28	7.36	6.61	114.46	144.84	37,645,140	27,331,345	218,532,379	492,017.17	1.405E-06, 5.940E+05	8.298E-03, 2.676E+05	2.675E-04, 1.663E+04
E4SMA	FEASIBLE SOLUTION	1,380.10	119.56	1.34	114.77	56.45	1,232.34	1.94	1.51	14.07	16.09	4,229,763	4,437,740	32,382,036	573,610.54	7.402E-16, 4.407E+06	9.557E-06, 1.000E+09	1.258E-09, 1.610E+06
GAZNAT_V1_Instance	FEASIBLE SOLUTION	15,308.41	124.12	3.59	117.25	94.38	15,160.20	1.86	1.42	15.87	17.32	5,336,684	4,555,729	34,188,921	362,266.71	1.566E-14, 2.291E+05	3.516E-09, 3.000E+06	1.623E-07, 3.352E+06
POLIZERO_V1_Instance	FEASIBLE SOLUTION	2,862.76	214.38	5.42	201.52	116.00	2,613.12	4.28	3.16	16.01	18.03	5,640,942	6,991,820	48,190,465	415,435.04	7.918E-19, 5.440E+05	3.837E-19, 1.000E+07	7.309E-08, 4.178E+06
POLIZERO_V2_Instance	FEASIBLE SOLUTION	5,083.34	368.69	4.33	353.80	192.03	4,642.94	5.91	4.66	25.82	29.62	10,540,301	13,847,195	69,064,648	359,653.64	1.918E-18, 8.825E+06	1.823E-19, 2.919E+07	9.620E-19, 2.269E+06
TIMES-DK-Instance	OPTIMAL	148.18	15.77	1.95	13.27	7.94	129.56	0.31	0.23	2.03	3.40	867,042	1,031,037	4,740,351	597,247.20	3.577E-08, 7.453E+04	4.606E-05, 4.500E+04	1.909E-09, 4.282E+06
UK-TIMES	OPTIMAL	99.78	17.13	0.50	16.02	10.94	78.30	0.33	0.28	2.22	2.48	875,780	932,609	6,883,510	629,378.26	4.659E-05, 6.697E+05	6.000E-04, 5.867E+05	2.793E-11, 3.746E+05

Ground Truth `SolveLink=2`

Instance	Model Status	Elapsed Time [s]	Total Seconds [s]	Compilation Time [s]	Execution Time [s]	Generation Time [s]	Solver Time [s]	Startup Time [s]	Closedown Time [s]	Highwater RSS [GB]	Highwater VSS [GB]	# Rows	# Columns	# Non Zeros	# Non Zeros per s	RHS [min, max]	Bound [min, max]	Matrix [min, max]
1p5C_OS_SSP2	FEASIBLE SOLUTION	75.11	70.16	1.75	40.36	13.39	28.03	0.02	0.00	2.85	3.20	624,436	863,666	4,650,991	347,322.16	9.222E-05, 3.258E+06	7.001E-06, 6.612E+07	6.977E-09, 2.535E+06
BAU_IND_V2	OPTIMAL	7,753.52	7,740.36	2.52	166.44	113.83	7,571.39	0.02	0.00	22.81	26.23	6,750,106	6,399,364	40,479,629	355,621.02	1.566E-14, 2.285E+05	3.516E-09, 5.000E+04	1.623E-07, 3.352E+06
BEAM-ME_Instances/672_11_8	OPTIMAL	13,005.10	12,979.09	0.56	363.55	251.36	12,614.98	0.00	0.00	62.67	70.11	21,108,711	15,111,159	124,095,043	493,694.47	4.097E-06, 5.940E+05	2.680E-02, 2.676E+05	1.882E-04, 1.663E+04
BEAM-ME_Instances/672_22_8	INTERMEDIATE INFEASIBLE	43,946.87	43,891.17	0.97	653.31	432.31	43,236.88	0.01	0.00	112.88	163.87	37,645,140	27,331,345	218,532,379	505,495.74	1.405E-06, 5.940E+05	8.298E-03, 2.676E+05	2.675E-04, 1.663E+04
BEAM-ME_Instances/2016_22_8	ERROR NO SOLUTION	3,330.59	1,969.62	2.09	1,930.45	1,752.70	1,201.00	17.39	19.69	115.10	215.29	112,857,882	81,947,383	658,167,482	375,515.69	2.460E-07, 5.940E+05	8.298E-03, 2.676E+05	1.946E-06, 1.663E+04
E4SMA	FEASIBLE SOLUTION	1,379.17	1,360.83	1.89	140.36	56.80	1,218.56	0.02	0.00	17.91	19.95	4,229,763	4,437,740	32,382,036	570,136.38	7.402E-16, 4.407E+06	9.557E-06, 1.000E+09	1.258E-09, 1.610E+06
GAZNAT_V1_Instance	FEASIBLE SOLUTION	15,483.52	15,470.69	3.59	137.92	93.22	15,329.16	0.02	0.00	19.17	20.62	5,336,684	4,555,729	34,188,921	366,759.15	1.566E-14, 2.291E+05	3.516E-09, 3.000E+06	1.623E-07, 3.352E+06
POLIZERO_V1_Instance	FEASIBLE SOLUTION	2,866.62	2,845.72	4.83	236.00	114.53	2,604.88	0.01	0.00	22.65	24.69	5,640,942	6,991,820	48,190,465	420,763.51	7.918E-19, 5.440E+05	3.837E-19, 1.000E+07	7.309E-08, 4.178E+06
POLIZERO_V2_Instance	FEASIBLE SOLUTION	5,111.24	5,064.34	4.27	420.73	192.95	4,639.33	0.01	0.00	36.28	40.14	10,540,301	13,847,195	69,064,648	357,935.08	1.918E-18, 8.825E+06	1.823E-19, 2.919E+07	9.620E-19, 2.269E+06
TIMES-DK-Instance	OPTIMAL	149.12	147.76	2.12	16.14	7.94	129.50	0.00	0.00	2.65	4.01	867,042	1,031,037	4,740,351	597,247.20	3.577E-08, 7.453E+04	4.606E-05, 4.500E+04	1.909E-09, 4.282E+06
UK-TIMES	OPTIMAL	101.42	99.08	0.53	20.06	11.23	78.48	0.00	0.00	3.03	3.29	875,780	932,609	6,883,510	612,684.47	4.659E-05, 6.697E+05	6.000E-04, 5.867E+05	2.793E-11, 3.746E+05

JBroihan commented 8 months ago

Freegamsmodel

While the following analysis does not focus on performance metrics, it is important to note that the following findings are memory-related. We include them for completeness, as memory management is crucial, particularly for handling large instances.

freegamsmodel (boolean): preserves memory by dumping the GAMS model instance representation temporarily to disk

In order to provide the maximum amount of memory to the solver this option dumps the internal representation of the model instance temporarily to disk and frees memory. This option only works with SolveLink=0 and only for models without quadratic constraints.

Based on the results below, we obtain an additional reduction (on top of SolveLink=0) on the Highwater memory footprint of ~15% by setting freegamsmodel 1.

Relative Difference obtained by `freegamsmodel 1`

Instance	Elapsed Time [%]	Total Seconds [%]	Compilation Time [%]	Execution Time [%]	Generation Time [%]	Solver Time [%]	Startup Time [%]	Closedown Time [%]	Highwater RSS [%]	Highwater VSS [%]	# Non Zeros per s [%]
1p5C_OS_SSP2	0.02	-2.35	-26.68	-1.34	0.59	3.65	-0.08	-3.74	0.66	-12.72	-0.58
BAU_IND_V2	-3.43	-1.96	-19.51	-1.64	-1.88	-3.45	4.16	-3.48	-20.39	-17.24	1.92
BEAM-ME_Instances/672_11_8	-1.64	-2.07	77.81	-2.34	-2.29	-1.60	5.37	-3.49	-20.45	-18.69	2.35
BEAM-ME_Instances/672_22_8	0.01	-3.08	3.15	-3.44	-3.53	0.10	22.72	-5.90	3.07	-1.26	3.66
E4SMA	-2.57	-2.03	96.50	-3.26	-2.94	-2.62	3.25	-3.04	0.00	0.00	3.02
GAZNAT_V1_Instance	-4.15	-3.59	-31.73	-2.80	-2.96	-4.15	0.91	-3.31	-19.03	-17.90	3.05
POLIZERO_V1_Instance	-1.27	-3.86	-38.03	-3.03	-2.96	-1.04	-0.37	-2.98	-24.98	-22.80	3.05
POLIZERO_V2_Instance	-3.71	-2.78	-32.14	-2.54	-2.62	-3.77	1.61	0.64	-25.95	-23.02	2.69
TIMES-DK-Instance	2.60	1.88	32.00	-2.47	-2.36	2.85	5.13	-6.81	-24.63	-15.29	2.41
UK-TIMES	-0.81	-0.10	65.60	-2.25	-1.85	-0.88	4.88	-0.35	0.90	1.61	1.88
mean	-1.50	-1.99	12.70	-2.51	-2.28	-1.09	4.76	-3.25	-13.08	-12.73	2.34

`SL=0` & `freegamsmodel`

Instance	Model Status	Elapsed Time [s]	Total Seconds [s]	Compilation Time [s]	Execution Time [s]	Generation Time [s]	Solver Time [s]	Startup Time [s]	Closedown Time [s]	Highwater RSS [GB]	Highwater VSS [GB]	# Rows	# Columns	# Non Zeros	# Non Zeros per s	RHS [min, max]	Bound [min, max]	Matrix [min, max]
1p5C_OS_SSP2	FEASIBLE SOLUTION	75.03	38.91	1.16	35.75	13.41	29.73	1.20	0.80	1.52	1.51	624,436	863,666	4,650,991	346,907.66	9.222E-05, 3.258E+06	7.001E-06, 6.612E+07	6.977E-09, 2.535E+06
BAU_IND_V2	OPTIMAL	7,411.48	145.50	2.58	138.88	112.48	7,241.16	2.33	1.72	14.88	18.29	6,750,106	6,399,364	40,479,629	359,870.11	1.566E-14, 2.285E+05	3.516E-09, 5.000E+04	1.623E-07, 3.352E+06
BEAM-ME_Instances/672_11_8	OPTIMAL	12,688.61	286.31	1.25	278.23	244.89	12,341.19	3.38	3.45	42.08	48.30	21,108,711	15,111,159	124,095,043	506,735.83	4.097E-06, 5.940E+05	2.680E-02, 2.676E+05	1.882E-04, 1.663E+04
BEAM-ME_Instances/672_22_8	INTERMEDIATE INFEASIBLE	43,880.30	487.56	1.02	471.30	428.49	43,273.92	9.03	6.22	117.97	143.02	37,645,140	27,331,345	218,532,379	510,011.74	1.405E-06, 5.940E+05	8.298E-03, 2.676E+05	2.675E-04, 1.663E+04
E4SMA	FEASIBLE SOLUTION	1,344.68	117.14	2.64	111.03	54.80	1,200.06	2.00	1.47	14.07	16.09	4,229,763	4,437,740	32,382,036	590,956.20	7.402E-16, 4.407E+06	9.557E-06, 1.000E+09	1.258E-09, 1.610E+06
GAZNAT_V1_Instance	FEASIBLE SOLUTION	14,673.24	119.67	2.45	113.97	91.58	14,531.33	1.88	1.38	12.85	14.22	5,336,684	4,555,729	34,188,921	373,331.16	1.566E-14, 2.291E+05	3.516E-09, 3.000E+06	1.623E-07, 3.352E+06
POLIZERO_V1_Instance	FEASIBLE SOLUTION	2,826.45	206.09	3.36	195.41	112.56	2,585.84	4.26	3.06	12.01	13.92	5,640,942	6,991,820	48,190,465	428,119.94	7.918E-19, 5.440E+05	3.837E-19, 1.000E+07	7.309E-08, 4.178E+06
POLIZERO_V2_Instance	FEASIBLE SOLUTION	4,894.56	358.42	2.94	344.80	187.00	4,467.78	6.00	4.69	19.12	22.80	10,540,301	13,847,195	69,064,648	369,329.67	1.918E-18, 8.825E+06	1.823E-19, 2.919E+07	9.620E-19, 2.269E+06
TIMES-DK-Instance	OPTIMAL	152.04	16.06	2.58	12.94	7.75	133.25	0.33	0.22	1.53	2.88	867,042	1,031,037	4,740,351	611,658.19	3.577E-08, 7.453E+04	4.606E-05, 4.500E+04	1.909E-09, 4.282E+06
UK-TIMES	OPTIMAL	98.97	17.11	0.83	15.66	10.73	77.61	0.34	0.28	2.24	2.52	875,780	932,609	6,883,510	641,221.24	4.659E-05, 6.697E+05	6.000E-04, 5.867E+05	2.793E-11, 3.746E+05

`SL=0`

Instance	Model Status	Elapsed Time [s]	Total Seconds [s]	Compilation Time [s]	Execution Time [s]	Generation Time [s]	Solver Time [s]	Startup Time [s]	Closedown Time [s]	Highwater RSS [GB]	Highwater VSS [GB]	# Rows	# Columns	# Non Zeros	# Non Zeros per s	RHS [min, max]	Bound [min, max]	Matrix [min, max]
1p5C_OS_SSP2	FEASIBLE SOLUTION	75.01	39.84	1.58	36.23	13.33	28.69	1.20	0.83	1.51	1.73	624,436	863,666	4,650,991	348,937.73	9.222E-05, 3.258E+06	7.001E-06, 6.612E+07	6.977E-09, 2.535E+06
BAU_IND_V2	OPTIMAL	7,674.73	148.41	3.20	141.19	114.64	7,500.16	2.23	1.78	18.69	22.10	6,750,106	6,399,364	40,479,629	353,099.06	1.566E-14, 2.285E+05	3.516E-09, 5.000E+04	1.623E-07, 3.352E+06
BEAM-ME_Instances/672_11_8	OPTIMAL	12,900.51	292.37	0.70	284.89	250.64	12,542.28	3.20	3.58	52.90	59.40	21,108,711	15,111,159	124,095,043	495,112.68	4.097E-06, 5.940E+05	2.680E-02, 2.676E+05	1.882E-04, 1.663E+04
BEAM-ME_Instances/672_22_8	INTERMEDIATE INFEASIBLE	43,875.57	503.06	0.98	488.11	444.16	43,231.28	7.36	6.61	114.46	144.84	37,645,140	27,331,345	218,532,379	492,017.17	1.405E-06, 5.940E+05	8.298E-03, 2.676E+05	2.675E-04, 1.663E+04
E4SMA	FEASIBLE SOLUTION	1,380.10	119.56	1.34	114.77	56.45	1,232.34	1.94	1.51	14.07	16.09	4,229,763	4,437,740	32,382,036	573,610.54	7.402E-16, 4.407E+06	9.557E-06, 1.000E+09	1.258E-09, 1.610E+06
GAZNAT_V1_Instance	FEASIBLE SOLUTION	15,308.41	124.12	3.59	117.25	94.38	15,160.20	1.86	1.42	15.87	17.32	5,336,684	4,555,729	34,188,921	362,266.71	1.566E-14, 2.291E+05	3.516E-09, 3.000E+06	1.623E-07, 3.352E+06
POLIZERO_V1_Instance	FEASIBLE SOLUTION	2,862.76	214.38	5.42	201.52	116.00	2,613.12	4.28	3.16	16.01	18.03	5,640,942	6,991,820	48,190,465	415,435.04	7.918E-19, 5.440E+05	3.837E-19, 1.000E+07	7.309E-08, 4.178E+06
POLIZERO_V2_Instance	FEASIBLE SOLUTION	5,083.34	368.69	4.33	353.80	192.03	4,642.94	5.91	4.66	25.82	29.62	10,540,301	13,847,195	69,064,648	359,653.64	1.918E-18, 8.825E+06	1.823E-19, 2.919E+07	9.620E-19, 2.269E+06
TIMES-DK-Instance	OPTIMAL	148.18	15.77	1.95	13.27	7.94	129.56	0.31	0.23	2.03	3.40	867,042	1,031,037	4,740,351	597,247.20	3.577E-08, 7.453E+04	4.606E-05, 4.500E+04	1.909E-09, 4.282E+06
UK-TIMES	OPTIMAL	99.78	17.13	0.50	16.02	10.94	78.30	0.33	0.28	2.22	2.48	875,780	932,609	6,883,510	629,378.26	4.659E-05, 6.697E+05	6.000E-04, 5.867E+05	2.793E-11, 3.746E+05

JBroihan commented 8 months ago

Names No

While the following analysis does not focus on performance metrics, it is important to note that the following findings are memory-related. We include them for completeness, as memory management is crucial, particularly for handling large instances.

By default, GAMS passes variable and equation names to CPLEX. Turning the CPLEX option names off results in a slight reduction of the overall memory footprint.

Based on the results we gain a small reduction by about 3% in memory use by using names no.

Relative Difference obtained by `names no`

Instance	Elapsed Time [%]	Total Seconds [%]	Compilation Time [%]	Execution Time [%]	Generation Time [%]	Solver Time [%]	Startup Time [%]	Closedown Time [%]	Highwater RSS [%]	Highwater VSS [%]	# Non Zeros per s [%]
1p5C_OS_SSP2	-4.06	-4.32	-33.94	-5.19	-2.80	-1.22	0.00	nan	-4.91	-4.38	2.88
BAU_IND_V2	-2.52	-2.52	0.04	-4.01	-1.43	-2.49	-6.25	nan	0.00	0.00	1.45
BEAM-ME_Instances/672_11_8	0.61	0.60	2.85	-6.86	-1.97	0.82	nan	nan	1.39	0.00	2.01
BEAM-ME_Instances/672_22_8	-0.12	-0.12	51.60	-8.34	0.23	0.00	6.67	nan	5.33	-0.03	-0.23
E4SMA	-1.64	-1.64	-29.77	-8.99	-3.74	-0.75	0.00	nan	-5.53	-5.01	3.89
GAZNAT_V1_Instance	-3.04	-3.04	-30.86	-4.40	-1.22	-3.02	-6.25	nan	0.00	0.00	1.24
POLIZERO_V1_Instance	-3.01	-3.03	1.62	-5.91	-1.34	-2.78	0.00	nan	-6.31	-5.79	1.35
POLIZERO_V2_Instance	-3.69	-3.72	-3.31	-6.35	-0.70	-3.48	6.67	nan	-7.39	-6.73	0.70
TIMES-DK-Instance	0.46	0.49	-11.01	-7.65	-1.17	1.68	inf	nan	-7.17	-4.74	1.19
UK-TIMES	1.97	2.07	158.95	-8.80	-2.23	3.74	inf	nan	-6.60	-6.08	2.28
mean	-1.50	-1.52	10.62	-6.65	-1.64	-0.75	inf	nan	-3.12	-3.28	1.68

New `names no`

Instance	Model Status	Elapsed Time [s]	Total Seconds [s]	Compilation Time [s]	Execution Time [s]	Generation Time [s]	Solver Time [s]	Startup Time [s]	Highwater RSS [GB]	Highwater VSS [GB]	# Rows	# Columns	# Non Zeros	# Non Zeros per s	RHS [min, max]	Bound [min, max]	Matrix [min, max]
1p5C_OS_SSP2	FEASIBLE SOLUTION	72.06	67.12	1.16	38.27	13.02	27.69	0.02	2.71	3.06	624,436	863,666	4,650,991	357,328.75	9.222E-05, 3.258E+06	7.001E-06, 6.612E+07	6.977E-09, 2.535E+06
BAU_IND_V2	OPTIMAL	7,558.44	7,545.12	2.52	159.77	112.20	7,382.83	0.01	22.81	26.23	6,750,106	6,399,364	40,479,629	360,768.15	1.566E-14, 2.285E+05	3.516E-09, 5.000E+04	1.623E-07, 3.352E+06
BEAM-ME_Instances/672_11_8	OPTIMAL	13,084.65	13,057.45	0.58	338.59	246.41	12,718.28	0.00	63.54	70.11	21,108,711	15,111,159	124,095,043	503,620.22	4.097E-06, 5.940E+05	2.680E-02, 2.676E+05	1.882E-04, 1.663E+04
BEAM-ME_Instances/672_22_8	INTERMEDIATE INFEASIBLE	43,894.04	43,837.84	1.47	598.86	433.31	43,237.50	0.02	118.90	163.82	37,645,140	27,331,345	218,532,379	504,329.15	1.405E-06, 5.940E+05	8.298E-03, 2.676E+05	2.675E-04, 1.663E+04
E4SMA	FEASIBLE SOLUTION	1,356.61	1,338.56	1.33	127.74	54.67	1,209.48	0.02	16.92	18.95	4,229,763	4,437,740	32,382,036	592,296.53	7.402E-16, 4.407E+06	9.557E-06, 1.000E+09	1.258E-09, 1.610E+06
GAZNAT_V1_Instance	FEASIBLE SOLUTION	15,013.56	15,000.98	2.48	131.86	92.08	14,866.62	0.01	19.17	20.62	5,336,684	4,555,729	34,188,921	371,303.91	1.566E-14, 2.291E+05	3.516E-09, 3.000E+06	1.623E-07, 3.352E+06
POLIZERO_V1_Instance	FEASIBLE SOLUTION	2,780.23	2,759.53	4.91	222.06	113.00	2,532.55	0.01	21.22	23.26	5,640,942	6,991,820	48,190,465	426,464.29	7.918E-19, 5.440E+05	3.837E-19, 1.000E+07	7.309E-08, 4.178E+06
POLIZERO_V2_Instance	FEASIBLE SOLUTION	4,922.68	4,876.00	4.12	394.03	191.61	4,477.83	0.02	33.60	37.44	10,540,301	13,847,195	69,064,648	360,445.74	1.918E-18, 8.825E+06	1.823E-19, 2.919E+07	9.620E-19, 2.269E+06
TIMES-DK-Instance	OPTIMAL	149.80	148.48	1.89	14.91	7.84	131.67	0.01	2.46	3.82	867,042	1,031,037	4,740,351	604,328.28	3.577E-08, 7.453E+04	4.606E-05, 4.500E+04	1.909E-09, 4.282E+06
UK-TIMES	OPTIMAL	103.42	101.12	1.38	18.30	10.98	81.42	0.03	2.83	3.09	875,780	932,609	6,883,510	626,628.13	4.659E-05, 6.697E+05	6.000E-04, 5.867E+05	2.793E-11, 3.746E+05

Ground Truth

Instance	Model Status	Elapsed Time [s]	Total Seconds [s]	Compilation Time [s]	Execution Time [s]	Generation Time [s]	Solver Time [s]	Startup Time [s]	Closedown Time [s]	Highwater RSS [GB]	Highwater VSS [GB]	# Rows	# Columns	# Non Zeros	# Non Zeros per s	RHS [min, max]	Bound [min, max]	Matrix [min, max]
1p5C_OS_SSP2	FEASIBLE SOLUTION	75.11	70.16	1.75	40.36	13.39	28.03	0.02	0.00	2.85	3.20	624,436	863,666	4,650,991	347,322.16	9.222E-05, 3.258E+06	7.001E-06, 6.612E+07	6.977E-09, 2.535E+06
BAU_IND_V2	OPTIMAL	7,753.52	7,740.36	2.52	166.44	113.83	7,571.39	0.02	0.00	22.81	26.23	6,750,106	6,399,364	40,479,629	355,621.02	1.566E-14, 2.285E+05	3.516E-09, 5.000E+04	1.623E-07, 3.352E+06
BEAM-ME_Instances/672_11_8	OPTIMAL	13,005.10	12,979.09	0.56	363.55	251.36	12,614.98	0.00	0.00	62.67	70.11	21,108,711	15,111,159	124,095,043	493,694.47	4.097E-06, 5.940E+05	2.680E-02, 2.676E+05	1.882E-04, 1.663E+04
BEAM-ME_Instances/672_22_8	INTERMEDIATE INFEASIBLE	43,946.87	43,891.17	0.97	653.31	432.31	43,236.88	0.01	0.00	112.88	163.87	37,645,140	27,331,345	218,532,379	505,495.74	1.405E-06, 5.940E+05	8.298E-03, 2.676E+05	2.675E-04, 1.663E+04
BEAM-ME_Instances/2016_22_8	ERROR NO SOLUTION	3,330.59	1,969.62	2.09	1,930.45	1,752.70	1,201.00	17.39	19.69	115.10	215.29	112,857,882	81,947,383	658,167,482	375,515.69	2.460E-07, 5.940E+05	8.298E-03, 2.676E+05	1.946E-06, 1.663E+04
E4SMA	FEASIBLE SOLUTION	1,379.17	1,360.83	1.89	140.36	56.80	1,218.56	0.02	0.00	17.91	19.95	4,229,763	4,437,740	32,382,036	570,136.38	7.402E-16, 4.407E+06	9.557E-06, 1.000E+09	1.258E-09, 1.610E+06
GAZNAT_V1_Instance	FEASIBLE SOLUTION	15,483.52	15,470.69	3.59	137.92	93.22	15,329.16	0.02	0.00	19.17	20.62	5,336,684	4,555,729	34,188,921	366,759.15	1.566E-14, 2.291E+05	3.516E-09, 3.000E+06	1.623E-07, 3.352E+06
POLIZERO_V1_Instance	FEASIBLE SOLUTION	2,866.62	2,845.72	4.83	236.00	114.53	2,604.88	0.01	0.00	22.65	24.69	5,640,942	6,991,820	48,190,465	420,763.51	7.918E-19, 5.440E+05	3.837E-19, 1.000E+07	7.309E-08, 4.178E+06
POLIZERO_V2_Instance	FEASIBLE SOLUTION	5,111.24	5,064.34	4.27	420.73	192.95	4,639.33	0.01	0.00	36.28	40.14	10,540,301	13,847,195	69,064,648	357,935.08	1.918E-18, 8.825E+06	1.823E-19, 2.919E+07	9.620E-19, 2.269E+06
TIMES-DK-Instance	OPTIMAL	149.12	147.76	2.12	16.14	7.94	129.50	0.00	0.00	2.65	4.01	867,042	1,031,037	4,740,351	597,247.20	3.577E-08, 7.453E+04	4.606E-05, 4.500E+04	1.909E-09, 4.282E+06
UK-TIMES	OPTIMAL	101.42	99.08	0.53	20.06	11.23	78.48	0.00	0.00	3.03	3.29	875,780	932,609	6,883,510	612,684.47	4.659E-05, 6.697E+05	6.000E-04, 5.867E+05	2.793E-11, 3.746E+05

JBroihan commented 8 months ago

v4.7.9

The following analysis does not document improvements achieved by us; rather, it compares enhancements made by Antti between versions 4.7.8 and 4.7.9.

Improvement v4.7.8 vs v4.7.9

Instance	Elapsed Time [%]	Total Seconds [%]	Compilation Time [%]	Execution Time [%]	Generation Time [%]	Solver Time [%]	Startup Time [%]	Closedown Time [%]	Highwater RSS [%]	Highwater VSS [%]	# Non Zeros per s [%]
1p5C_OS_SSP2	-1.99	-2.07	-33.94	-1.39	-1.87	-1.06	0.00	nan	0.00	0.00	1.90
BAU_IND_V2	-4.66	-4.67	0.00	-1.17	-0.80	-4.75	0.00	nan	0.04	0.00	0.80
BEAM-ME_Instances/672_11_8	-2.35	-2.36	22.42	-5.08	-2.67	-2.28	nan	nan	0.70	0.01	2.75
BEAM-ME_Instances/672_22_8	-0.05	-0.06	6.40	-3.45	-0.08	-0.01	-100.00	nan	5.90	-0.09	0.08
E4SMA	-0.20	-0.18	18.19	-3.07	-3.88	0.13	-6.25	nan	0.00	-0.05	4.04
GAZNAT_V1_Instance	-5.13	-5.14	-13.49	-0.87	-1.01	-5.17	0.00	nan	0.00	0.00	1.02
Offshore	-35.48	-35.90	27.73	-56.46	0.58	-18.49	93.75	nan	0.00	0.00	-0.58
POLIZERO_V1_Instance	-0.39	-0.38	-4.87	-1.19	-1.31	-0.30	6.67	nan	-0.04	0.00	1.33
POLIZERO_V2_Instance	-3.61	-3.63	-2.20	-1.72	-3.24	-3.81	0.00	nan	0.00	0.00	3.35
TIMES-DK-Instance	2.39	2.42	-4.42	0.01	-1.78	2.84	nan	nan	0.00	0.00	1.81
UK-TIMES	-1.31	-1.31	64.78	-2.18	-2.37	-1.53	nan	nan	0.00	-0.30	2.43
mean	-4.80	-4.84	7.33	-6.96	-1.68	-3.13	-0.73	nan	0.60	-0.04	1.72

v4.7.9

Instance	Model Status	Elapsed Time [s]	Total Seconds [s]	Compilation Time [s]	Execution Time [s]	Generation Time [s]	Solver Time [s]	Startup Time [s]	Highwater RSS [GB]	Highwater VSS [GB]	# Rows	# Columns	# Non Zeros	# Non Zeros per s	RHS [min, max]	Bound [min, max]	Matrix [min, max]
1p5C_OS_SSP2	FEASIBLE SOLUTION	73.61	68.70	1.16	39.80	13.14	27.73	0.02	2.85	3.20	624,436	863,666	4,650,991	353,929.76	9.222E-05, 3.258E+06	7.001E-06, 6.612E+07	6.977E-09, 2.535E+06
BAU_IND_V2	OPTIMAL	7,391.95	7,378.95	2.52	164.48	112.92	7,211.94	0.02	22.82	26.23	6,750,106	6,399,364	40,479,629	358,477.42	1.566E-14, 2.285E+05	3.516E-09, 5.000E+04	1.623E-07, 3.352E+06
BEAM-ME_Instances/672_11_8	OPTIMAL	12,698.88	12,672.86	0.69	345.09	244.64	12,327.08	0.00	63.11	70.12	21,108,711	15,111,159	124,095,043	507,255.73	4.097E-06, 5.940E+05	2.680E-02, 2.676E+05	1.882E-04, 1.663E+04
BEAM-ME_Instances/672_22_8	INTERMEDIATE INFEASIBLE	43,923.21	43,865.22	1.03	630.75	431.98	43,233.44	0.00	119.54	163.72	37,645,140	27,331,345	218,532,379	505,880.72	1.405E-06, 5.940E+05	8.298E-03, 2.676E+05	2.675E-04, 1.663E+04
E4SMA	FEASIBLE SOLUTION	1,376.43	1,358.42	2.23	136.05	54.59	1,220.12	0.01	17.91	19.94	4,229,763	4,437,740	32,382,036	593,142.76	7.402E-16, 4.407E+06	9.557E-06, 1.000E+09	1.258E-09, 1.610E+06
GAZNAT_V1_Instance	FEASIBLE SOLUTION	14,688.61	14,676.02	3.11	136.72	92.28	14,536.17	0.02	19.17	20.62	5,336,684	4,555,729	34,188,921	370,487.11	1.566E-14, 2.291E+05	3.516E-09, 3.000E+06	1.623E-07, 3.352E+06
Offshore	FEASIBLE SOLUTION	535.64	525.78	1.08	164.20	111.28	360.47	0.03	7.31	8.16	3,326,614	2,091,377	13,911,856	125,014.43	1.000E+00, 6.000E+03	NA, NA	2.546E-05, 2.735E+06
POLIZERO_V1_Instance	FEASIBLE SOLUTION	2,855.53	2,834.97	4.59	233.19	113.03	2,597.17	0.02	22.64	24.69	5,640,942	6,991,820	48,190,465	426,347.33	7.918E-19, 5.440E+05	3.837E-19, 1.000E+07	7.309E-08, 4.178E+06
POLIZERO_V2_Instance	FEASIBLE SOLUTION	4,926.91	4,880.33	4.17	413.52	186.70	4,462.62	0.01	36.28	40.14	10,540,301	13,847,195	69,064,648	369,917.18	1.918E-18, 8.825E+06	1.823E-19, 2.919E+07	9.620E-19, 2.269E+06
TIMES-DK-Instance	OPTIMAL	152.68	151.34	2.03	16.14	7.80	133.17	0.00	2.65	4.01	867,042	1,031,037	4,740,351	608,049.13	3.577E-08, 7.453E+04	4.606E-05, 4.500E+04	1.909E-09, 4.282E+06
UK-TIMES	OPTIMAL	100.09	97.78	0.88	19.62	10.97	77.28	0.00	3.03	3.28	875,780	932,609	6,883,510	627,542.16	4.659E-05, 6.697E+05	6.000E-04, 5.867E+05	2.793E-11, 3.746E+05

v4.7.8

Instance	Model Status	Elapsed Time [s]	Total Seconds [s]	Compilation Time [s]	Execution Time [s]	Generation Time [s]	Solver Time [s]	Startup Time [s]	Closedown Time [s]	Highwater RSS [GB]	Highwater VSS [GB]	# Rows	# Columns	# Non Zeros	# Non Zeros per s	RHS [min, max]	Bound [min, max]	Matrix [min, max]
1p5C_OS_SSP2	FEASIBLE SOLUTION	75.11	70.16	1.75	40.36	13.39	28.03	0.02	0.00	2.85	3.20	624,436	863,666	4,650,991	347,322.16	9.222E-05, 3.258E+06	7.001E-06, 6.612E+07	6.977E-09, 2.535E+06
BAU_IND_V2	OPTIMAL	7,753.52	7,740.36	2.52	166.44	113.83	7,571.39	0.02	0.00	22.81	26.23	6,750,106	6,399,364	40,479,629	355,621.02	1.566E-14, 2.285E+05	3.516E-09, 5.000E+04	1.623E-07, 3.352E+06
BEAM-ME_Instances/672_11_8	OPTIMAL	13,005.10	12,979.09	0.56	363.55	251.36	12,614.98	0.00	0.00	62.67	70.11	21,108,711	15,111,159	124,095,043	493,694.47	4.097E-06, 5.940E+05	2.680E-02, 2.676E+05	1.882E-04, 1.663E+04
BEAM-ME_Instances/672_22_8	INTERMEDIATE INFEASIBLE	43,946.87	43,891.17	0.97	653.31	432.31	43,236.88	0.01	0.00	112.88	163.87	37,645,140	27,331,345	218,532,379	505,495.74	1.405E-06, 5.940E+05	8.298E-03, 2.676E+05	2.675E-04, 1.663E+04
BEAM-ME_Instances/2016_22_8	ERROR NO SOLUTION	3,330.59	1,969.62	2.09	1,930.45	1,752.70	1,201.00	17.39	19.69	115.10	215.29	112,857,882	81,947,383	658,167,482	375,515.69	2.460E-07, 5.940E+05	8.298E-03, 2.676E+05	1.946E-06, 1.663E+04
E4SMA	FEASIBLE SOLUTION	1,379.17	1,360.83	1.89	140.36	56.80	1,218.56	0.02	0.00	17.91	19.95	4,229,763	4,437,740	32,382,036	570,136.38	7.402E-16, 4.407E+06	9.557E-06, 1.000E+09	1.258E-09, 1.610E+06
GAZNAT_V1_Instance	FEASIBLE SOLUTION	15,483.52	15,470.69	3.59	137.92	93.22	15,329.16	0.02	0.00	19.17	20.62	5,336,684	4,555,729	34,188,921	366,759.15	1.566E-14, 2.291E+05	3.516E-09, 3.000E+06	1.623E-07, 3.352E+06
Offshore	FEASIBLE SOLUTION	830.24	820.20	0.84	377.11	110.64	442.24	0.02	0.00	7.31	8.16	3,326,614	2,091,377	13,911,856	125,739.84	1.000E+00, 6.000E+03	NA, NA	2.546E-05, 2.735E+06
POLIZERO_V1_Instance	FEASIBLE SOLUTION	2,866.62	2,845.72	4.83	236.00	114.53	2,604.88	0.01	0.00	22.65	24.69	5,640,942	6,991,820	48,190,465	420,763.51	7.918E-19, 5.440E+05	3.837E-19, 1.000E+07	7.309E-08, 4.178E+06
POLIZERO_V2_Instance	FEASIBLE SOLUTION	5,111.24	5,064.34	4.27	420.73	192.95	4,639.33	0.01	0.00	36.28	40.14	10,540,301	13,847,195	69,064,648	357,935.08	1.918E-18, 8.825E+06	1.823E-19, 2.919E+07	9.620E-19, 2.269E+06
TIMES-DK-Instance	OPTIMAL	149.12	147.76	2.12	16.14	7.94	129.50	0.00	0.00	2.65	4.01	867,042	1,031,037	4,740,351	597,247.20	3.577E-08, 7.453E+04	4.606E-05, 4.500E+04	1.909E-09, 4.282E+06
UK-TIMES	OPTIMAL	101.42	99.08	0.53	20.06	11.23	78.48	0.00	0.00	3.03	3.29	875,780	932,609	6,883,510	612,684.47	4.659E-05, 6.697E+05	6.000E-04, 5.867E+05	2.793E-11, 3.746E+05

Antti-L commented 8 months ago

Thanks for adding the Offshore case. What I find interesting about it is the relatively poor performance of generating some of the model equations, e.g. EQE_ACTEFF and EQE_INSHR, which would thus deserve some additional investigation, if possible.

It is also interesting that for this case, like many others, you seem to get only a feasible solution (apparently run without crossover?), while in my test I am getting an optimal solution both with and without crossover in less time, but I guess it is probably related to different Cplex options...

JBroihan commented 8 months ago

What I find interesting about it is the relatively poor performance of generating some of the model equations, e.g. EQE_ACTEFF and EQE_INSHR, which would thus deserve some additional investigation, if possible.

It is interesting, indeed. While we find a model generation time of 111s to create 13,911,856 non zeros for the offshore instance, we are able to generate the E4SMA model instance in only 54s with 32,382,036 non zeros.

Offshore

Model Generation 111.28s

MODEL STATISTICS

BLOCKS OF EQUATIONS         104     SINGLE EQUATIONS    3,326,614
BLOCKS OF VARIABLES          25     SINGLE VARIABLES    2,091,377
NON ZERO ELEMENTS    13,911,856

--- Profile Summary (225 records processed)
    26.313   2.359GB    156650 Equation   EQE_ACTEFF (285120)
    25.828   2.294GB    145263 Equation   EQE_INSHR (285120)

E4SMA

Model Generation 54.59s

MODEL STATISTICS

BLOCKS OF EQUATIONS         104     SINGLE EQUATIONS    4,229,763
BLOCKS OF VARIABLES          25     SINGLE VARIABLES    4,437,740  1,526 projected
NON ZERO ELEMENTS    32,382,036

--- Profile Summary (468 records processed)
     3.188   3.983GB    774138 Equation   EQE_ACTEFF (508014)
     2.062   3.822GB    762751 Equation   EQE_INSHR (401577)

Not only does GAMS generate more rows in less time for E4SMA compared to Offshore, isolating those equations also shows that the number of Non-zeroes in those equations is higher in the E4SMA instance as well.

NZ counts:

equation	Offshore	E4SMA
EQE_ACTEFF	855,361	1,998,294
EQE_INSHR	570,241	2,110,919

So why does Offshore perform so much slower?

By examining the set cardinalities of the instances, we can see that the offshore instance looks significantly different to the other instances. Especially |S| = 9013 is significantly larger while |C| = 8 and |P| = 15 are significantly smaller. In itself, this is not a very satisfactory answer why Offshore is slower, but it at least explains why the behavior is significantly different.

                            IO  COM_VAR  BD  TSL  L  TSLVL  UC_GRPTYPE  T  R  V  ALLSOW  ALLYEAR    C  COM  COM2  ALL_TS   SL    S   TS   CG  COM_GRP    P  UC_N  ITEM
Instance                                                                                                                                                              
1p5C_OS_SSP2                 2        2   3    4  4      4          10 11 16 37      96      438  668  668   668      10   10   10   10 1419     1419 3715   275  7164
BAU_IND_V2                   2        2   3    4  4      4           9  6  1 72      96      277  760  760   760     305  305  305  305 1235     1235 1528  9152 13771
BEAM-ME_Instances/672_11_8   2        2   3    4  4      4           9  8 11 50      96      257   24   24    24     677  677  677  677   31       31  128    14  2396
BEAM-ME_Instances/672_22_8   2        2   3    4  4      4           9  8 22 48      96      257   24   24    24     677  677  677  677   31       31  150    14  2418
E4SMA                        2        2   3    4  4      4           9 13 23 14      96      222  909  909   909      25   25   25   25 1564     1564 2007    96  5319
GAZNAT_V1_Instance           2        2   3    4  4      4           9  6  1 73      96      277  534  534   534     305  305  305  305  711      711  940  9208 12726
Offshore                     2        2   3    4  4      4           9 11  3 12      96      222    8    8     8    9013 9013 9013 9013   17       17   15     2 10526
POLIZERO_V1_Instance         2        2   3    4  4      4           9 10 30 47      96      262 1142 1142  1142      17   17   17   17 2578     2578 2752   212  7167
POLIZERO_V2_Instance         2        2   3    4  4      4           9 10 30 50      96      262  748  748   748      17   17   17   17 2777     2777 3334  1368  9110
TIMES-DK-Instance            2        2   3    4  4      4           9 10  2 11      96      232  541  541   541      45   45   45   45 1284     1284 1645   221  4709
UK-TIMES                     2        2   3    4  4      4           9 13  1 14      96      222  824  824   824      21   21   21   21 2056     2056 2169   464  6205
mean                         2        2   3    4  4      4           9 10 13 39      96      266  562  562   562    1010 1010 1010 1010 1246     1246 1671  1911  7410
median                       2        2   3    4  4      4           9 10 12 43      96      260  615  615   615     175  175  175  175 1265     1265 1658   248  7166

So far, we did not manage to make the generation faster.

I guess it is probably related to different Cplex options

Yes, the different outcome in terms of optimal vs. feasible solution is most likely due to different cplex options (could also be different hardware and/or different versions). All our experiments use the option file that we received with the instance data. However, for some of the instances (1p5C_OS_SSP2, BEAM-ME, E4SMA, Offshore, UK-TIMES) there is no option file. In this case our experiments are subject to ["lpmethod 4", "solutiontype 2", "threads 16", "reslim 43200"].

JBroihan commented 8 months ago

TIMES Code Review

Pull request #4 contains several changes of the TIMES source code to improve the GAMS execution time focusing particularly on the model generation time. The suggested changes are benchmarked on various TIMES instances and results are presented in the following tables. We recommend a comprehensive review of the changes prior to their potential inclusion in the official TIMES repository.

Relative Difference obtained by #4

Instance	Elapsed Time [%]	Total Seconds [%]	Compilation Time [%]	Execution Time [%]	Generation Time [%]	Solver Time [%]	Startup Time [%]	Closedown Time [%]	Highwater RSS [%]	Highwater VSS [%]	# Non Zeros per s [%]
1p5C_OS_SSP2	1.26	1.98	-2.47	-2.38	-3.49	8.79	-53.12	nan	0.00	0.00	3.61
BAU_IND_V2	-10.55	-10.58	-19.00	-2.84	-2.97	-10.75	-65.96	nan	-0.04	0.00	3.06
BEAM-ME_Instances/672_11_8	8.71	8.72	-10.02	-13.07	-18.09	9.33	6.67	nan	0.50	0.50	22.09
BEAM-ME_Instances/672_22_8	-0.17	-0.18	4.18	-12.17	-18.07	-0.00	0.00	nan	0.74	0.00	22.05
E4SMA	14.88	15.08	-1.90	-0.52	-1.29	16.91	0.00	nan	0.06	0.05	1.31
GAZNAT_V1_Instance	-25.00	-25.02	-3.04	-1.30	-1.15	-25.25	-100.00	nan	-0.16	-0.10	1.17
Offshore	-4.98	-5.12	0.00	-20.46	0.37	0.81	6.67	nan	-0.27	0.00	-0.37
POLIZERO_V1_Instance	-0.46	-0.47	-0.69	-1.27	-0.85	-0.40	0.00	nan	0.00	0.00	0.86
POLIZERO_V2_Instance	-20.72	-20.94	-2.31	-11.07	-26.54	-21.86	-100.00	nan	-0.50	0.07	36.14
TIMES-DK-Instance	-0.22	-0.22	-6.72	-0.90	-1.82	-0.04	0.00	nan	0.38	0.00	1.85
UK-TIMES	-6.78	-6.85	4.13	-1.22	-0.89	-8.32	nan	nan	-0.33	-0.60	0.90
sum	-4.88	-4.89	-4.79	-9.22	-12.93	-4.78	-35.32	nan	0.31	0.09	8.51

New

Instance	Model Status	Elapsed Time [s]	Total Seconds [s]	Compilation Time [s]	Execution Time [s]	Generation Time [s]	Solver Time [s]	Startup Time [s]	Highwater RSS [GB]	Highwater VSS [GB]	# Rows	# Columns	# Non Zeros	# Non Zeros per s	RHS [min, max]	Bound [min, max]	Matrix [min, max]
1p5C_OS_SSP2	FEASIBLE SOLUTION	76.47	71.56	2.48	39.08	12.56	29.98	0.01	2.84	3.20	624,436	863,666	4,650,991	370,242.88	9.222E-05, 3.258E+06	7.001E-06, 6.612E+07	6.977E-09, 2.535E+06
BAU_IND_V2	OPTIMAL	6,562.41	6,549.28	3.27	157.92	107.62	6,388.08	0.02	22.80	26.23	6,750,106	6,399,364	40,479,629	376,117.34	1.566E-14, 2.285E+05	3.516E-09, 5.000E+04	1.623E-07, 3.352E+06
BEAM-ME_Instances/672_11_8	OPTIMAL	13,631.71	13,603.17	0.84	296.02	196.53	13,306.30	0.02	63.88	70.46	21,108,711	15,111,159	124,095,043	631,427.32	4.097E-06, 5.940E+05	2.680E-02, 2.676E+05	1.882E-04, 1.663E+04
BEAM-ME_Instances/672_22_8	INTERMEDIATE INFEASIBLE	43,842.90	43,785.27	1.17	540.36	346.33	43,243.72	0.02	121.24	163.72	37,645,140	27,331,345	218,532,379	630,998.30	1.405E-06, 5.940E+05	8.298E-03, 2.676E+05	2.675E-04, 1.663E+04
E4SMA	FEASIBLE SOLUTION	1,544.59	1,526.44	2.42	135.53	53.83	1,388.45	0.03	17.91	19.95	4,229,763	4,437,740	32,382,036	601,583.49	7.402E-16, 4.407E+06	9.557E-06, 1.000E+09	1.258E-09, 1.610E+06
GAZNAT_V1_Instance	FEASIBLE SOLUTION	11,077.91	11,065.14	2.97	132.75	89.73	10,929.42	0.00	19.13	20.60	5,336,684	4,555,729	34,188,921	381,002.98	1.566E-14, 2.291E+05	3.516E-09, 3.000E+06	1.623E-07, 3.352E+06
Offshore	FEASIBLE SOLUTION	560.28	550.20	0.42	128.51	109.84	421.25	0.02	7.31	8.16	3,326,614	2,091,377	13,911,856	126,651.03	1.000E+00, 6.000E+03	NA, NA	2.546E-05, 2.735E+06
POLIZERO_V1_Instance	FEASIBLE SOLUTION	2,741.83	2,721.08	4.45	230.20	111.00	2,486.41	0.02	22.63	24.68	5,640,942	6,991,820	48,190,465	434,148.33	7.918E-19, 5.440E+05	3.837E-19, 1.000E+07	7.309E-08, 4.178E+06
POLIZERO_V2_Instance	FEASIBLE SOLUTION	3,896.03	3,848.50	3.98	363.22	137.33	3,481.30	0.00	36.11	40.17	10,540,301	13,847,195	69,064,648	502,917.45	1.918E-18, 8.825E+06	1.823E-19, 2.919E+07	9.620E-19, 2.269E+06
TIMES-DK-Instance	OPTIMAL	145.93	144.61	1.74	15.67	7.61	127.19	0.01	2.65	4.01	867,042	1,031,037	4,740,351	622,992.64	3.577E-08, 7.453E+04	4.606E-05, 4.500E+04	1.909E-09, 4.282E+06
UK-TIMES	OPTIMAL	93.85	91.64	0.78	18.91	10.50	71.95	0.00	3.02	3.30	875,780	932,609	6,883,510	655,572.38	4.659E-05, 6.697E+05	6.000E-04, 5.867E+05	2.793E-11, 3.746E+05
sum		84,173.91	83,956.89	24.53	2,058.17	1,182.89	81,874.05	0.14	319.52	384.48	96,945,519	83,593,041	597,119,829	5,333,654.14

Ground Truth v4.7.9

Instance	Model Status	Elapsed Time [s]	Total Seconds [s]	Compilation Time [s]	Execution Time [s]	Generation Time [s]	Solver Time [s]	Startup Time [s]	Highwater RSS [GB]	Highwater VSS [GB]	# Rows	# Columns	# Non Zeros	# Non Zeros per s	RHS [min, max]	Bound [min, max]	Matrix [min, max]
1p5C_OS_SSP2	FEASIBLE SOLUTION	75.51	70.17	2.55	40.03	13.02	27.56	0.03	2.84	3.20	624,436	863,666	4,650,991	357,328.75	9.222E-05, 3.258E+06	7.001E-06, 6.612E+07	6.977E-09, 2.535E+06
BAU_IND_V2	OPTIMAL	7,336.80	7,323.91	4.03	162.53	110.92	7,157.30	0.05	22.81	26.23	6,750,106	6,399,364	40,479,629	364,941.08	1.566E-14, 2.285E+05	3.516E-09, 5.000E+04	1.623E-07, 3.352E+06
BEAM-ME_Instances/672_11_8	OPTIMAL	12,539.04	12,512.01	0.94	340.53	239.94	12,170.53	0.01	63.56	70.11	21,108,711	15,111,159	124,095,043	517,196.29	4.097E-06, 5.940E+05	2.680E-02, 2.676E+05	1.882E-04, 1.663E+04
BEAM-ME_Instances/672_22_8	INTERMEDIATE INFEASIBLE	43,917.88	43,862.08	1.12	615.24	422.70	43,245.70	0.02	120.35	163.72	37,645,140	27,331,345	218,532,379	516,986.78	1.405E-06, 5.940E+05	8.298E-03, 2.676E+05	2.675E-04, 1.663E+04
E4SMA	FEASIBLE SOLUTION	1,344.53	1,326.39	2.47	136.23	54.53	1,187.66	0.03	17.90	19.94	4,229,763	4,437,740	32,382,036	593,817.13	7.402E-16, 4.407E+06	9.557E-06, 1.000E+09	1.258E-09, 1.610E+06
GAZNAT_V1_Instance	FEASIBLE SOLUTION	14,770.60	14,758.12	3.06	134.50	90.78	14,620.55	0.02	19.16	20.62	5,336,684	4,555,729	34,188,921	376,608.77	1.566E-14, 2.291E+05	3.516E-09, 3.000E+06	1.623E-07, 3.352E+06
Offshore	FEASIBLE SOLUTION	589.66	579.88	0.42	161.56	109.44	417.88	0.01	7.33	8.16	3,326,614	2,091,377	13,911,856	127,122.05	1.000E+00, 6.000E+03	NA, NA	2.546E-05, 2.735E+06
POLIZERO_V1_Instance	FEASIBLE SOLUTION	2,754.59	2,734.02	4.48	233.17	111.95	2,496.34	0.02	22.63	24.68	5,640,942	6,991,820	48,190,465	430,452.65	7.918E-19, 5.440E+05	3.837E-19, 1.000E+07	7.309E-08, 4.178E+06
POLIZERO_V2_Instance	FEASIBLE SOLUTION	4,914.09	4,867.83	4.08	408.45	186.95	4,455.28	0.01	36.29	40.14	10,540,301	13,847,195	69,064,648	369,422.52	1.918E-18, 8.825E+06	1.823E-19, 2.919E+07	9.620E-19, 2.269E+06
TIMES-DK-Instance	OPTIMAL	146.25	144.92	1.86	15.81	7.75	127.23	0.01	2.64	4.01	867,042	1,031,037	4,740,351	611,658.19	3.577E-08, 7.453E+04	4.606E-05, 4.500E+04	1.909E-09, 4.282E+06
UK-TIMES	OPTIMAL	100.67	98.38	0.75	19.14	10.59	78.48	0.00	3.03	3.32	875,780	932,609	6,883,510	649,755.52	4.659E-05, 6.697E+05	6.000E-04, 5.867E+05	2.793E-11, 3.746E+05
sum		88,489.64	88,277.70	25.77	2,267.21	1,358.58	85,984.52	0.22	318.54	384.13	96,945,519	83,593,041	597,119,829	4,915,289.73

Model Generation Time: Achieved a reduction of 12.93% across all instances.
Improvement of the PAR_IPRIC assignment has led to reduced execution time.
We've managed to cut down the GAMS time (execution time) by 9.22% seconds across all instances.

The result table also shows a mild reduction of the average solver time. The different solver behavior can be attributed to row permutations stemming from index swaps in equation domains. Although such permutations can alter solver performance, predicting their impact — whether positive or negative — remains uncertain. We view these effects as random, anticipating no impact solver performance from row and column permutations on average.

Performance approvements were achieved mainly by switching indices and introducing new symbols for isolated precomputations.

There are some general rules of thumb which are, however, not always applicable. Best practice is to try to apply the following rules:

Ordering indices from small to large (e.g. (R, V, P) with |R| = 11, |V| = 50, |P| = 128)
Following a consistent index order for variables and parameters
Following the index order from equation domains in summation statements (e.g. eq(R, V)... sum(P$(VP(V, P)), ...) instead of eq(R, V)... sum(p$(PV(P, V)), ...)

Antti-L commented 8 months ago

Many thanks for arriving at these suggested changes. I am sure we will adopt all of them, with small adaptations. Would it be possible to have some of the listing files from your test runs, to see the impact of some of the changes and maybe give some priorities? I would be especially interested about the POLIZERO_V2_Instance, because I think it may be a good representative case for many big TIMES models. Therefore, if you have the listing file from this case, preferably before and after the changes, I would be grateful if you can provide them to me (e.g. by email).

JBroihan commented 7 months ago

Pull request #6 contains another improvement we came across which improves:

Description	Total Time [s]
Equation EQ_ACTRAMP	70.69
Equation EQ_ACTRAMP new	60.41
Equation EQE_ACTUPS	12.44
Equation EQE_ACTUPS new	12.14
Equation EQ_ACTPL	11.20
Equation EQ_ACTPL new	9.98
Equation EQL_STGCCL	0.11
Equation EQL_STGCCL new	0.11
Assignment selectK	1.70

Note that the following computation times skip the solve statement.

Improvement

Instance	Elapsed Time [%]	Total Seconds [%]	Compilation Time [%]	Execution Time [%]	Generation Time [%]	Solver Time [%]	Startup Time [%]	Closedown Time [%]	Highwater RSS [%]	Highwater VSS [%]	# Non Zeros per s [%]
1p5C_OS_SSP2	2.39	2.76	23.21	1.92	0.11	nan	0.00	nan	-0.64	-0.64	-0.11
BAU_IND_V2	-2.56	-2.76	-3.80	-2.74	-3.42	nan	0.00	nan	0.00	0.00	3.54
BEAM-ME_Instances/672_11_8	-2.83	-3.09	-9.99	-3.07	-3.38	nan	-6.25	nan	0.00	0.00	3.49
BEAM-ME_Instances/672_22_8	-2.54	-2.81	-10.26	-2.80	-3.01	nan	inf	nan	0.00	0.00	3.11
E4SMA	-1.09	-1.21	-9.50	-1.06	-1.67	nan	-100.00	nan	0.17	0.00	1.70
GAZNAT_V1_Instance	-2.82	-3.05	-6.73	-2.96	-3.02	nan	0.00	nan	0.27	0.27	3.12
Offshore	-1.02	-1.06	0.00	-1.06	-0.96	nan	0.00	nan	0.00	0.00	0.97
POLIZERO_V1_Instance	-0.94	-1.04	32.56	-1.58	-1.85	nan	inf	nan	-0.26	-0.39	1.88
POLIZERO_V2_Instance	-0.47	-0.52	29.24	-0.77	-0.92	nan	6.67	nan	0.15	0.15	0.93
TIMES-DK-Instance	1.06	1.03	4.46	0.58	0.21	nan	0.00	nan	0.00	0.00	-0.21
UK-TIMES	0.75	0.83	67.77	-1.15	-0.87	nan	0.00	nan	0.00	0.00	0.88
mean	-0.92	-0.99	10.63	-1.34	-1.71	nan	inf	nan	-0.03	-0.06	1.75

New

Instance	Model Status	Elapsed Time [s]	Total Seconds [s]	Compilation Time [s]	Execution Time [s]	Generation Time [s]	Solver Time [s]	Startup Time [s]	Highwater RSS [GB]	Highwater VSS [GB]	# Rows	# Columns	# Non Zeros	# Non Zeros per s	RHS [min, max]	Bound [min, max]	Matrix [min, max]
1p5C_OS_SSP2	NO SOLUTION RETURNED	45.18	40.16	1.91	38.23	13.27	nan	0.02	1.55	1.55	624,436	863,666	4,650,991	350,621.26	9.222E-05, 3.258E+06	7.001E-06, 6.612E+07	6.977E-09, 2.535E+06
BAU_IND_V2	NO SOLUTION RETURNED	157.14	144.74	3.14	141.58	111.17	nan	0.02	4.56	4.56	6,750,106	6,399,364	40,479,629	364,117.12	1.566E-14, 2.285E+05	3.516E-09, 5.000E+04	1.623E-07, 3.352E+06
BEAM-ME_Instances/672_11_8	NO SOLUTION RETURNED	302.33	276.26	0.70	275.55	236.95	nan	0.01	11.58	11.63	21,108,711	15,111,159	124,095,043	523,711.63	4.097E-06, 5.940E+05	2.680E-02, 2.676E+05	1.882E-04, 1.663E+04
BEAM-ME_Instances/672_22_8	NO SOLUTION RETURNED	512.73	460.41	0.95	459.44	418.14	nan	0.02	20.56	20.66	37,645,140	27,331,345	218,532,379	522,628.44	1.405E-06, 5.940E+05	8.298E-03, 2.676E+05	2.675E-04, 1.663E+04
E4SMA	NO SOLUTION RETURNED	137.33	119.66	1.78	117.88	54.42	nan	0.00	5.94	5.95	4,229,763	4,437,740	32,382,036	595,028.32	7.402E-16, 4.407E+06	9.557E-06, 1.000E+09	1.258E-09, 1.610E+06
GAZNAT_V1_Instance	NO SOLUTION RETURNED	130.60	118.55	2.83	115.70	90.19	nan	0.01	3.70	3.70	5,336,684	4,555,729	34,188,921	379,085.03	1.566E-14, 2.291E+05	3.516E-09, 3.000E+06	1.623E-07, 3.352E+06
Offshore	NO SOLUTION RETURNED	164.14	154.44	0.48	153.94	109.59	nan	0.02	2.42	2.42	3,326,614	2,091,377	13,911,856	126,939.94	1.000E+00, 6.000E+03	NA, NA	2.546E-05, 2.735E+06
POLIZERO_V1_Instance	NO SOLUTION RETURNED	227.65	207.38	4.33	203.03	111.12	nan	0.01	7.57	7.59	5,640,942	6,991,820	48,190,465	433,659.98	7.918E-19, 5.440E+05	3.837E-19, 1.000E+07	7.309E-08, 4.178E+06
POLIZERO_V2_Instance	NO SOLUTION RETURNED	408.06	362.03	3.80	358.22	182.77	nan	0.02	13.69	13.75	10,540,301	13,847,195	69,064,648	377,885.65	1.918E-18, 8.825E+06	1.823E-19, 2.919E+07	9.620E-19, 2.269E+06
TIMES-DK-Instance	NO SOLUTION RETURNED	16.61	15.33	1.83	13.48	7.78	nan	0.01	0.68	0.67	867,042	1,031,037	4,740,351	609,143.02	3.577E-08, 7.453E+04	4.606E-05, 4.500E+04	1.909E-09, 4.282E+06
UK-TIMES	NO SOLUTION RETURNED	19.18	16.95	0.81	16.12	10.66	nan	0.02	0.87	0.87	875,780	932,609	6,883,510	645,975.04	4.659E-05, 6.697E+05	6.000E-04, 5.867E+05	2.793E-11, 3.746E+05

Ground Truth

Instance	Model Status	Elapsed Time [s]	Total Seconds [s]	Compilation Time [s]	Execution Time [s]	Generation Time [s]	Solver Time [s]	Startup Time [s]	Highwater RSS [GB]	Highwater VSS [GB]	# Rows	# Columns	# Non Zeros	# Non Zeros per s	RHS [min, max]	Bound [min, max]	Matrix [min, max]
1p5C_OS_SSP2	ERROR NO SOLUTION	44.13	39.08	1.55	37.52	13.25	nan	0.02	1.56	1.56	624,436	863,666	4,650,991	351,018.19	9.222E-05, 3.258E+06	7.001E-06, 6.612E+07	6.977E-09, 2.535E+06
BAU_IND_V2	ERROR NO SOLUTION	161.27	148.84	3.27	145.56	115.11	nan	0.02	4.56	4.56	6,750,106	6,399,364	40,479,629	351,660.40	1.566E-14, 2.285E+05	3.516E-09, 5.000E+04	1.623E-07, 3.352E+06
BEAM-ME_Instances/672_11_8	ERROR NO SOLUTION	311.12	285.08	0.78	284.28	245.23	nan	0.02	11.58	11.63	21,108,711	15,111,159	124,095,043	506,027.07	4.097E-06, 5.940E+05	2.680E-02, 2.676E+05	1.882E-04, 1.663E+04
BEAM-ME_Instances/672_22_8	ERROR NO SOLUTION	526.07	473.73	1.06	472.67	431.12	nan	0.00	20.56	20.66	37,645,140	27,331,345	218,532,379	506,888.67	1.405E-06, 5.940E+05	8.298E-03, 2.676E+05	2.675E-04, 1.663E+04
E4SMA	ERROR NO SOLUTION	138.84	121.12	1.97	119.14	55.34	nan	0.02	5.93	5.95	4,229,763	4,437,740	32,382,036	585,104.73	7.402E-16, 4.407E+06	9.557E-06, 1.000E+09	1.258E-09, 1.610E+06
GAZNAT_V1_Instance	ERROR NO SOLUTION	134.40	122.28	3.03	119.23	93.00	nan	0.01	3.69	3.69	5,336,684	4,555,729	34,188,921	367,622.81	1.566E-14, 2.291E+05	3.516E-09, 3.000E+06	1.623E-07, 3.352E+06
Offshore	ERROR NO SOLUTION	165.84	156.09	0.48	155.59	110.66	nan	0.02	2.42	2.42	3,326,614	2,091,377	13,911,856	125,721.66	1.000E+00, 6.000E+03	NA, NA	2.546E-05, 2.735E+06
POLIZERO_V1_Instance	ERROR NO SOLUTION	229.80	209.56	3.27	206.30	113.22	nan	0.00	7.59	7.62	5,640,942	6,991,820	48,190,465	425,643.14	7.918E-19, 5.440E+05	3.837E-19, 1.000E+07	7.309E-08, 4.178E+06
POLIZERO_V2_Instance	ERROR NO SOLUTION	409.99	363.94	2.94	360.98	184.47	nan	0.01	13.67	13.73	10,540,301	13,847,195	69,064,648	374,397.04	1.918E-18, 8.825E+06	1.823E-19, 2.919E+07	9.620E-19, 2.269E+06
TIMES-DK-Instance	ERROR NO SOLUTION	16.44	15.17	1.75	13.41	7.77	nan	0.01	0.68	0.67	867,042	1,031,037	4,740,351	610,398.02	3.577E-08, 7.453E+04	4.606E-05, 4.500E+04	1.909E-09, 4.282E+06
UK-TIMES	ERROR NO SOLUTION	19.04	16.81	0.48	16.31	10.75	nan	0.02	0.87	0.87	875,780	932,609	6,883,510	640,326.51	4.659E-05, 6.697E+05	6.000E-04, 5.867E+05	2.793E-11, 3.746E+05

JBroihan commented 4 months ago

Final Results

Elapsed time: Total time to run the complete code
Compilation time: The time to read the complete GAMS model and translate it into specific byte code
Execution time: Time required to execute the different GAMS statements: loops, assignments, model generation, etc.
Generation time: The time required to generate the problem to be solved. Note that Generation time is included in the Execution time
Solver time: The time spent by the solver to solve the GAMS model
Startup & Closedown time: Time to start and close GAMS (occurs for certain SolveLink settings)
Highwater RSS & VSS: Maximum memory requirements observed including the solution phase

Improvement from v4.7.4 to v4.8.0

Instance	Elapsed Time [%]	Total Seconds [%]	Compilation Time [%]	Execution Time [%]	Generation Time [%]	Solver Time [%]	Startup Time [%]	Closedown Time [%]	Highwater RSS [%]	Highwater VSS [%]	# Non Zeros per s [%]
1p5C_OS_SSP2	3.37	3.61	76.69	-1.05	-2.27	7.13	106.67	nan	0.00	0.00	2.32
BAU_IND_V2	-8.45	-8.46	22.48	-16.36	-22.98	-8.29	6.67	nan	-0.09	0.08	29.84
BEAM-ME_Instances/672_11_8	-4.98	-5.00	-39.56	-33.14	-41.47	-3.97	0.00	nan	-0.05	0.04	70.85
BEAM-ME_Instances/672_22_8	-0.66	-0.66	-7.59	-35.22	-45.50	0.03	-6.25	nan	1.06	1.68	83.49
E4SMA	4.24	4.29	-22.18	-1.33	-0.66	5.00	106.67	nan	-0.06	-0.05	0.66
GAZNAT_V1_Instance	-18.40	-18.42	1.07	-20.69	-28.50	-18.40	0.00	nan	1.20	3.20	39.87
Offshore	-43.03	-43.55	57.56	-66.05	0.84	-23.58	3.23	nan	0.41	0.12	-0.83
POLIZERO_V1_Instance	-7.89	-7.91	-0.02	-7.88	-3.80	-7.93	100.00	nan	-0.04	-0.04	3.95
POLIZERO_V2_Instance	-27.29	-27.58	0.37	-9.96	-24.33	-29.26	0.00	nan	-0.55	0.25	32.15
TIMES-DK-Instance	3.77	3.80	-19.75	-0.29	0.62	4.72	-48.39	nan	0.38	-19.95	-0.61
UK-TIMES	-2.04	-2.13	-1.80	0.16	-0.44	-2.71	-6.25	nan	0.66	1.22	0.44
mean	-9.21	-9.27	6.12	-17.44	-15.32	-7.02	23.85	nan	0.27	-1.22	23.83

v4.8.0

Instance	Model Status	Elapsed Time [s]	Total Seconds [s]	Compilation Time [s]	Execution Time [s]	Generation Time [s]	Solver Time [s]	Startup Time [s]	Closedown Time [s]	Highwater RSS [GB]	Highwater VSS [GB]	# Rows	# Columns	# Non Zeros	# Non Zeros per s	RHS [min, max]	Bound [min, max]	Matrix [min, max]
1p5C_OS_SSP2	FEASIBLE SOLUTION	75.08	70.14	1.14	40.03	13.11	28.95	0.01	0.00	2.84	3.20	624,436	863,666	4,650,991	354,793.73	9.222E-05, 3.258E+06	7.001E-06, 6.612E+07	6.977E-09, 2.535E+06
BAU_IND_V2	OPTIMAL	7,200.06	7,187.06	2.50	164.52	112.33	7,020.03	0.01	0.00	22.81	26.23	6,750,106	6,399,364	40,479,629	360,369.89	1.566E-14, 2.285E+05	3.516E-09, 5.000E+04	1.623E-07, 3.352E+06
BEAM-ME_Instances/672_11_8	OPTIMAL	12,663.56	12,636.41	1.19	447.48	340.88	12,187.72	0.02	0.00	63.56	70.12	21,108,711	15,111,159	124,095,043	364,048.53	4.097E-06, 5.940E+05	2.680E-02, 2.676E+05	1.882E-04, 1.663E+04
BEAM-ME_Instances/672_22_8	INTERMEDIATE INFEASIBLE	44,138.21	44,081.23	1.01	857.08	641.92	43,223.12	0.02	0.00	116.82	163.84	37,645,140	27,331,345	218,532,379	340,434.47	1.405E-06, 5.940E+05	8.298E-03, 2.676E+05	2.675E-04, 1.663E+04
BEAM-ME_Instances/2016_22_8	ERROR NO SOLUTION	5,175.38	3,767.83	2.66	3,728.11	3,548.52	1,100.03	18.37	18.69	119.26	219.52	112,857,882	81,947,383	658,167,482	185,476.83	2.460E-07, 5.940E+05	8.298E-03, 2.676E+05	1.946E-06, 1.663E+04
E4SMA	FEASIBLE SOLUTION	1,342.36	1,324.19	2.61	137.14	54.94	1,184.42	0.01	0.00	17.91	19.95	4,229,763	4,437,740	32,382,036	589,428.74	7.402E-16, 4.407E+06	9.557E-06, 1.000E+09	1.258E-09, 1.610E+06
GAZNAT_V1_Instance	FEASIBLE SOLUTION	14,502.81	14,490.08	3.00	136.78	92.31	14,350.28	0.01	0.00	19.16	20.62	5,336,684	4,555,729	34,188,921	370,362.69	1.566E-14, 2.291E+05	3.516E-09, 3.000E+06	1.623E-07, 3.352E+06
Offshore	FEASIBLE SOLUTION	823.94	814.00	0.52	383.74	110.39	429.72	0.03	0.00	7.30	8.16	3,326,614	2,091,377	13,911,856	126,024.60	1.000E+00, 6.000E+03	NA, NA	2.546E-05, 2.735E+06
POLIZERO_V1_Instance	FEASIBLE SOLUTION	2,833.50	2,812.67	4.55	233.69	113.12	2,574.42	0.02	0.00	22.63	24.68	5,640,942	6,991,820	48,190,465	425,993.06	7.918E-19, 5.440E+05	3.837E-19, 1.000E+07	7.309E-08, 4.178E+06
POLIZERO_V2_Instance	FEASIBLE SOLUTION	4,840.59	4,793.86	4.02	412.39	185.81	4,377.44	0.02	0.00	36.28	40.14	10,540,301	13,847,195	69,064,648	371,691.00	1.918E-18, 8.825E+06	1.823E-19, 2.919E+07	9.620E-19, 2.269E+06
TIMES-DK-Instance	OPTIMAL	153.45	152.11	2.38	16.03	7.80	133.67	0.03	0.00	2.64	4.01	867,042	1,031,037	4,740,351	608,049.13	3.577E-08, 7.453E+04	4.606E-05, 4.500E+04	1.909E-09, 4.282E+06
UK-TIMES	OPTIMAL	99.78	97.47	0.89	19.42	10.80	77.14	0.02	0.00	3.02	3.29	875,780	932,609	6,883,510	637,539.13	4.659E-05, 6.697E+05	6.000E-04, 5.867E+05	2.793E-11, 3.746E+05

v4.7.4

Instance	Model Status	Elapsed Time [s]	Total Seconds [s]	Compilation Time [s]	Execution Time [s]	Generation Time [s]	Solver Time [s]	Startup Time [s]	Closedown Time [s]	Highwater RSS [GB]	Highwater VSS [GB]	# Rows	# Columns	# Non Zeros	# Non Zeros per s	RHS [min, max]	Bound [min, max]	Matrix [min, max]
1p5C_OS_SSP2	FEASIBLE SOLUTION	77.61	72.67	2.02	39.61	12.81	31.02	0.03	0.00	2.84	3.20	624,436	863,666	4,650,991	363,018.34	9.222E-05, 3.258E+06	7.001E-06, 6.612E+07	6.977E-09, 2.535E+06
BAU_IND_V2	OPTIMAL	6,591.76	6,578.73	3.06	137.61	86.52	6,438.05	0.02	0.00	22.79	26.25	6,750,106	6,399,364	40,479,629	467,891.45	1.566E-14, 2.285E+05	3.516E-09, 5.000E+04	1.623E-07, 3.352E+06
BEAM-ME_Instances/672_11_8	OPTIMAL	12,032.29	12,004.11	0.72	299.19	199.52	11,704.19	0.02	0.00	63.53	70.15	21,108,711	15,111,159	124,095,043	621,980.41	4.097E-06, 5.940E+05	2.680E-02, 2.676E+05	1.882E-04, 1.663E+04
BEAM-ME_Instances/672_22_8	INTERMEDIATE INFEASIBLE	43,846.96	43,790.44	0.94	555.25	349.84	43,234.23	0.01	0.00	118.06	166.59	37,645,140	27,331,345	218,532,379	624,656.64	1.405E-06, 5.940E+05	8.298E-03, 2.676E+05	2.675E-04, 1.663E+04
BEAM-ME_Instances/2016_22_8	ERROR NO SOLUTION	2,912.60	1,594.75	2.61	1,550.64	1,462.36	1,032.20	22.92	18.58	118.58	219.52	112,857,882	81,947,383	658,134,218	450,049.69	2.460E-07, 5.940E+05	8.298E-03, 2.676E+05	1.946E-06, 1.663E+04
E4SMA	FEASIBLE SOLUTION	1,399.33	1,380.98	2.03	135.31	54.58	1,243.61	0.03	0.00	17.90	19.94	4,229,763	4,437,740	32,382,036	593,316.65	7.402E-16, 4.407E+06	9.557E-06, 1.000E+09	1.258E-09, 1.610E+06
GAZNAT_V1_Instance	FEASIBLE SOLUTION	11,833.81	11,821.22	3.03	108.48	66.00	11,709.69	0.01	0.00	19.39	21.28	5,336,684	4,555,729	34,188,921	518,013.95	1.566E-14, 2.291E+05	3.516E-09, 3.000E+06	1.623E-07, 3.352E+06
Offshore	FEASIBLE SOLUTION	469.40	459.50	0.81	130.28	111.31	328.38	0.03	0.00	7.33	8.17	3,326,614	2,091,377	13,911,856	124,979.62	1.000E+00, 6.000E+03	NA, NA	2.546E-05, 2.735E+06
POLIZERO_V1_Instance	OPTIMAL	2,609.86	2,590.11	4.55	215.27	108.83	2,370.27	0.03	0.00	22.62	24.67	5,640,942	6,991,820	48,190,465	442,813.11	7.918E-19, 5.440E+05	3.837E-19, 1.000E+07	7.309E-08, 4.178E+06
POLIZERO_V2_Instance	FEASIBLE SOLUTION	3,519.83	3,471.78	4.03	371.30	140.61	3,096.44	0.02	0.00	36.08	40.24	10,540,301	13,847,195	69,064,648	491,182.27	1.918E-18, 8.825E+06	1.823E-19, 2.919E+07	9.620E-19, 2.269E+06
TIMES-DK-Instance	OPTIMAL	159.24	157.89	1.91	15.98	7.84	139.98	0.02	0.00	2.65	3.21	867,042	1,031,037	4,740,351	604,328.28	3.577E-08, 7.453E+04	4.606E-05, 4.500E+04	1.909E-09, 4.282E+06
UK-TIMES	OPTIMAL	97.74	95.39	0.88	19.45	10.75	75.05	0.01	0.00	3.04	3.33	875,780	932,609	6,883,510	640,326.51	4.659E-05, 6.697E+05	6.000E-04, 5.867E+05	2.793E-11, 3.746E+05

FFiand / TIMES_model

ETSAP TIMES Code Review Project #1

Evaluation of Base Line / Ground Truth runs

Identification of Slow Equations and Assignments

Evaluation of `SolveLink` Impact

Relative Difference obtained by `SolveLink=0`

Relative Difference obtained by `SolveLink=5`

New `SolveLink=5`

New `SolveLink=0`

Ground Truth `SolveLink=2`

Freegamsmodel

Relative Difference obtained by `freegamsmodel 1`

`SL=0` & `freegamsmodel`

`SL=0`

Names No

Relative Difference obtained by `names no`

New `names no`

Ground Truth

v4.7.9

Improvement v4.7.8 vs v4.7.9

v4.7.9

v4.7.8

Offshore

E4SMA

TIMES Code Review

Relative Difference obtained by #4

New

Ground Truth v4.7.9

Improvement

New

Ground Truth

Final Results

Improvement from v4.7.4 to v4.8.0

v4.8.0

v4.7.4

FFiand / TIMES_model

ETSAP TIMES Code Review Project #1

Evaluation of Base Line / Ground Truth runs

Identification of Slow Equations and Assignments

Evaluation of SolveLink Impact

Relative Difference obtained by SolveLink=0

Relative Difference obtained by SolveLink=5

New SolveLink=5

New SolveLink=0

Ground Truth SolveLink=2

Freegamsmodel

Relative Difference obtained by freegamsmodel 1

SL=0 & freegamsmodel

SL=0

Names No

Relative Difference obtained by names no

New names no

Ground Truth

v4.7.9

Improvement v4.7.8 vs v4.7.9

v4.7.9

v4.7.8

Offshore

E4SMA

TIMES Code Review

Relative Difference obtained by #4

New

Ground Truth v4.7.9

Improvement

New

Ground Truth

Final Results

Improvement from v4.7.4 to v4.8.0

v4.8.0

v4.7.4

Evaluation of `SolveLink` Impact

Relative Difference obtained by `SolveLink=0`

Relative Difference obtained by `SolveLink=5`

New `SolveLink=5`

New `SolveLink=0`

Ground Truth `SolveLink=2`

Relative Difference obtained by `freegamsmodel 1`

`SL=0` & `freegamsmodel`

`SL=0`

Relative Difference obtained by `names no`

New `names no`