SnAr, domain limits and hypervolume nadir points

[konkouz]

Description

On the definition of the SnAr domain:

1. the target limits set between 0 - 100 for yield and 0-10 for the e-factor but much higher values are reported in the graphs and tables. Could you please provide the domain limits that correspond to the predictions ?

When computing hypervolumes, can you please explain:

1. how the reference/nadir point is defined?
2. why you have used this point [-2957, 10.7] 4th cell from the end in : https://colab.research.google.com/github/sustainable-processes/summit/blob/master/experiments/snar_benchmark/visualization.ipynb

[marcosfelt]

Hi Kostas!

First question just needs a change to this line: https://github.com/sustainable-processes/summit/blob/f4df0ee8edd6d4cde8226a6de8db8f2a2a23abe1/summit/benchmarks/snar.py#L82

Probably the limits should be something like 0 -130000

Second answer is in the SI of our paper:

I hope that helps!

[konkouz]

Hello again Kobi, Thank you very much for your reply ! In regard to the target limits, did you find setting different limits had an effect on your results ? If yes, could you please share the exact limits you used for the published results so we can compare within identical domains? Do you also happen to know what the limits are for the e-factor ?

On the nadir point this is helpful, however I want to know some more on the why. In specific, the nadir point is stated as "estimated" could you please explain the thought process behind this estimation ? If we are to use the same dataset but a different tool than summit (optimization algorithm) would you recommend still using that nadir point ? Can you think of any other scenarios where a different nadir point should be considered?

Many thanks for your time once again :)

[marcosfelt]

The target limits have no effect on the results; neither the benchmark or the strategies currently used in Summit make use of the limits.

Since we don't know the exact pareto front of the benchmark, we estimated it using a optimization algorithm (NSGA-II) with a large number of trials. This estimated pareto front was used to calculate the Nadir point. The estimated Nadir point is only used for post-hoc evaluation of the hypervolume and not by any of the optimization algorithms. Therefore, if you want to compare the performance of a new optimization algorithm to those benchmarked in Summit, I suggest using the estimated Nadir point when calculating hypervolume.

I hope that makes sense!

[konkouz]

Hello again Kobi, Just to make sure I've understood the procedure. The iterative process/trials you run with the optimization algorithm (NSGA-II), are for the definition of the pareto front. Then after this has been defined, you filter the dominated points out, and you take the max_x and max_y of the non-dominated points as your reference point x and y.

Is the above correct ? If not, do you run trials specifically for the calculation of the reference point ? If yes, should the non-dominated points not include any that are at y's (e-factors) larger than 10.7 in this case ?

Once again many thanks, and please do feel free to reach out for any clarifications on this :)

[marcosfelt]

Just to make sure I've understood the procedure. The iterative process/trials you run with the optimization algorithm (NSGA-II), are for the definition of the pareto front. Then after this has been defined, you filter the dominated points out, and you take the max_x and max_y of the non-dominated points as your reference point x and y.

Yes, this correct.

If yes, should the non-dominated points not include any that are at y's (e-factors) larger than 10.7 in this case ?

No, you don not need to clip the values at 10.7. With a recent commit, this should be fixed (i.e., the bounds are much larger). I'll be pushing a new version to pypi in the next couple of days.

https://github.com/sustainable-processes/summit/blob/34162c8157d56ee860631ac7b713edfbe9c21e3a/summit/benchmarks/snar.py#L91