[Diversity] New version

[vivekaxl]

How to Read the Chart:

1. For all the measures (GD, IGD and Evals) lower the better.
2. The shaded cell is statistically better than other cells.

AL4

AL5

[vivekaxl]

AL4:

1. Predict scores for all the points
2. Non-dominated Sort (still BDOM)
3. Choose point whose predict score is furthest away from the already sample points (Intuition: this would help in increasing diversity)

AL5: (Faster Version of AL4)

1. Predict scores for all the points (in batched for 128)
2. Non-dominated Sort (still BDOM)
3. Choose point whose predict score is furthest away from the already sample points (Intuition: this would help in increasing diversity)

Other difference from AL2

AL2 the approximated Pareto front is the (first level) of (Deb's) non-dominated sort. For most cases, this is a really low number. So, one of the strategies to increase IGD will be to increase the size of the approximated Pareto Front. In this version, we use all the points which were sampled.

[vivekaxl]

Observation:

1. It is expected that IGD of AL4/AL5 should be better (lower) than AL2 -- it is observed that the AL4/AL5 is never worse than AL2. This means that strategy defined in (AL4, AL5)-3 is useful.
2. It is also expected that GD of AL4/AL5 should be worse. This is because for AL4/AL5 the approximated Pareto Front is all the sampled points. It is also observed that AL4/Al5 has indeed worse GD scores.
3. There is no significant difference between scores of Al4 and AL5. This means that faster version (AL5) is good enough and makes FLASH more FLASHey.

[vivekaxl]

For Next iteration of FLASH

1. A better strategy for converting sampled points to approximated Pareto Front (undecided).
2. Sample more points: I want to enforce a budget of 100 (population size of EAs).

[timm]

lets get runtimes into the above charts. something called flash should be fast

does al4, al5 correspond to my flash2 or flash3 or r u going in a different direction? which is fine btw just as long as flash's runtimes really are flash

[timm]

i agree that AL4==AL5. but do i get AL5? each generation you just eval 128 instances?

your new diversity method seems to be ungood.

• you were winning on MONRP before and you are still winning
• AL5 < AL2. compare the AL2, AL5 IGD columns (so not sure why you say "it is observed that the AL4/AL5 is never worse than AL2")

so i would remove your new diversity operator . have you considered:

• guessing of 256, not 128 items
• evaluating the instance with the lowest and highest scores?

also, what is IGD,GD of the baseline? i.e. is a "bad" igd of 0.78 actually pretty good since we started at, say, 200?