Scatter plot of ranks as a more readable contour plot

[contramundum53]

Motivation

Current contour plot looks like this with around 2000 trials and 8 parameters.

(Plotly version freezes my VSCode so I show the matplotlib version.)

These figures are not very readable and helpful. There are several points to improve:

• The plotted contour is currently based on interpolation of values, in the projected plane of parameters. This is not mathematically meaningful (e.g. this is not the value the sampler predicted, nor is it the result of some kind of regression), and it produces ugly noises especially when the objective function is noisy. We could remove the contour altogether.
• The points currently have no information about the objective values. We could color the points based on them.
• The figure size currently does not scale up when there are many parameters. Instead, we should make the figure size proportional to the number of parameters.
• The order of the parameters are the default order of parameters. In many cases, it is difficult to know which figure out of the N(N-1) figures you should look at. Instead, we could add an option to sort the parameters in the order of importance, and move the important figures in the upper left corner.
• The scaling of objective values are important to make the figure readable. As a robust scaling method, we could use ranks of the objective values.

I improved these points to create this figure.

From these plots, we can analyze that 1. the optimization landscape is very smooth, 2. the parameters are pretty independent, 3. the last three parameters are less important than other parameters, and 4. we already have enough samples to explore the search space.

[c-bata]

@contramundum53 Cool! We welcome your contribution 👍