numbbo / coco

Numerical Black-Box Optimization Benchmarking Framework
https://numbbo.github.io/coco
Other
254 stars 85 forks source link

Convergence graphs of single runs #1278

Open brockho opened 7 years ago

brockho commented 7 years ago

It will be nice to provide convergence graphs of all runs instead of only the median convergence graph. This will be definitely handy for designing algorithms. Additional plots such as the dispersion over time will be nice-to-have as well.

nikohansen commented 5 years ago

~For a start, this should be done at least for the first function and the first algorithm by default.~

The simplest implementation route is probably to add the single instance-wise graphs to the median figures (with an option to turn the instance-wise plots off).

nikohansen commented 5 years ago

This is a recipe to plot all runs on F6 in dimension 20 in the first data set of the first "known archive":

%matplotlib ipympl
import cocopp
import cocopp.archiving as ac

dsd = cocopp.load(ac.get(ac.ArchivesKnown()[0]).get(0)).dictByDimFunc()
dsd[20][6][0].plot()

This will work soon on the development branch.

nikohansen commented 5 years ago

@dtusar as you labeled this issue as easy, how do we now go about creating an html page with all these (24x6-or-so) figures included?

nikohansen commented 5 years ago

FTR, the current implementation of the plot method used above:

    def plot(self, plot_function=plt.semilogy, smallest_target=8e-9,
             median_format='k--', color_map=None, **kwargs):
        """plot all data from `evals` attribute and the median.

        Plotted are Delta f-value vs evaluations. The color heatmap is
        based on the final performance.

        `color_map` is called to create an array of rgb values like with
        the `matplotlib.colors.LinearSegmentedColormap` attributes of
        module `matplotlib.cm`. Default is `brg` between 0 and 0.5.

        `**kwargs` is passed to `plot_function`.
        """
        kwargs.setdefault('clip_on', False)
        colors = iter(color_map or plt.cm.brg(np.linspace(0, 0.5, self.nbRuns())))
        # colors = iter(plt.cm.plasma(np.linspace(0, 0.7, self.nbRuns())))
        for i in self._argsort(smallest_target):  # ranges from 1 to nbRuns included
            evals = self.evals.T[i]  # i == 0 are the target values
            idx = np.logical_and(self.evals[:, 0] >= smallest_target, np.isfinite(evals))
            plot_function(evals[idx], self.evals[idx, 0], color=next(colors), **kwargs)
        xmedian = self.median_evals()
        idx = np.logical_and(self.evals[:, 0] >= smallest_target, np.isfinite(xmedian))
        if np.any(idx):
            plot_function(xmedian[idx], self.evals[idx, 0], median_format, **kwargs)
            i = np.where(idx)[0][-1]  # mark the last median with a circle
            plot_function(xmedian[i], self.evals[i, 0], 'ok')
        plt.ylabel(r'$\Delta f$')
        plt.xlabel('function evaluations')
        plt.xlim(left=0.85)  # right=max(self.maxevals)
        plt.ylim(bottom=smallest_target if smallest_target is not None else self.precision)
        plt.title("F %d in dimension %d" % (self.funcId, self.dim))
        return plt.gca()  # not sure which makes most sense
nikohansen commented 5 years ago
def save_single_functions_html(filename,
                               algname='',
                               extension='svg',
                               add_to_names='',
                               dimensions=None,
                               htmlPage=HtmlPage.NON_SPECIFIED,
                               function_groups=None,
                               parentFileName=None,  # used only with HtmlPage.NON_SPECIFIED
                               header=None,  # used only with HtmlPage.NON_SPECIFIED
                               caption=None):  # used only with HtmlPage.NON_SPECIFIED

in cocopp.ppfig seems to be right function to generate a new html page. The precondition is to have saved all figures with filenames like

'%s_f%03d_%02dD.%s' % (filename, function_number, dimension, extension)
nikohansen commented 1 week ago

This should become soonish another default entry in the html root page.