DCS-Rank, DCS-OLA, DCS-LCA and DCS-MLA keep producing the same result

[jayahm]

Hi,

It is weird that Rank and OLA produced the same result. Also, LCA and MLA produced the same result

You can see an example below (in example_heterogeneous.ipynb) https://www.dropbox.com/sh/8fhd97flal4d076/AADqIL9cvP8_2z-Sbue1rLg0a?dl=0

There are two different datasets. But, both produced similar behaviour.

This is just 2 examples.

It happened to all datasets that I have.

[Menelau]

@jayahm Hello,

Actually it is not strange that these techniques produce the same results for a few datasets, in fact, their classifier selection mechanism is quite correlated. You can see an analysis in this paper that shows how the OLA, LCA, and MLA are quite similar and sometimes gives exactly the same result:

Analyzing Dynamic Ensemble Selection Techniques Using Dissimilarity Analysis https://link.springer.com/chapter/10.1007/978-3-319-11656-3_6

[jayahm]

@Menelau

I see.

Is there any reason why Rank = OLA and LCA = MLA. And this happens in many datasets.

For example:

Dataset 1

Evaluating DS techniques:
Classification accuracy of RANK:  0.7040816326530612
Classification accuracy of OLA:  0.7040816326530612
Classification accuracy of LCA:  0.38571428571428573
Classification accuracy of MLA:  0.38571428571428573
Classification accuracy of KNORAE:  0.7979591836734694

Dataset 2:

Evaluating DS techniques:
Classification accuracy of RANK:  0.702928870292887
Classification accuracy of OLA:  0.702928870292887
Classification accuracy of LCA:  0.46234309623430964
Classification accuracy of MLA:  0.46234309623430964
Classification accuracy of KNORAE:  0.8326359832635983

[Menelau]

My guess here is that LCA and MLA are "a posteriori methods", in other words, it takes into account which class the base classifier c_i is predicting and computes its competence level only with respect to this class (ignoring whether it correctly classified or misclassified examples from different classes). In our experience, this property found in these techniques does not work well for datasets that suffer from class imbalance and are more sensitive to noise.

OLA and Rank on the other hand are "a priori methods", they consider examples of all classes in the competence level estimation. That makes these methods more robust in those scenarios.

The reason for these pair of techniques having exactly the same results can be due to the size of the pool of classifiers as well as the pool of classifiers not having enough diversity (and consequently not having multiple base classifiers that are local experts) for distinct data points.