Decide on document parameters that we can use to describe a dataset

[wpertsch]

How many parameters do we need? 20-30! We can use https://github.com/Evenaar/active-learning-dataset-benchmark to find those parameters --> will be ready in a few days.

[jembie]

These are all the features which are currently considered for implementation. They are orientated from the master thesis' work and we are in contact / collaboration with them. See documents/data_analysis.md for details about the features that are mentioned here.

simple metafeatures

• [x] dimensionality/number of attributes (features), p
• [x] scale / number of examples (instances), n
• [ ] number of classes, c
• [ ] proportion of discrete attributes;
• [ ] proportion of missing values of feature xi;
• [ ] proportion of outliers of feature xi.
• [ ] number of examples per class n/c

statistical metafeatures

• [x] Skewness of xi;
• [x] Kurtosis of xi;
• [ ] Probability of class y.
• [x] correlation of xi and xj , p(xi; xj) --> provide an estimate of feature interdependence
• [x] covariance of xi and xj --> provide an estimate of feature interdependence
• [ ] (concentration of xi and xj)

Information-theoretic metafeatures

• [x] Feature entropy
• [ ] class entropy -> provides an estimate of difficulty of classification task and class imbalance (second can be replaced with imbalanced ratio)
• [ ] mutual information between xi and y, MI(xi,y)
• [ ] Intrinsic task dimensionality
• [ ] Noise-signal ratio

Concept and complexity-based metafeatures

• [ ] Concept variation/roughness in output space
• [ ] Overlap of individual features (Fisher’s discriminant ratio; Volume overlap region; Feature efficiency)
• [ ] Separability of classes (Linear separability, silhouettescore)

Establishing Similarity Between Datasets

• [ ] Similarity based on metafeatures
• [ ] Similarity based on performance results of algorithms
• [ ] Item Response Theory and True Score

[wpertsch]

Most of the metrics are relying on the class, but also on the label. We can't use those and have to decide what we want to keep/what we can recycle to our advantage.