[feature request] Dimensionality Reduction API

[wbadry]

Features Reduction

It would be great if dimensionality reduction API could be added to ML.NET. This will be a major advantage in shortening training time.

[antoniovs1029]

Thanks for submitting your request. Can you please tell us which dimensionality reduction algorithms would you like to see included?

By the way, ML.NET does provide a PCA Transform: https://docs.microsoft.com/en-us/dotnet/api/microsoft.ml.pcacatalog.projecttoprincipalcomponents?view=ml-dotnet

[wbadry]

PCA is great but from my personal experience, it could destroy classification separation in either binary or multiclass problems.

This is a toolbox made by Laurens van der Maaten in MATLAB covering many dimensionality reductions.

As a start as I am aware your queue is already full of tasks, perhaps LDA and GDA are very good candidates (I tried both with really great projected features output and decent class separation based on this projection).

This is a "work on progress" that I am on it. With projection, I can get such a nice separation based on 110 features from healthy and COVID-19 patients.

[justinormont]

@wbadry: There are various feature selection and feature compression techniques available in ML․NET.

Feature selection:

• Count Feature Selection transform, which removes slots in a column which have few non-default values. E.g. it will keep slots in a Single column which have many values besides 0.0.
• Mutual Information Feature Selection transform, which keeps the slots in a column most correlated to your label column.
• I'm uncertain if it's exposed or not, though there is also a Learner Feature Selection transform, which keeps slots in a column which have the highest feature importance after training a user specified ML model
• Feature Contribution Calculator, which you can use for a fun trick of having it create a model and use its feature importance to do a per-row feature selection (above techniques are per-slot/column); works very well on n-grams.

Feature compression:

• Principal Component Analysis (PCA) transform
• Latent Dirichlet Allocation (LDA) transform
• TreeFeaturizer transform, which learns a tree (random forest / boosted decision tree) and uses the leaf nodes, the example lands in, as a new feature. Output feature count is the number of leaf nodes in the trees. Fun trick is using FastForest w/ multi-class by turning on label rotation; this greatly reduces the number of trees need as each tree is learning all classes. Another fun trick is learning the trees on a sub-sample of your data to learn the feature interactions, then streaming terabytes of data though it using a final linear model; which gives strong, scalable models.
• K-means as a featurizer, training a k-means model and using the distance to each centroid as features; works very well on numeric data, though best to keep also keep the original features
• Model stacking, generalization of the above two, though its output is the class probabilities/scores (very few, but strong features)
• Random Fourier Features transform, uses fourier features to create feature interactions and reduce the features
• Word Embeddings, specifically for text, compresses the feature count from ~100k sparse features for n-grams to a 900 feature dense embedding space

You can also use the CustomMapper or Expression transforms for custom selection/compression.

Related issue: https://github.com/dotnet/machinelearning-modelbuilder/issues/702 (having AutoML try the techniques for you)