Databricks AutoML toolkit

[vipmaax]

Toolkit for Apache Spark ML for Feature clean-up, feature Importance calculation suite, Information Gain selection, Distributed SMOTE, Model selection and training, Hyper parameter optimization and selection, Model interprability.

https://github.com/databrickslabs/automl-toolkit

1) github stars: 161

2) last release: 0.8.1 (may 26, 2021) first release: 0.7.0 (Feb 23, 2020)

3) Language: Scala 2.12, python (pyspark)

Currently supported models:

"XGBoost" - XGBoost Classifier or XGBoost Regressor

"RandomForest" - Random Forest Classifier or Random Forest Regressor

"GBT" - Gradient Boosted Trees Classifier or Gradient Boosted Trees Regressor

"Trees" - Decision Tree Classifier or Decision Tree Regressor

"LinearRegression" - Linear Regressor

"LogisticRegression" - Logistic Regressor (supports both Binomial and Multinomial)

"MLPC" - Multi-Layer Perceptron Classifier

"SVM" - Linear Support Vector Machines

"LightGBM" (currently suspended, pending library improvements to LightGBM) LightGBM

https://github.com/databrickslabs/automl-toolkit/tree/master/src/main/scala/com/databricks/labs/automl/model

[se-teryoshkin]

1. GitHub: Stars - 161, Forks - 35, Contributors - 7

2. Releases: Last Release - 0.8.1 (28th May 2021), First Release - (7th March 2020)

3. Language: Scala, Python (pyspark)

4. Data Types: Structured

5. Execution backends: Spark

6. Distributed framework: Spark

7. Distributed algorithms libraries: SparkML

8. AutoML pipeline elements:

• Feature clean-up
• Feature Importance calculation suite
• Feature Interaction with Information Gain selection
• Feature vectorization
• Advanced train/test split techniques
• Model selection and training
• Hyper parameter optimization and selection
• Batch Prediction
• Logging of model results and training runs
• Model interprability

9. Elements of pipeline are covered in distributed mode: all

10. Is it possible to mix distributed and non-distributed backends?: No

11. How is the pipeline converted to computing applications?: Distributed application

12. Approach to ML models selection computing: Multiple distributed models ?

13. Data processing steps pruning: Yes

14. Resource managers: Spark's resource managers

15. Hyperparameter tuner: Spark ML

16. Task types: Classification, Regression

17. Tests: local mode, distributed mode.

18. Comments: --