search

automl / Auto-PyTorch

Automatic architecture search and hyperparameter optimization for PyTorch
Apache License 2.0
2.36k stars 284 forks source link

Generate a leaderboard for each model? #495

Open caimiao0714 opened 1 year ago

caimiao0714 commented 1 year ago

This is actually a question rather than an issue. I also tried other AutoML frameworks such as AutoGluon and H2O AutoML, which both have functions producing a leader board (roc_auc, accuracy, log loss, etc.). However, I do not find an equivalent function on Auto-PyTorch in the official documentation. I wonder if such a function exit in Auto-PyTorch?

Below I'm showing examples of leaderboard in AutoGluon and H2O AutoML.

Leaderboard from AutoGluon:

                      model score_test   roc_auc  accuracy   log_loss score_val pred_time_test pred_time_val     fit_time pred_time_test_marginal pred_time_val_marginal fit_time_marginal stack_level can_infer fit_order
 1:       LightGBMXT_BAG_L1  0.7590175 0.7590175 0.9742501 -0.1082179 0.7645787     24.7286382      28.34122 1.159951e+02             24.72863817             28.3412232      1.159951e+02           1      TRUE         3
 2:     WeightedEnsemble_L2  0.7588045 0.7588045 0.9742501 -0.1082454 0.7656998    401.1412535     219.29514 2.083729e+04              0.01033139              0.2191315      1.726000e+02           2      TRUE        12
 3:  NeuralNetFastAI_BAG_L1  0.7587008 0.7587008 0.9742501 -0.1082499 0.7648229    209.7337284      72.73679 6.181634e+03            209.73372841             72.7367890      6.181634e+03           1      TRUE         9
 4:   NeuralNetTorch_BAG_L1  0.7586281 0.7586281 0.9742501 -0.1082789 0.7656290    353.2530944     174.99615 2.044177e+04            353.25309443            174.9961455      2.044177e+04           1      TRUE        10
 5:         LightGBM_BAG_L1  0.7580392 0.7580392 0.9742501 -0.1083533 0.7637019     26.2219682      18.38476 8.965047e+01             26.22196817             18.3847609      8.965047e+01           1      TRUE         4
 6:    LightGBMLarge_BAG_L1  0.7574986 0.7574986 0.9742501 -0.1084190 0.7630075     21.6558595      25.69510 1.332672e+02             21.65585947             25.6951005      1.332672e+02           1      TRUE        11
 7:   ExtraTreesGini_BAG_L1  0.7317531 0.7317531 0.9741663 -0.1527889 0.7362975      0.4827056      18.03142 8.914248e+00              0.48270559             18.0314159      8.914248e+00           1      TRUE         7
 8: RandomForestEntr_BAG_L1  0.7315053 0.7315053 0.9741607 -0.1523725 0.7367013      0.5803573      19.19548 8.536943e+00              0.58035731             19.1954811      8.536943e+00           1      TRUE         6
 9:   ExtraTreesEntr_BAG_L1  0.7313867 0.7313867 0.9741719 -0.1529712 0.7365097      0.4960597      17.40264 9.324817e+00              0.49605966             17.4026434      9.324817e+00           1      TRUE         8
10: RandomForestGini_BAG_L1  0.7313856 0.7313856 0.9741607 -0.1524167 0.7365686      0.4857991      19.06845 8.353514e+00              0.48579907             19.0684481      8.353514e+00           1      TRUE         5
11:   KNeighborsDist_BAG_L1  0.5295357 0.5295357 0.9742501 -0.7495739 0.5245484    404.8888049     490.56608 5.202553e-01            404.88880491            490.5660815      5.202553e-01           1      TRUE         2
12:   KNeighborsUnif_BAG_L1  0.5295357 0.5295357 0.9742501 -0.7495739 0.5245486    421.0486317    1249.74371 5.273466e-01            421.04863167           1249.7437119      5.273466e-01           1      TRUE         1

Leaderboard from H2O AutoML:

                                                  model_id       auc   logloss     aucpr mean_per_class_error      rmse       mse training_time_ms predict_time_per_row_ms         algo
  1:        XGBoost_grid_1_AutoML_4_20230314_14900_model_2 0.6417932 0.4666353 0.3618462            0.4057249 0.3836492 0.1471867              845                0.004340      XGBoost
  2:                         GBM_2_AutoML_4_20230314_14900 0.6412184 0.4667908 0.3611882            0.4052551 0.3837011 0.1472265              602                0.017406          GBM
  3:                         GBM_3_AutoML_4_20230314_14900 0.6411061 0.4669249 0.3608521            0.4050711 0.3837799 0.1472870              613                0.012203          GBM
  4:                     XGBoost_3_AutoML_4_20230314_14900 0.6406661 0.4669883 0.3610508            0.4064628 0.3837821 0.1472887              986                0.003698      XGBoost
  5:            GBM_grid_1_AutoML_4_20230314_14900_model_6 0.6406521 0.4670791 0.3608027            0.4064283 0.3838409 0.1473338              627                0.011967          GBM
 ---                                                                                                                                                                                   
239: DeepLearning_grid_1_AutoML_4_20230314_14900_model_177 0.4058350 2.4122755 0.1671561            0.5000000 0.4448776 0.1979161            28876                0.006443 DeepLearning
240:  DeepLearning_grid_1_AutoML_4_20230314_14900_model_75 0.3926166 2.2398423 0.1600879            0.5000000 0.4448339 0.1978772            36780                0.007232 DeepLearning
241: DeepLearning_grid_1_AutoML_4_20230314_14900_model_191 0.3900886 3.0261471 0.1664013            0.5000000 0.4447404 0.1977940            71510                0.006048 DeepLearning
242: DeepLearning_grid_1_AutoML_4_20230314_14900_model_127 0.3887585 4.0422234 0.1596719            0.5000000 0.4448826 0.1979205            37300                0.006333 DeepLearning
243: DeepLearning_grid_1_AutoML_4_20230314_14900_model_173 0.3759081 2.3878755 0.1563064            0.5000000 0.4448267 0.1978708            33531                0.005443 DeepLearning