search

dotnet / machinelearning

ML.NET is an open source and cross-platform machine learning framework for .NET.
https://dot.net/ml
MIT License
9.05k stars 1.89k forks source link

New PFI API, inconstancies in return types for Binary vs Regression and MultiClass #6027

Closed aforoughi1 closed 2 years ago

aforoughi1 commented 2 years ago

System Information (please complete the following information):

Describe the bug I tried to write a generic method to Calculate PFI using the New PFI API which was released as part of Ml.Net 1.7. Firstly, The online documentation is not matching the release. See https://docs.microsoft.com/enus/dotnet/api/microsoft.ml.permutationfeatureimportanceextensions Secondly, the return types for Regression and Multiclass is Dictionary but Binary is an Array ImmutableDictionary<string,RegressionMetricsStatistics> ImmutableDictionary<string,MulticlassClassificationMetricsStatistics> Immutable.ImmutableArray

Is this a bug ?

aforoughi1 commented 2 years ago

Apologies, It is not the return type issue but the compiler reports that 'BinaryClassificationCatalog' does not contain a definition for PermutationFeatureImportance.

michaelgsharp commented 2 years ago

Can you double check? All the extension for PFI are in the same file, so if you have them for the others you should have them for BinaryClassificationCatalog as well. Make sure you are using version 1.7.

aforoughi1 commented 2 years ago

yes, I am using 1.7.0 please see the attached test sample Test6027.zip

michaelgsharp commented 2 years ago

Ah, the method name isn't just PermutationFeatureImportance, its PermutationFeatureImportanceNonCalibrated. That was done because the original PermutationFeatureImportance is specifically evaluating "non calibrated" metrics, so we renamed the new PFI API to correctly show that and to leave room to add one that is for "calibrated" metrics. If you change the method you are calling to be PermutationFeatureImportanceNonCalibrated it will work correctly.

Closing this issue for now since this resolves the issue, but feel free to reopen it if you have further issues.

aforoughi1 commented 2 years ago

I changed the sample to var pfiDict = context.BinaryClassification.PermutationFeatureImportanceNonCalibrated(model, predictions); however, I get System.ArgumentNullException: 'The model provided does not have a compatible predictor Arg_ParamName_Name'

JakeRadMSFT commented 2 years ago

@michaelgsharp I'm also hitting this issue

JakeRadMSFT commented 2 years ago

TaxiModel_Notebook5.txt

(Save as IPYNB)

The PFI cell results in:

System.ArgumentNullException: The model provided does not have a compatible predictor (Parameter 'lastTransformer')
   at Microsoft.ML.Runtime.Contracts.CheckValue[T](IExceptionContext ctx, T val, String paramName, String msg)
   at Microsoft.ML.PermutationFeatureImportanceExtensions.PermutationFeatureImportance[TMetric,TResult](IHostEnvironment env, ITransformer model, IDataView data, Func`1 resultInitializer, Func`2 evaluationFunc, Func`3 deltaFunc, Int32 permutationCount, Boolean useFeatureWeightFilter, Nullable`1 numberOfExamplesToUse)
   at Microsoft.ML.PermutationFeatureImportanceExtensions.PermutationFeatureImportance(RegressionCatalog catalog, ITransformer model, IDataView data, String labelColumnName, Boolean useFeatureWeightFilter, Nullable`1 numberOfExamplesToUse, Int32 permutationCount)
   at Submission#22.<<Initialize>>d__0.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.CodeAnalysis.Scripting.ScriptExecutionState.RunSubmissionsAsync[TResult](ImmutableArray`1 precedingExecutors, Func`2 currentExecutor, StrongBox`1 exceptionHolderOpt, Func`2 catchExceptionOpt, CancellationToken cancellationToken)