search

microsoft / onnxruntime

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
https://onnxruntime.ai
MIT License
14.46k stars 2.9k forks source link

Need to be able to explain predictions from ONNX models (aka need Shapley values from onnxruntime) #1176

Closed jim-meyer closed 4 years ago

jim-meyer commented 5 years ago

Is your feature request related to a problem? Please describe. Many ML frameworks supported by ONNX provide Shapley values . These values are import for explaining a model's predictions. LightGBM optionally provides them if the predict() function's pred_contrib parameter is set to True. The shap python package supports providing the values for most tree ensembles like XGBoost, sklearn, LightGBM. But onnxruntime does not provide a way to get these values.

System information

Describe the solution you'd like A clear and concise description of what you want to happen. See above.

Describe alternatives you've considered I looked at using the 'shap' python package to get the Shapley values. But it doesn't support ONNX.

Additional context

faxu commented 5 years ago

Thanks for sharing the feature request. We'll discuss this and get back to you soon.

faxu commented 4 years ago

Explainability will not be part of the core ONNX Runtime. If you need to explain a model, you can try the resources/tools from https://github.com/interpretml/interpret-community. Any model can be explained using those tools provided there is a wrapper that exposes predict and predict_proba according to the scikit convention. There are notebook examples in that repo, and it can be used in conjunction with ONNX Runtime for explaining your model.

MilesZhao commented 1 year ago

Not sure whether it's too late. We just released the paper ONNXExplainer and it can explain an arbitrary neural network in ONNX ecosystem. We will release the code soon once it's approved.

tadhgpearson commented 1 year ago

I just read your paper - this is awesome, well done. I'd really look forward to using this. Can this also explain trees, or only neural networks?

MilesZhao commented 1 year ago

I just read your paper - this is awesome, well done. I'd really look forward to using this. Can this also explain trees, or only neural networks?

Thank you for your interest in our work. Once our legal team approves the publishing of the codebase, I will share the link here. Currently, we do not support TreeSHAP which is used in explaining tree models. Nevertheless, we definitely on the way to add the support for TreeSHAP, also fast TreeSHAP and linear TreeSHAP. If I am understanding correctly, ONNX does not support recursive functions. I am figuring out a way to convert the super complicated recursive process to iterative process. Stay tuned for further updates.

tadhgpearson commented 1 year ago

Cool, I look forward to it! I'm happy to help if you're stuck on unrolling the recursion - it is usually possible with a stack and a loop but it sometimes gets messy :D We can continue this correspondence by email if you like - can I contact you at the email address in the header of your research paper?