Week of 2/6

[everettbu]

2/6

Meeting notes

• Youtube video not the best supplemental tool (Nengfeng would have to get permission from Wells Fargo)
• He really wants us to find weaknesses of the models and present our findings
• He doesn't want us focus too much on the documentation

Overarching Question: Do the interpretable models have comparable performance to black box models?

To do answer this question:

• Focus on testing interpretable models and see how much better black box models work. We really want to see if the PiML models are good enough to use.
• Find any powerful black box models and test them. (Don't overfeed the models)
• Collect concrete figures for how much better the black box models preform compared to the interpretable models
• Use new and existing datasets

Focus on the following models: EBM: Explainable Boosting Machine (Nori, et al. 2019; Lou, et al. 2013) GAMI-Net: Generalized Additive Model with Structured Interactions (Yang, Zhang and Sudjianto, 2021) ReLU-DNN: Deep ReLU Networks using Aletheia Unwrapper and Sparsification (Sudjianto, et al. 2020)

[xybljy0122]

Notes for Feb6: [with client]

Youtube a bit complicated, need manager approval 3.2 PiML team are fixing it anyways 3.4 main goal, understand interpretable model better, compare to black-box, and traditional regression model.

The work flow of 3.4? Interpretable model has constraints, main effect, two way interaction Blackbox, more complicated How much better does black box better than interpretable model, if significant better, why it is The experient, if the interpretable model good enough

When look at black box model - use any model in github (avoid overfeeding)

Eg. Look for examples that interpretable model is much worse than Hypothesis: Interpretable model can usually as good as black box

Financial field: regulation, blackbox is hard to get approval

(focusing on at least ⅔ of the three models) EBM: Explainable Boosting Machine (Nori, et al. 2019; Lou, et al. 2013) GAMI-Net: Generalized Additive Model with Structured Interactions (Yang, Zhang and Sudjianto, 2021) ReLU-DNN: Deep ReLU Networks using Aletheia Unwrapper and Sparsification (Sudjianto, et al. 2020)

UCI data Examples in the email

[everettbu]

Next Steps:

• Understand the three models Nengfeng mentioned as important
• Go through the PiML repo and test out those three models
• Find datasets that we want to use
• Find black box algorithms that we are interested in using

[reinabhatkuly]

Meeting Minutes (01/06/23)

• Discussion of the SOW, specifically the objectives and the stretch objectives
• Unsure about the potential of a YouTube video due to complications with usage of Wells Fargo brand (approval would most likely be needed)
• Objective 3.2 seems supplemental but it isn’t the main objective
• MAIN GOAL: understand the interpretable models and if black box models can do a better job, highlighting performance differences
• Ideally the black box model should perform as interpretables models have more constraints and black box allows for greater complexity → performance indicators that are relative to the magnitude of pros and cons associated with each model
• Experiment with different datasets to see if interpretable model is good enough/sufficient and succinct as black models are difficult to get approved within the financial industry
• Any models present on the GitHub repository can be used, can figure out how to register new models on GitHub and and how to add new datasets to it
• Model comparison measures: built-in within PiML
• What state of the art model can be achieved from experimentation and comparison? What is the best model?
• Models to focus on within PiML (2 of 3): EBM Model, GAMI-Net Model, ReLU-DNN Model

Fintech Practicum_Meeting Minutes .pdf

[agallagher23]

Statement of Work, Spring 2023.pdf

[agallagher23]

Helpful link - http://aix360.mybluemix.net/

This has good overview information and a demo that walks you through a credit approval model

[agallagher23]

Maybe we should focus on having a good understanding of whats specifically happening in each of these steps (of one of their PiML low-code case studies)...

• Load and Prepare Data
• Train Interpretable Models
• Interpretability and Explainability
• Model Diagnostics and Outcome Testing
• Model Comparison and Benchmarking