Week 2/13

[reinabhatkuly]

Items Completed -Successfully uploaded a dataset of Google stock prices over the past 10 years on PiML on Google Collab (files to be uploaded onto Google Collab must be 10MB or smaller). -Ran the entirety of the low code setup on the dataset to check functionality. -Tried uploading multiple file formats found CSV files to be the only format that could be uploaded onto Google Collab. Still attempting to look for more accepted file formats. -Found kaggle.com to have a useful data repository

[everettbu]

Items Completed

• Successfully trained an arbitrary black-box model, LightGBM, on the California housing dataset
• Extensively compared the LGBM model to GAMI-Net model by generating all of the comparative graphs in a high-code environment
• https://colab.research.google.com/drive/1dTLhhjiQiAaTfViL3SqxbDtLr_eGcI6l?usp=sharing

[agallagher23]

Wells Fargo Meeting Minutes (02/13/23)

Feedback on SOW:

• Slight changes to the project background regarding concerns/restrictions associated with black box models
• The project background shouldn’t just state lending decisions - its to see how robust each model is (lending decisions association is misleading) - remove the line about lending decisions
• Objectives are in a good position
• Change the week numbers to dates (feedback from Mike)
• Potentially do Week 4 and 6 objectives in parallel to achieve more stretch objectives

What should we be looking for in the black box models and how should we compare it to the PiML models?

• Notable findings for analysis: compare model performance by comparing regression errors, compare testing data rather than tuning data, model weaknesses (weaker performance in certain areas relative to other models), situations where black box models perform better and why (is it something interpretable models can’t do?)
• Try different datasets and how models perform with them

Everrett shared his screen to show confirm with Nengfeng that we got the external black box model successfully registered in PiML

• Compare performance on the testing data NOT training data
• Use the comparison tools inside the PiML
• Make sure hyperparameters are optimized for both interpretable and black box models
• Black box will most likely not have hugely better results
• Try to do everything in low code environment/only use high code if need it

Next Steps

• Try to register a black box model in low code?
  • Email errors and code to Nengfeng for cross checking with PiML team when run into problems
• Find a new dataset
  • Datasets should have 10,000- 50,000 observations to ensure the data doesn't run too slowly
• Research on how the results from each model differ to understand pros and cons of the models better
• Send Nengfeng mid-week updates
• Create meeting agendas prior to Monday meetings

[agallagher23]

Mike Meeting Minutes (02/14/23)

Logistics

• Concrete deliverable is for our team to learn
• Everyone should individually post their agenda for the week

This week’s deliverables

1. Send NF new SOW
2. Interpret the graphs that the library outputs
   • Stick with the housing dataset (hardest to explain will be lat/long)
   • Every person picks one
3. Midterm presentation
   • Will present to NG, Rosh, other Wells group
   • 80% SOW, 20% new things
   • Go through each section
   • At end some of graphs and explanation
   • Statement about future plans
   • Appropriate text size and white space, no complete sentences

Stretch tasks

4. Create a files that has both the new dataset and blackbox model
   • Can be either high or low code
5. Finalized list of datasets
   • Five at the most
   • Have a large diversity between each other
     • Where is a gap in their datasets?
   • Some classification and some regression

[xybljy0122]

High Code Stage 3 Model Explain and Interpret

textbook: https://christophm.github.io/interpretable-ml-book/index.html

Permutation Feature Importance: 8.5 Everett Partial Dependence Plot Feature Importance: 8.1 Aly Individual Conditional Expectation (ICE): 9.1 Jenny Accumulated Local Effects: 8.2 Reina