Week 4/3

[agallagher23]

Meeting Notes (04/03) Took a bunch of notes on the google colab within the code

Optimizing LGM:

• Try changing max depth
• LGM Optimized just means much better than EBM

Optimizing EBM:

• First step is finding best possible
• Do that by underfitting and overfitting
• Use number of bins/interactions
• Main objective for EBM will be getting a small test MSE

Other models:

Overfit

• Get train close to zero

Underfit

• MSE bad
• Gap is small

Best results: small test MSE and small gap

[agallagher23]

Tasks completed

• Met with group to try to troubleshoot PiML errors, we emailed WF to get help twice
• Worked on background information section of final report

[everettbu]

Tasks completed

• Worked through PiML errors
• Obtained initial PiML optimization results to show Nengfeng
• Took notes on next steps for optimization

[reinabhatkuly]

Tasks completed:

• started work on final report, primarily focused on light GBM
• conducted research on the expectations from optimization

[xybljy0122]

4/3 Meeting with Wells Team:

Try to figure out which hypoerparameter more sensitive

Overfit model

• The gap between the test and train is large, train perform super well (extreme case: MSE = 0)

Underfit model (model too simple)

• The gap between the test and train is small, and both do not perform well (MSE around 0.02)

Normally, if the gap between train and test become small, model underfits

Project goal:

• Find the best EBM model (most optimized)
  • Doing the overfit and underfit help us to get the most optimized model
• Want small test MSE(try to find the best combination of parameter in EBM to get as close as possible to the test MSE in LGBM)
  • If cannot be accomplished, then we reach the conclusion: we sacrifice model performance for the explainability

Add max_bins in EBM

We don’t need to match up the parameters in the two models

max depth in LGBM The depth of decision tree, the deeper, the more complicated and flexible the model

Max_bins in EBM bin - the number of parameters to each variable max_bin=3, too small, not much flexibility max_bin too large, might overfit

Next step

• Compare model explainability, understand relationship between input and output(pdp..), figure out which variable most important
• 3 LGBM (benchmark, underfit, overfit), 3 EBM (benchmark, underfit, overfit)
  • Compare the two benchmarks model for comparison for blackbox vs. explainable)

more questions to look into: Overfit underfit, concerns

Ultimate goal: find the lowest test MSE and samllest gap Given the same test MSE, we want the test-gap as small as possible

[xybljy0122]

Task completed:

• Completed lightGBM research document