Week 3/20

[xybljy0122]

3/20 meeting w. Wells team

(Optimizing lambda1, 2, drop rate, max_depth) Pi-ml cannot adjust all the parameter all the same time Manually do it one by one

Overfitting We want to look at whether the training data has much higher MSE to testing data The difference between the testing data MSE

How to adjust the parameters: Look at lightGBM library and try to figure it out Rosh can help

Try look into low code for parameter

To-do : Put the results into the midterm presentation slides

In future: robustness tests on the models Look at the robust test graph slope The trend of model performance if we perturbation increase Perturbation - the noise in the data

Add new models to make better comparison A overfitted LGBM model (adjust the parameter to make it overfit) Adjust parameter: make the max depth to 16-20 Underfit: smaller max depth, smaller lambda, make the model simple, so it would underfit

[agallagher23]

A couple more notes from Wells team meeting:

How to optimize and what does it mean to under-fit and over-fit?

• Goal: make testing MSE smaller
• If training MSE is too good, it might be getting too complicated to scale
• Train will be better than testing, but don’t want it to be TOO much better

Hyper-parameter magnitudes will be different for each model

• Look through library ExplainableBoostingRegression to find hyper parameters
• Can look at low code EBM housing to see what the hyper parameters are

[agallagher23]

Agenda for tomorrow morning's meeting:

• We would love help walking through optimizing one of the hyper-parameters (lambda1, lambda2, drop rate, or max_depth)
  • Wells team told us to look at lightGBM library for help
• Need help understanding how to balance overfitting and underfitting

[everettbu]

Boosting Models notes:

• For most models, classification and regression versions
  • Classification is easier for explainability
• Basic idea: We can take a simple model, and make a more complicated model from it
• ML decides the decision boundary (between the classifications)
• Boosting machine lets you construct a more complicated thing automatically by adding simple models together
  • Regression + line1 + line2 = majority vote will find better classification
• Infamous for being hard to interpret
• LightGBM replaces function with a bunch of more complicated things
  • EBM is an approximation of LightGBM
  • Make LightGBM use linear regression as its base model
    • Use least squares regression instead of a decision tree
    • Also compare to best possible LightGBM

[reinabhatkuly]

Tasks Completed:

• Created and completed the attached document on EBM Models
• Emailed the same to the Wells Fargo Team for Feedback

EBM Model.pdf

[agallagher23]

3/21 Meeting with Mike

Model background

• Sum of decision trees
• Final prediction (F(x)) = weighted sum of a bunch of different functions
• Increasing complexity of individual tree increases total sum
• Changing nodes changes individual f(x)
• Hyperparemeters:
• Overfitting (too complex)
• Underfit (too simple)

Changing weighted term in equation (Wi):

• If Wi INC → complexity INC

• With a small Wi, the function does not exist in equation, increasing Wi increases its weight on the sum

• INC lambda → Wi DEC → complexity DEC

• Lamba1 ( = reg alpha)

• Likely to make to zero

• Lambda2 ( = reg lambda)

• Likely to make small, but not to zero

• Magnitudes

• 10^(-6) to 10^(6)

Can set lambda to zero in LightGBM to be comparable to EBM, then compare it to the optimized LightGBM

Max depth (max leafs): maximum depth of the decision tree of each function

• DT

• INC max depth → INC complexity

• LightGBM

• INC max depth → INC complexity

• EBM

• INC max depth → INC complexity

• Adjust linearly, (1 to 10)

Number of base models (interactions): number of decision trees involved in the sum function

• LightGBM

• INC base models → INC complexity

• EBM

• INC base models → INC complexity

• Change things on a logarithmic scale (multiple values by ten for each next step)

Next Steps

• Find optimal parameters
• Need to disable parameters that only exist in one of the models
• Ask Wells team if final report structure can be google collab notebook

• Intro, EBM section, black box section, overfitting, underfitting

[agallagher23]