Check recommended default values for tuning parameters

[jialul]

Question: ML algorithms perform poorly if we further tune the parameters in the training models using cross-validations. However, from our earlier conversations, how we train the training set wouldn't cause a problem for our later estimation of pape/papd/aupec? @MichaelLLi

Here is one example to tune the lambda in lasso with cv.lasso

cv.lasso <- glmnet::cv.glmnet(
        training_data_elements_lasso[["X_expand"]], 
        training_data_elements_lasso[["Y"]], 
        alpha = 1, 
        family = "binomial")

• No tuning in the training model

  

• Tuning in the training model

  

[kosukeimai]

I think that a part of the issue is that lasso is not trained using the right objective (just predictive accuracy). We need to use a causal target! But, that's what we are going to do once we have the package infrastructure ready. In the package documentation, let's make sure that you make a note about this. Also, do you have an option of X-learner, T-learner, and R-learner for lasso and BART? Unless the ML algorithms are designed to be causal (e.g., causal Forest and causal BART), we should allow for these different meta-learner types.

[jialul]

@kosukeimai I believe current infrastructure is using a causal target. For the testing dataset, we impute potential outcomes and estimate tau = Y(1) - Y(0), using the model we trained with the training set. Is this what you have in mind?

 ## predict 
  Y0t_total = predict(
    fit_train,
    total_data_elements_lasso[["X0t_expand"]],
    type = "response")

  Y1t_total = predict(
    fit_train,
    total_data_elements_lasso[["X1t_expand"]],
    type = "response")

  tau_total=Y1t_total-Y0t_total + runif(n_df,-1e-6,1e-6)

[jialul]

Currently we don't allow users to choose from X-learner, T-learner, and R-learner for lasso and BART but we can definitely incorporate that. I'll create a new issue for this.

[kosukeimai]

@jialul I don't think I understand what you mean above. How are you cross-validating? Causal Forest has the causal target. I don't think Lasso and BART have causal quantities as the target when you do cross-validation (after all, they are generic ML algorithms used for prediction).

[jialul]

@kosukeimai Sorry for the confusing. The cross-validation I was referring to was a function of the glmnet package that allow users to find the best lambda in the first stage and then use that lambda to fit the lasso model in the second stage. In the code below, cv.glmnet gives a value for lambda and we can use this lambda to fit the lasso model by using lambda = cv.lasso$lambda.min.

In Michael's original code, he chose lambda = 0.05 and I once asked whether we should choose the best lambda using function like cv.glmnet instead of choosing an arbitrary value 0.05. From our earlier conversation, either using the best lambda from cv.glmnet or 0.05 will not have too much impact on the estimated results. Therefore, I got confused when seeing the two plots shown above.

Just to clarify, this is different from the cross-validation we have to estimate PAPE, PAPD and AUPEC. We have created separate functions to do k-folds cross-validation and estimate the causal quantities.

## find the best lambda
cv.lasso <- glmnet::cv.glmnet(
        training_data_elements_lasso[["X_expand"]], 
        training_data_elements_lasso[["Y"]], 
        alpha = 1, 
        family = "binomial")

## fit
fit <- glmnet::glmnet(
        training_data_elements_lasso[["X_expand"]],
        training_data_elements_lasso[["Y"]],
        alpha = 1,
        family = "binomial",
        lambda = cv.lasso$lambda.min)

[kosukeimai]

My understanding is that the lambda value you get by minimizing the prediction error is too aggressive. Instead, people choose a value that corresponds to some larger value (there is a rule of thumb in the literature you should look up). That's probably should be the default. You should also create an option for users to do what they wish to do.

[jialul]

I see, I'll check what default values are recommended and add an option for users to change them.

[jialul]

Incorporated caret functions to tuning hyper parameters.

[jialul]

Allow users to specify tuning parameters in trainControl outside the estimate_itr