search

benkeser / slapnap

Super LeArner Predictions using NAb Panels
https://benkeser.github.io/slapnap/
0 stars 5 forks source link

Only printing descriptive statistics if cvtune = FALSE and cvperf = FALSE #10

Closed bdwilliamson closed 4 years ago

bdwilliamson commented 4 years ago

Do we want to print anything else besides descriptive statistics when cvtune and cvperf are both FALSE (and no variable importance is requested)?

benkeser commented 4 years ago

I thought I'd included some information about the super learner in the report for this case? I.e., weights given to algorithms, cvRisk of algorithms, etc...? Is that not the case?

bdwilliamson commented 4 years ago

We're only fitting a single algorithm in this case rather than a super learner, and when cvperf = FALSE we shouldn't be estimating cvRisk, right? I've attached an example of what the report looks like with both cv* options set to false. I mostly wanted to make sure that we were comfortable with the behavior, since I didn't necessarily expect this (but would be fine if that's what we decide to go with). slapnap_report_nocv

benkeser commented 4 years ago

I think there's two behaviors to sort out here: when length(learner) == 1, cvperf=FALSE, cvtune=FALSE and when length(learner) > 1, cvperf=FALSE, cvtune=FALSE

You show the former case here. I think this behavior is OK. One can turn on variable importance and get back algorithm-specific information.

For the latter case, I would expect the behavior to be to fit a super learner with the specified library (limited to one type of each learner because cvtune=FALSE). In that case, the report should include information on the super learner model. Has this situation been evaluated in your tests?

bdwilliamson commented 4 years ago

Yes, and currently we're only fitting a single learner (like I mentioned in #11 ). In the example pasted below, I specified learners="lasso;rf", cvtune=FALSE, cvperf=FALSE; as in #11 , only lasso is run (since it was entered first). It sounds like changing that behavior should fix this issue too! slapnap_report_nocv_multilearner

benkeser commented 4 years ago

Great. Was that in your code or mine? I.e., do you know where to go looking for it? Or should I start digging?

bdwilliamson commented 4 years ago

The actual fitting is done in sl_one_outcome, which says that if cvperf=FALSE and cvtune=FALSE, then fit only a single learner by directly calling the wrapper function. I'll change it so that if cvperf=FALSE and cvtune=FALSE, we check if length(opts$learners) > 1; if true, we fit a super learner with only the "default" learners; if false, we fit only the wrapper for the "default" learner.

I also changed some of the description statements in the report so that it would be more consistent with the behavior, but I'll roll those back.

bdwilliamson commented 4 years ago

Fixed (see #11 )