Open barrettk opened 5 months ago
seth127
commented
5 months ago @barrettk thanks for reporting. Am I understanding this correctly that this is only failing when trying to print the bbi_nonmem_summary object that's in the summary log tibble?
Maybe I missed it, but have you tried extracting one of those summary objects and calling param_estimates() directly on it? Does that fail too, or is the issue in the model_summary() print method?
barrettk
commented
5 months ago It unfortunately seems to be more than just printing. The underlying data itself seems to have NA values instead of the true values. But to answer your question, when I call model_summary(mod) %>% param_estimates() it reads in and displays the information as expected (even for a bootstrap model run). The two dropdown menus in the original issue description illustrate the differences.
It does seem odd we wouldn't have caught this sooner though. To the point where I would like someone to confirm they're seeing the same issue when reproduced (any summary_log() call should be sufficient).
barrettk
commented
3 months ago @seth127 FYI I figured out the issue:
It happens when no_ext_file = TRUE. There are a number of entry points where we could fix this (param_estimates, summary_log_impl, etc), though we could improve the bootstrap by not setting that to TRUE within summarize_bootstrap_run:
boot_sum_log <- summary_log(
boot_dir, .bbi_args = list(
no_grd_file = TRUE, no_ext_file = TRUE, no_shk_file = TRUE # proposing --> remove `no_ext_file = TRUE`
)That change could make sense to include within the current PR. I know we wanted to increase the probability model_summary calls succeed, but im wondering if this specific flag should/could be removed. cc @kylebaron
Alternative: we could make .bbi_args an argument, that defaults to list(no_grd_file = TRUE, no_shk_file = TRUE)
seth127
commented
3 months ago So, if I'm understanding this correctly, can this issue be boiled down to something like
model_summary() is called with .bbi_args = list(no_ext_file = TRUE) then several of the elements of the resulting object are either NA or empty.
summary_log() or model_summaries(), but this is the core issue.param_estimates() to fail, because it expects them to be there
fixed elementprint() method failing, but I'm pretty sure it's the param_estimates() call inside it that's actually failing. model_summary(mod, .bbi_args = list(no_ext_file = TRUE)) %>% param_estimates()If that's the correct, I think the first thing to look into is whether we actually need to those elements to come back for a successful param_estimates() call? If not, we should just handle that case more elegantly and not error. In other words: do we need and .ext file for param_estimates() I would guess no, but I'm not sure.
Some other things we probably want to look into at some point as well:
no_ext_file = TRUE in the relevant summary_log() calls? Is there a reason we should need that .ext file, or do we only need it as a by-product of avoiding this (fairly edge-case) error? Is there a reason to think it won't be there for bootstraps?model_summary(mod, .bbi_args = list(no_ext_file = TRUE))? This is probably a bbi question, and we may not need to look into it at this point, but I'm interested to know if there's a reason we can only get those from the .ext file. To be clear, I think this is the lowest priority thing to look into.
When looking at a summary log of a model directory, we see a table that looks like this:
However, it seems we cant actually print the
bbi_summaryobject due to an issue in tabulating several outputs in the nestedparam_estimates()call:The above error gets triggered during this section of the print call, which traces back to this section of the
param_estimates()call. As you can see, several parameters are not tabulated correctly:summary via `summary_log`
```r > .summary[[SUMMARY_PARAM_DATA]] [[1]] [[1]]$method [1] "METHOD NOT DETECTED" [[1]]$estimates [[1]]$estimates$theta [1] 2.3100 53.9000 454.0000 -0.0734 4.1100 [[1]]$estimates$omega [1] 0.095 0.000 0.192 [[1]]$estimates$sigma [1] 1 [[1]]$std_err [[1]]$std_err$theta [1] 0.0900 3.0600 32.7000 0.0542 1.3900 [[1]]$std_err$omega [1] 0.0174 NA 0.0240 [[1]]$std_err$sigma [1] NA [[1]]$random_effect_sd [[1]]$random_effect_sd$omega [1] NA NA NA [[1]]$random_effect_sd$sigma [1] NA [[1]]$random_effect_sdse [[1]]$random_effect_sdse$omega [1] NA NA NA [[1]]$random_effect_sdse$sigma [1] NA [[1]]$fixed named list() ```For reference, here is what we see when looking at the
model_summaryoutput ofMOD1:summary via `model_summary`
```r > .summary[[SUMMARY_PARAM_DATA]] [[1]] [[1]]$method [1] "TABLE NO. 1: First Order Conditional Estimation with Interaction: Goal Function=MINIMUM VALUE OF OBJECTIVE FUNCTION: Problem=1 Subproblem=0 Superproblem1=0 Iteration1=0 Superproblem2=0 Iteration2=0" [[1]]$estimates [[1]]$estimates$theta [1] 2.3171600 54.6151000 462.5140000 -0.0820082 4.1795900 [[1]]$estimates$omega [1] 0.0985328 0.0000000 0.1568250 [[1]]$estimates$sigma [1] 1 [[1]]$std_err [[1]]$std_err$theta [1] 0.0872080 3.3755100 30.2623000 0.0561272 1.3779600 [[1]]$std_err$omega [1] 2.03449e-02 1.00000e+10 2.72401e-02 [[1]]$std_err$sigma [1] 1e+10 [[1]]$random_effect_sd [[1]]$random_effect_sd$omega [1] 0.313899 0.000000 0.396011 [[1]]$random_effect_sd$sigma [1] 1 [[1]]$random_effect_sdse [[1]]$random_effect_sdse$omega [1] 3.24068e-02 1.00000e+10 3.43931e-02 [[1]]$random_effect_sdse$sigma [1] 1e+10 [[1]]$fixed [[1]]$fixed$theta [1] 0 0 0 0 0 [[1]]$fixed$omega [1] 0 1 0 [[1]]$fixed$sigma [1] 1 ```Here is the print of the model summary call on the same model that failed via the
summary_log()call:Im focusing on other
bbrfunctionality at the moment, and did not dig any deeper, though this should likely be addressed somewhat soon. Though this was discovered via the upcomingbootstrapfunctions, the issue is unrelated as far as I can tell.