pfmc-assessments / indexwc

Estimate indices of abundance for west coast fish species
2 stars 1 forks source link

[Feature]: save information from sanity() to a file #31

Closed kellijohnson-NOAA closed 5 months ago

kellijohnson-NOAA commented 6 months ago

Describe the problem your feature request is related to.

Often, when running multiple models or sending a finished model to someone else it would be good to have the output from sanity() saved to a text file that can easily be read back in.

Describe the solution you'd like

Information saved in a file that can be batch read back in for multiple models.

Describe alternatives you have considered

Require users to run the code on the saved model.

Additional context

No response

kellijohnson-NOAA commented 6 months ago

@chantelwetzel-noaa I am curious if we should save the information returned from sanity() which is just a list of logicals or if we should save the cli output that is printed to the screen to a file? Either is possible.

chantelwetzel-noaa commented 5 months ago

I am leaning towards saving the cli output as it is printed to the screen

kellijohnson-NOAA commented 5 months ago

Sounds good. This is what it looks like printed to the screen

✔ Non-linear minimizer suggests successful convergence
✔ Hessian matrix is positive definite
✔ No extreme or very small eigenvalues detected
✔ No gradients with respect to fixed effects are >= 0.001
✔ No fixed-effect standard errors are NA
✔ No standard errors look unreasonably large
✔ No sigma parameters are < 0.01
✔ No sigma parameters are > 100
✔ Range parameters don't look unreasonably large

and this is the best that I can do capturing it to a text string using cli::cli_fmt() but I can probably clean it up a little bit

[1] "\033[32m✔\033[39m Non-linear minimizer suggests successful convergence"   
[2] "\033[32m✔\033[39m Hessian matrix is positive definite"
[3] "\033[32m✔\033[39m No extreme or very small eigenvalues detected"
[4] "\033[32m✔\033[39m No gradients with respect to fixed effects are >= 0.001"
[5] "\033[32m✔\033[39m No fixed-effect standard errors are NA"
[6] "\033[32m✔\033[39m No standard errors look unreasonably large"
[7] "\033[32m✔\033[39m No sigma parameters are < 0.01"
[8] "\033[32m✔\033[39m No sigma parameters are > 100"
[9] "\033[32m✔\033[39m Range parameters don't look unreasonably large"

and finally, I am thinking about making a data frame with the following three columns: name, status, text, that would look like

            names logical    text
1      hessian_ok    TRUE    Non-linear minimizer suggests successful convergence
2 eigen_values_ok    TRUE    Hessian matrix is positive definite
3       nlminb_ok    TRUE    No extreme or very small eigenvalues detected
4        range_ok    TRUE    No gradients with respect to fixed effects are >= 0.001
5    gradients_ok    TRUE    No fixed-effect standard errors are NA
6 se_magnitude_ok    TRUE    No standard errors look unreasonably large
7        se_na_ok    TRUE    No sigma parameters are < 0.01
8       sigmas_ok    TRUE    No sigma parameters are > 100
9          all_ok    TRUE    Range parameters don't look unreasonably large

Does this all seem okay?

chantelwetzel-noaa commented 5 months ago

I think both the cli::cli_fmt() and the data frame look great. The information you have captured in the data frame and the formatting is better than I was expecting so if you only wanted to save the file, I think that would be fine.