metrumresearchgroup / bbr

R interface for model and project management
https://metrumresearchgroup.github.io/bbr/
Other
23 stars 2 forks source link

model_summaries: use concurrency of `bbi nonmem summary` if available #527

Closed kyleam closed 2 years ago

kyleam commented 2 years ago

This series addresses gh-395 and builds on top of changes on the bbi side (https://github.com/metrumresearchgroup/bbi/pull/272) that are available under the v3.2.0-alpha.3 dev release.

Commit overview:

* check_bbi_version_constraint: improve an error message
* check_bbi_version_constraint: rename a variable
* check_bbi_version_constraint: return bbi exe path
* check_bbi_version_constraint: rename to assert_bbi_version
* add test_* variant of assert_bbi_version
* skip_if_old_bbi: reduce number of spaces used for indentation
* skip_if_old_bbi: take minimum requirement not last incompatible
* check_run_times: use test_bbi_version
* cov_cor.bbi_nonmem_model: drop bbi version check

  This first set reworks check_bbi_version_constraint(), renaming it
  to assert_bbi_version() and adding a new test_bbi_version() helper.
  test_bbi_version() is needed by the new model_summaries()
  implementation.

  I can split this off into its own series if desired.

* test-refs/1_summary_obj.R: run through RStudio's "Reformat Code"
* test-refs/1_summary_obj.R: regenerate with development bbi
* setup-workflow-ref.R: add "success" to list of names

  These are adjusting the tests for the new output format of
  `bbi nonmem summary --json`.

* bbi_exec: let caller bypass non-zero exit error signaling
* model_summaries.list: drop unnecessary re-assignment
* model_summaries.list: extract model_summary loop to its own function
* test-summary-log.R: set bbi_exe_path via file-wide
* model_summaries.list: pass multiple models to bbi if supported
* ci: test with bbi v3.2.0-alpha.3

  The fifth commit is the main commit of interest.  It reimplements
  model_summaries() using `bbi nonmem summary --json MOD1 MOD2 ...`.

Closes #395.

kyleam commented 2 years ago

Marked as draft until I figure out the CI failures (devtools::check() passed for me on metworx with alpha.3).

kyleam commented 2 years ago

script/boot-collect.R

Thanks for testing this with a larger case.

[weird result 1] I can follow the code path and see why this happens and I guess it's fine. But in my thinking (and the previous behavior) I'm pretty sure this was only TRUE when the model initially errored, but then didn't error with the .fail_flags.

Thanks. I missed that detail in my conversion. I think it'd be good for both code paths to be consistent (even if it's unlikely to matter in practice).

[weird result 2] the 005 model, where I overwrote the .lst with only the text "this will break", actually parses when no_ext_file = T is passed. I noticed this in summary_log(), but I can reproduce it with a single model_summary() call too. Not a big deal, but this should... probably fail.

Hmm, yeah. I'll take a closer look.

kyleam commented 2 years ago

[weird result 1]

This one should be resolved with 06562409 (model_summaries_concurrent: fix .fail_flags discrepancy, 2022-08-02).

While working on that, I realized that gh-529 is too aggressive because it leads to some JSON parsing failures with older versions of bbi. I've added back check_lst_file (220a931b), calling it only for bbi versions before 3.2.0. @seth127, let me know if you have any objections to that.

remaining todos:

kyleam commented 2 years ago

@seth127: there have been some substantial changes since you last took a look (c36155b6d), especially reintroducing check_lst_file, so I've re-requested your review.

$ git log --oneline --first-parent --no-decorate c36155b6d..
8a44798e Merge branch 'main' into summary-multi
ce785a40 model_summaries_concurrent: replace a subsetted map with modify_at
06562409 model_summaries_concurrent: fix .fail_flags discrepancy
bc74814d model_summaries: handle NULL .fail_flags
220a931b model-summary: restore check_lst_file() check for older bbi versions
2073187c test-refs/1_summary_obj.R: regenerate with build-model-summary-refs.R
kyleam commented 2 years ago

@seth127 Thanks for taking another look.

It's a bit unfortunate that we need that convoluted code in there to support the pre-3.2.0 behavior, but oh well

Agreed. I looked forward to whenever we bump the min bbi version and can strip out that and similar compatibility kludges.