search

American-Institutes-for-Research / WeMix

WeMix public repository
GNU General Public License v2.0
10 stars 2 forks source link

Failure with dev testthat #7

Closed hadley closed 1 year ago

hadley commented 1 year ago

With dev testthat I see:

── Failure ('test-1-main.R:1140'): Model with top level groups that have entirely 0 columns in Z ──
m3$varDF not equal to `varDF0`.
Component "vcov": Mean relative difference: 1.203325e-06

I suspect this is just a numerical accuracy issue but I wanted to let you know.

(PS. It would be easier to check your package if the tests didn't take >40+ minutes, and you organised them into multiple files)

pdbailey0 commented 1 year ago

@hadley, yes, that's a numerical difference, I'm not concerned about it. It's a bit hard to write acceptance and regression tests for the result of numerical optimization. Right now I test on a variety of systems and back things off, but I agree that I should be a bit more aggressive.

The CRAN checks execute in under 5 min. The full suite is more for devs to do regression testing. Perhaps you're politely pointing out I should .Rbuildignore those to spare the user.

hadley commented 1 year ago

TBH the length of the tests wouldn't be such a problem if they were divided up into pieces so it was easier to just run the subset that has problems.

hadley commented 1 year ago

Just to let you know, we're planning on submitting testthat to CRAN on Oct 7.

pdbailey0 commented 1 year ago

Thanks for the heads up, I'll plan on updating the CRAN version before then. Just pushed reworked test files @blue-webb made that should also make our lives simpler--thanks for the nudge.

I also now put in test-00-skip_on_cran.R Sys.setenv(NOT_CRAN='skip') which will make it default to the shorter CRAN checks.