Closed sinanpl closed 4 months ago
Hmh, that's odd. I don't know how to see which test(s) failed in the GitHub diagnostics (I only see which OSes it failed on?)...
Two other points though:
devtools::test()
on my Ubuntu 20.04 laptop.Do we know whether this is the test that's failing on the other OSes?
As you say @sinanpl, if the test is only off by a tiny amount we could change the tolerance (though it looks like we might need to use testthat::expect_snapshot_value()
for that, or just round the numbers ourselves). However if the difference is larger, or if it's the problem I mention above about the difference between e8e9de50a4 and 31f6a5843f15, then I'm not sure how to debug this further?
FWIW, here's the full bootstrap distribution I get when I run that test interactively (and, for reference, the summary, which is what fails the test):
The analysis I'm working on now is only using threefold results, so this possible error in the twofold results aren't a super high priority for me at the moment... But of course @sinanpl let me know if I can help check anything for you on this?
Whoops, wrong bootstraps. Here are (I think) the ones that are going wrong in that test:
I don't know how to see which test(s) failed in the GitHub diagnostics (I only see which OSes it failed on?)...
Ah, never mind, apparently I was blocking the JS that did this. So I can see the results now. I may have to revisit the rest of my comment in light of that then lol.
Okay, that's really strange -- it looks like that one line of unexplained_a
bootstraps changes considerably across all the OSes (including, it seems, mine and yours @sinanpl)... That makes me think that something's going wrong with that calculation beyond just the floating point tolerance. I'll have a look.
Okay, looking at it more closely I'm coming back @sinanpl to your point of view that this in fact a rounding/tolerance issue: I forgot that the estimates that all look very different are in fact extremely close to zero and so only differ after the first nine decimal places or so. (In fact, I get different results when I run the test interactively compared to when I run it with devtools::test()
.
So I'm glad that it doesn't look like an error in the calculation code. But like I say above, unfortunately it doesn't look like testthat::snapshot()
has a tolerance option, so we'll have to get a little cleverer to get this test to run again.
In one of these expectations, I got an error on my laptop - thinking the expectation was not up to date. However, this is an OS problem I think.
Here you can see that all tests pass on mac-os and fail on others.
This test is disabled this for now & needs an update. Solution would probably be rounding the coefs and do an approx comparison with the snapshot.
Relevant commit: ca3ced79