Sampling tests to strict

[rstub]

It can happen that the difference in the confidence intervals is larger than the 0.005 you are expecting:

  library(fwildclusterboot)
  set.seed(123123)
  dqrng::dqRNGkind("Xoroshiro128+")
  dqrng::dqset.seed(123123)

  data1 <<- fwildclusterboot:::create_data(
    N = 1000,
    N_G1 = 20,
    icc1 = 0.5,
    N_G2 = 20,
    icc2 = 0.2,
    numb_fe1 = 10,
    numb_fe2 = 10,
    #seed = 908361239,
    seed = 123123,
    weights = 1:N / N
  )

  lm_fit <- lm(proposition_vote ~ treatment + log_income,
               data = data1
  )

  boot1 <- boottest(lm_fit,
                    param = c("log_income"),
                    clustid = c("group_id2"),
                    B = 99999,
                    sampling = "dqrng"
  )
#> Warning: Please note that the seeding behavior for random number generation for
#> `boottest()` has changed with `fwildclusterboot` version 0.13.
#> 
#> It will no longer be possible to exactly reproduce results produced by versions
#> lower than 0.13.
#> 
#> If your prior results were produced under sufficiently many bootstrap
#> iterations, none of your conclusions will change.  For more details about this
#> change, please read the notes in
#> [news.md](https://cran.r-project.org/web/packages/fwildclusterboot/news/news.html).
#> This warning is displayed once per session.
#> Too guarantee reproducibility, don't forget to set a global random seed
#> **both** via `set.seed()` and `dqrng::dqset.seed()`.
#> This message is displayed once every 8 hours.

  boot2 <- boottest(lm_fit,
                    param = c("log_income"),
                    clustid = c("group_id2"),
                    B = 99999,
                    sampling = "standard"
  )
  all.equal(confint(boot1), confint(boot2))
#> [1] "Mean relative difference: 0.005275242"

^{Created on 2024-05-13 with reprex v2.1.0}

Running the the two boottest() repeatedly w/o fixing the seed I see differences up to 0.007. This probably explains the surprising results I got in https://github.com/daqana/dqrng/issues/80 when doing reverse dependency checks for the new version of dqrng.

Could you upload a version of fwildclusterboot that only changes the the expectation w.r.t. the confidence interval?

  expect_equal(confint(boot1), confint(boot2), tolerance = 0.0075)

Note: I am basing my analysis on v0.13, which I think corresponds to the current CRAN version.

[s3alfisc]

Yes, I'll try to do that at the end of today :)

[s3alfisc]

Hi @rstub, is this something I should change for a single test file, or for all test files?

[s3alfisc]

Only test_samplers.R =)

[s3alfisc]

Will submit to CRAN once the CI tests pass.

[s3alfisc]

Hi, unfortunately the tests haven't passed yet. I might only find time to debug this weekend. Would this cause you any larger inconveniences @rstub ? If yes, I would try to find some time before, but unfortunately I am quite busy in the next evenings.

[rstub]

So far I have not been given any time-line from CRAN, so it should be OK if you can do this next weekend.

[rstub]

I have now heard from CRAN: They either want to publish dqrng 0.4 now or ask me to resubmit once fixed packages are available. Publishing now would (potentially) break fwildclusterboot and you would then be given a 2 week(?) timeline to provide a fixed version. Would that be OK with you?

[s3alfisc]

Yes, please go ahead!

[rstub]

The first failures are coming in now that dqrng 0.4.0 is on CRAN: https://cran.r-project.org/web/checks/check_results_fwildclusterboot.html. I am confused why they are getting that failure repeatedly. For me it happened only sometimes.

[s3alfisc]

Thanks. I will tackle this this weekend :)

[eddelbuettel]

:disappointed: I guess this got you tossed off CRAN now. Let me know if I can help but it looks like you know what to do.

[s3alfisc]

Yes, but it's not really this, I just didn't do it on time / last minute and then of course an incorrect url bit me... I'll take the long weekend here in DE to submit a new version that has been long in the making =) thanks for checking it, definitely encouraging to put in the effort to get the package back on cran!