maurolepore
commented
1 year ago @ropensci-review-bot assign @helske as editor
Open s3alfisc opened 2 years ago
maurolepore
commented
1 year ago @ropensci-review-bot assign @helske as editor
ropensci-review-bot
commented
1 year ago Assigned! @helske is now the editor
helske
commented
1 year ago @ropensci-review-bot check package
ropensci-review-bot
commented
1 year ago Thanks, about to send the query.
ropensci-review-bot
commented
1 year ago :rocket:
Editor check started
:wave:
ropensci-review-bot
commented
1 year ago git hash: 65c542e1
Important: All failing checks above must be addressed prior to proceeding
(Checks marked with :eyes: may be optionally addressed.)
Package License: GPL-3
The table below tallies all function calls to all packages ('ncalls'), both internal (r-base + recommended, along with the package itself), and external (imported and suggested packages). 'NA' values indicate packages to which no identified calls to R functions could be found. Note that these results are generated by an automated code-tagging system which may not be entirely accurate.
|type |package | ncalls|
|:----------|:----------------|------:|
|internal |base | 491|
|internal |fwildclusterboot | 114|
|internal |stats | 91|
|internal |utils | 9|
|internal |graphics | 3|
|internal |compiler | 1|
|imports |rlang | 72|
|imports |collapse | 16|
|imports |dreamerr | 10|
|imports |generics | 6|
|imports |MASS | 4|
|imports |dqrng | 2|
|imports |JuliaConnectoR | 2|
|imports |gtools | 1|
|imports |Matrix | 1|
|imports |summclust | 1|
|imports |Formula | NA|
|imports |Rcpp | NA|
|suggests |fabricatr | 6|
|suggests |fixest | NA|
|suggests |lfe | NA|
|suggests |ivreg | NA|
|suggests |clubSandwich | NA|
|suggests |lmtest | NA|
|suggests |data.table | NA|
|suggests |covr | NA|
|suggests |knitr | NA|
|suggests |rmarkdown | NA|
|suggests |broom | NA|
|suggests |modelsummary | NA|
|suggests |bench | NA|
|suggests |testthat | NA|
|suggests |tibble | NA|
|suggests |sandwich | NA|
|linking_to |Rcpp | NA|
|linking_to |RcppArmadillo | NA|
|linking_to |RcppEigen | NA|
Click below for tallies of functions used in each package. Locations of each call within this package may be generated locally by running 's <- pkgstats::pkgstats(
length (45), list (32), call (31), lapply (19), names (18), unique (16), c (15), t (15), as.vector (14), drop (14), vapply (13), paste0 (11), data.frame (10), mode (10), Reduce (10), rep (10), stop (10), vector (10), which (9), for (8), ifelse (8), match.call (7), matrix (7), nrow (7), as.matrix (6), ncol (6), sqrt (6), sum (6), is.null (5), seq_along (5), switch (5), as.numeric (4), crossprod (4), mean (4), replace (4), sample (4), solve (4), tcrossprod (4), abs (3), col (3), colnames (3), colSums (3), environment (3), min (3), q (3), as.factor (2), as.integer (2), attr (2), cbind (2), dim (2), getOption (2), if (2), loadedNamespaces (2), max (2), norm (2), paste (2), return (2), sign (2), suppressWarnings (2), append (1), as.character (1), class (1), do.call (1), eval (1), exp (1), grepl (1), gsub (1), integer (1), is.na (1), is.numeric (1), isTRUE (1), logical (1), message (1), nzchar (1), quote (1), Recall (1), row (1), sample.int (1), seq (1), sort (1), split (1), substr (1), Sys.getenv (1), Sys.which (1), unlist (1)
check_set_full_enumeration (7), get_ssc_julia (6), teststat (6), boot_algo_julia (5), crosstab (4), get_bootstrap_pvalue (4), get_cluster (4), cpp_get_nb_threads (3), eigenMapMatMult (3), get_ssc (3), get_weights (3), matrix_split (3), model_matrix (3), process_R (3), run_bootstrap (3), boot_algo3_crv1_denom (2), boottest (2), demean_fe (2), get_scores (2), preprocess2.felm (2), preprocess2.fixest (2), preprocess2.lm (2), transform_fe (2), vec2mat (2), boot_aggregate (1), boot_algo_fastnreliable (1), boot_algo_fastnwild (1), boot_algo3_crv3 (1), boottest.felm (1), boottest.fixest (1), boottest.ivreg (1), boottest.lm (1), check_boot_algo_fastnreliable (1), check_bootstrap_types (1), check_boottest_args_plus (1), check_engine_btype (1), check_mboottest_args_plus (1), check_params_in_model (1), check_r_lean (1), check_set_nthreads (1), confint.boottest (1), crosstab4 (1), find_proglang (1), get_seed (1), get_start_vals (1), getBoottest_engine (1), getBoottest_nthreads (1), glance.boottest (1), glance.mboottest (1), inform_seed (1), invert_p_val (1), manipulate_object (1), mboottest (1), mboottest.felm (1), p_val_null2 (1), p_val_null2_x (1), p_val_null2_x_sign_level (1), preprocess2.ivreg (1), to_integer (1), wildboottestHC (1)
weights (24), model.matrix (13), coef (9), formula (8), na.omit (6), frequency (5), model.frame (5), nobs (4), aggregate (3), model.response (3), na.exclude (3), reformulate (2), vcov (2), na.action (1), na.pass (1), rlnorm (1), update (1)
abort (56), inform (12), warn (4)
fsum (9), GRP (4), qF (3)
validate_dots (10)
data (7), combn (1), txtProgressBar (1)
draw_binary (2), draw_normal_icc (2), draw_ordered (1), fabricate (1)
tidy (4), glance (2)
ginv (4)
abline (3)
dqsample (2)
juliaEval (1), juliaImport (1)
cmpfun (1)
permutations (1)
Diagonal (1)
vcov_CR3J (1)
base
fwildclusterboot
stats
rlang
collapse
dreamerr
utils
fabricatr
generics
MASS
graphics
dqrng
JuliaConnectoR
compiler
gtools
Matrix
**NOTE:** Some imported packages appear to have no associated function calls; please ensure with author that these 'Imports' are listed appropriately.
summclust
This package features some noteworthy statistical properties which may need to be clarified by a handling editor prior to progressing.
The package has: - code in C++ (9% in 4 files) and R (91% in 32 files) - 2 authors - 5 vignettes - 1 internal data file - 12 imported packages - 31 exported functions (median 8 lines of code) - 154 non-exported functions in R (median 23 lines of code) - 17 R functions (median 18 lines of code) --- Statistical properties of package structure as distributional percentiles in relation to all current CRAN packages The following terminology is used: - `loc` = "Lines of Code" - `fn` = "function" - `exp`/`not_exp` = exported / not exported All parameters are explained as tooltips in the locally-rendered HTML version of this report generated by [the `checks_to_markdown()` function](https://docs.ropensci.org/pkgcheck/reference/checks_to_markdown.html) The final measure (`fn_call_network_size`) is the total number of calls between functions (in R), or more abstract relationships between code objects in other languages. Values are flagged as "noteworthy" when they lie in the upper or lower 5th percentile. |measure | value| percentile|noteworthy | |:------------------------|-----:|----------:|:----------| |files_R | 32| 90.5| | |files_src | 4| 87.0| | |files_vignettes | 5| 96.9| | |files_tests | 19| 96.0| | |loc_R | 4507| 94.6| | |loc_src | 446| 44.2| | |loc_vignettes | 701| 85.2| | |loc_tests | 6021| 98.6|TRUE | |num_vignettes | 5| 97.9|TRUE | |data_size_total | 14024| 72.7| | |data_size_median | 14024| 81.0| | |n_fns_r | 185| 88.6| | |n_fns_r_exported | 31| 79.2| | |n_fns_r_not_exported | 154| 90.4| | |n_fns_src | 17| 41.9| | |n_fns_per_file_r | 3| 55.2| | |n_fns_per_file_src | 4| 44.3| | |num_params_per_fn | 2| 11.9| | |loc_per_fn_r | 19| 57.3| | |loc_per_fn_r_exp | 8| 16.3| | |loc_per_fn_r_not_exp | 23| 68.7| | |loc_per_fn_src | 18| 62.9| | |rel_whitespace_R | 21| 95.3|TRUE | |rel_whitespace_src | 32| 57.7| | |rel_whitespace_vignettes | 34| 88.4| | |rel_whitespace_tests | 16| 98.2|TRUE | |doclines_per_fn_exp | 30| 32.9| | |doclines_per_fn_not_exp | 0| 0.0|TRUE | |fn_call_network_size | 152| 85.4| | ---
Click to see the interactive network visualisation of calls between objects in package
goodpractice and other checks#### 3a. Continuous Integration Badges [](https://github.com/s3alfisc/fwildclusterboot/actions) [](https://github.com/s3alfisc/fwildclusterboot/actions) **GitHub Workflow Results** | id|name |conclusion |sha | run_number|date | |----------:|:--------------------------|:----------|:------|----------:|:----------| | 4275569208|pages build and deployment |success |90bbf4 | 226|2023-02-26 | | 4275530992|pkgdown |success |65c542 | 196|2023-02-26 | | 4275530989|R-CMD-check |failure |65c542 | 630|2023-02-26 | | 4275530990|test-coverage |success |65c542 | 560|2023-02-26 | --- #### 3b. `goodpractice` results #### `R CMD check` with [rcmdcheck](https://r-lib.github.io/rcmdcheck/) R CMD check generated the following error: 1. checking tests ... Running ‘testthat.R’ ERROR Running the tests in ‘tests/testthat.R’ failed. Last 13 lines of output: 4. ├─fwildclusterboot::boottest(...) 5. └─fwildclusterboot:::boottest.ivreg(...) 6. └─fwildclusterboot:::boot_algo_julia(...) 7. └─JuliaConnectoR::juliaEval("using StableRNGs") 8. └─JuliaConnectoR::juliaCall("RConnector.mainevalcmd", expr) 9. ├─base::tryCatch(...) 10. │ └─base (local) tryCatchList(expr, classes, parentenv, handlers) 11. │ └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]]) 12. │ └─base (local) doTryCatch(return(expr), name, parentenv, handler) 13. └─JuliaConnectoR:::doCallJulia(funName, jlargs) 14. └─JuliaConnectoR:::handleCallbacksAndOutput() [ FAIL 13 | WARN 13 | SKIP 0 | PASS 420 ] Error: Test failures Execution halted R CMD check generated the following note: 1. checking installed package size ... NOTE installed size is 8.7Mb sub-directories of 1Mb or more: libs 7.7Mb R CMD check generated the following test_fail: 1. > library(testthat) > library(fwildclusterboot) > > test_check("fwildclusterboot") Loading required namespace: fixest Too guarantee reproducibility, don't forget to set a global random seed **both** via `set.seed()` and `dqrng::dqset.seed()`. This message is displayed once every 8 hours. The variable 'X2' has been removed because of collinearity (see $collin.var). Too guarantee reproducibility, don't forget to set a global random seed **both** via `set.seed()` and `dqrng::dqset.seed()`. This message is displayed once every 8 hours. Too guarantee reproducibility, don't forget to set a global random seed **both** via `set.seed()` and `dqrng::dqset.seed()`. This message is displayed once every 8 hours. Too guarantee reproducibility, don't forget to set a global random seed **both** via `set.seed()` and `dqrng::dqset.seed()`. This message is displayed once every 8 hours. Run the wild bootstrap: this might take some time...(but hopefully not too much time =) ). | | | 0% | |======================================================================| 100%Run the wild bootstrap: this might take some time...(but hopefully not too much time =) ). | | | 0% | |==== | 6% | |======== | 12% | |============ | 18% | |================ | 24% | |===================== | 29% | |========================= | 35% | |============================= | 41% | |================================= | 47% | |===================================== | 53% | |========================================= | 59% | |============================================= | 65% | |================================================= | 71% | |====================================================== | 76% | |========================================================== | 82% | |============================================================== | 88% | |================================================================== | 94% | |======================================================================| 100%Starting Julia ... Too guarantee reproducibility, don't forget to set a global random seed via `set.seed()`. This message is displayed once every 8 hours. Loading required namespace: data.table Note: The bootstrap usually performs best when the confidence level (here, 0.95 % times the number of replications plus 1 ( 1024 + 1 = 1025 ) is an integer. The variables 'log_income::17.2069212919256:Q1_immigration::3' and 'log_income::18.0802367820713:Q1_immigration::3' have been removed because of collinearity (see $collin.var). Note: The bootstrap usually performs best when the confidence level (here, 0.95 % times the number of replications plus 1 ( 17 + 1 = 18 ) is an integer. NOTE: 2 fixed-effect singletons were removed (2 observations). Loading required namespace: ivreg Variance contained negative values in the diagonal and was 'fixed' (a la Cameron, Gelbach & Miller 2011). Variance contained negative values in the diagonal and was 'fixed' (a la Cameron, Gelbach & Miller 2011). Variance contained negative values in the diagonal and was 'fixed' (a la Cameron, Gelbach & Miller 2011). Variance contained negative values in the diagonal and was 'fixed' (a la Cameron, Gelbach & Miller 2011). boottest() boottest() fnw1111* Confidence Intervals are currently only supported for the R engine with 'bootstrap_type = 'fnw11'. This message is displayed once every 8 hours. 133133Part 1: Large B Tests start ols/wls type: rademacher p-val: two-tailed null imposed: TRUE rademacher two-tailed TRUE [ FAIL 13 | WARN 13 | SKIP 0 | PASS 420 ] ══ Warnings ════════════════════════════════════════════════════════════════════ ── Warning ('test_fixest_sugar.R:19:3'): test fixest formula sugar ───────────── Please note that the seeding behavior for random number generation for `boottest()` has changed with `fwildclusterboot` version 0.13. It will no longer be possible to exactly reproduce results produced by versions lower than 0.13. If your prior results were produced under sufficiently many bootstrap iterations, none of your conclusions will change. For more details about this change, please read the notes in [news.md](https://cran.r-project.org/web/packages/fwildclusterboot/news/news.html). This warning is displayed once per session. Backtrace: 1. base::lapply(...) at test_fixest_sugar.R:19:2 2. fwildclusterboot (local) FUN(X[[i]], ...) 3. fwildclusterboot::boottest(x, B = 999, param = "treatment", clustid = "group_id1") at test_fixest_sugar.R:22:6 ── Warning ('test_tidy.R:25:5'): test tidiers with q = 1 ─────────────────────── There are only 1024 unique draws from the rademacher distribution for 10 bootstrap clusters. Therefore, B = 1024 with full enumeration. Consider using webb weights instead. Further, note that under full enumeration and with B = 1024 bootstrap draws, only 2^(#clusters - 1) = 512 distinct t-statistics and p-values can be computed. For a more thorough discussion, see Webb `Reworking wild bootstrap based inference for clustered errors` (2013). Backtrace: 1. fwildclusterboot::boottest(...) at test_tidy.R:25:4 2. fwildclusterboot:::boottest.lm(...) 3. fwildclusterboot:::check_set_full_enumeration(...) ── Warning ('test_tidy.R:31:5'): test tidiers with q = 1 ─────────────────────── There are only 1024 unique draws from the rademacher distribution for 10 bootstrap clusters. Therefore, B = 1024 with full enumeration. Consider using webb weights instead. Further, note that under full enumeration and with B = 1024 bootstrap draws, only 2^(#clusters - 1) = 512 distinct t-statistics and p-values can be computed. For a more thorough discussion, see Webb `Reworking wild bootstrap based inference for clustered errors` (2013). Backtrace: 1. fwildclusterboot::boottest(...) at test_tidy.R:31:4 2. fwildclusterboot:::boottest.lm(...) 3. fwildclusterboot:::check_set_full_enumeration(...) ── Warning ('test_tidy.R:38:5'): test tidiers with q = 1 ─────────────────────── There are only 1024 unique draws from the rademacher distribution for 10 bootstrap clusters. Therefore, B = 1024 with full enumeration. Consider using webb weights instead. Further, note that under full enumeration and with B = 1024 bootstrap draws, only 2^(#clusters - 1) = 512 distinct t-statistics and p-values can be computed. For a more thorough discussion, see Webb `Reworking wild bootstrap based inference for clustered errors` (2013). Backtrace: 1. fwildclusterboot::boottest(...) at test_tidy.R:38:4 2. fwildclusterboot:::boottest.lm(...) 3. fwildclusterboot:::check_set_full_enumeration(...) ── Warning ('test-error_warning.R:1142:9'): errors and warnings q = 1 ────────── Currently, boottest() calculates confidence intervals for one-sided hypotheses only for `engine = 'WildBootTests.jl'`. Backtrace: 1. testthat::expect_error(...) at test-error_warning.R:1142:8 8. fwildclusterboot:::boottest.fixest(...) 9. fwildclusterboot:::r_algo_checks(...) ── Warning ('test-error_warning.R:1154:9'): errors and warnings q = 1 ────────── Currently, boottest() calculates confidence intervals for one-sided hypotheses only for `engine = 'WildBootTests.jl'`. Backtrace: 1. testthat::expect_error(...) at test-error_warning.R:1154:8 8. fwildclusterboot:::boottest.felm(...) 9. fwildclusterboot:::r_algo_checks(...) ── Warning ('test-method_equivalence.R:141:5'): Do different, but equivalent ways to specify linear models lead to equivalent results? ── Negative eigenvalues set to zero in multiway clustered variance matrix. See felm(...,psdef=FALSE) Backtrace: 1. lfe::felm(...) at test-method_equivalence.R:141:4 2. lfe:::felm.mm(...) 3. lfe:::newols(...) ── Warning ('test-method_equivalence.R:166:5'): Do different, but equivalent ways to specify linear models lead to equivalent results? ── Negative eigenvalues set to zero in multiway clustered variance matrix. See felm(...,psdef=FALSE) Backtrace: 1. lfe::felm(...) at test-method_equivalence.R:166:4 2. lfe:::felm.mm(...) 3. lfe:::newols(...) ── Warning ('test-method_equivalence.R:173:5'): Do different, but equivalent ways to specify linear models lead to equivalent results? ── Negative eigenvalues set to zero in multiway clustered variance matrix. See felm(...,psdef=FALSE) Backtrace: 1. lfe::felm(...) at test-method_equivalence.R:173:4 2. lfe:::felm.mm(...) 3. lfe:::newols(...) ── Warning ('test-uncategorized.R:52:3'): uncategorized tests ────────────────── There are only 64 unique draws from the rademacher distribution for 6 bootstrap clusters. Therefore, B = 64 with full enumeration. Consider using webb weights instead. Further, note that under full enumeration and with B = 64 bootstrap draws, only 2^(#clusters - 1) = 32 distinct t-statistics and p-values can be computed. For a more thorough discussion, see Webb `Reworking wild bootstrap based inference for clustered errors` (2013). Backtrace: 1. fwildclusterboot::boottest(...) at test-uncategorized.R:52:2 2. fwildclusterboot:::boottest.fixest(...) 3. fwildclusterboot:::check_set_full_enumeration(...) ── Warning ('test-uncategorized.R:59:3'): uncategorized tests ────────────────── There are only 64 unique draws from the rademacher distribution for 6 bootstrap clusters. Therefore, B = 64 with full enumeration. Consider using webb weights instead. Further, note that under full enumeration and with B = 64 bootstrap draws, only 2^(#clusters - 1) = 32 distinct t-statistics and p-values can be computed. For a more thorough discussion, see Webb `Reworking wild bootstrap based inference for clustered errors` (2013). Backtrace: 1. fwildclusterboot::boottest(...) at test-uncategorized.R:59:2 2. fwildclusterboot:::boottest.fixest(...) 3. fwildclusterboot:::check_set_full_enumeration(...) ── Warning ('test-uncategorized.R:66:3'): uncategorized tests ────────────────── There are only 64 unique draws from the rademacher distribution for 6 bootstrap clusters. Therefore, B = 64 with full enumeration. Consider using webb weights instead. Further, note that under full enumeration and with B = 64 bootstrap draws, only 2^(#clusters - 1) = 32 distinct t-statistics and p-values can be computed. For a more thorough discussion, see Webb `Reworking wild bootstrap based inference for clustered errors` (2013). Backtrace: 1. fwildclusterboot::boottest(...) at test-uncategorized.R:66:2 2. fwildclusterboot:::boottest.fixest(...) 3. fwildclusterboot:::check_set_full_enumeration(...) ── Warning ('test-uncategorized.R:73:3'): uncategorized tests ────────────────── There are only 64 unique draws from the rademacher distribution for 6 bootstrap clusters. Therefore, B = 64 with full enumeration. Consider using webb weights instead. Further, note that under full enumeration and with B = 64 bootstrap draws, only 2^(#clusters - 1) = 32 distinct t-statistics and p-values can be computed. For a more thorough discussion, see Webb `Reworking wild bootstrap based inference for clustered errors` (2013). Backtrace: 1. fwildclusterboot::boottest(...) at test-uncategorized.R:73:2 2. fwildclusterboot:::boottest.fixest(...) 3. fwildclusterboot:::check_set_full_enumeration(...) ══ Failed tests ════════════════════════════════════════════════════════════════ ── Error ('test_tidy.R:38:5'): test tidiers with q = 1 ───────────────────────── Error: Evaluation in Julia failed. Original Julia error message: LoadError: ArgumentError: Package StableRNGs not found in current path. - Run `import Pkg; Pkg.add("StableRNGs")` to install the StableRNGs package. Stacktrace: [1] macro expansion @ ./loading.jl:1163 [inlined] [2] macro expansion @ ./lock.jl:223 [inlined] [3] require(into::Module, mod::Symbol) @ Base ./loading.jl:1144 [4] eval @ ./boot.jl:368 [inlined] [5] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) @ Base ./loading.jl:1428 [6] include_string (repeats 2 times) @ ./loading.jl:1438 [inlined] [7] mainevalcmd(str::String) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:150 [8] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ Base ./essentials.jl:729 [9] invokelatest(::Any, ::Any, ::Vararg{Any}) @ Base ./essentials.jl:726 [10] evaluate!(call::Main.RConnector.Call, communicator::Main.RConnector.CommunicatoR{Sockets.TCPSocket}) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:67 [11] evaluate_checked!(call::Main.RConnector.Call, communicator::Main.RConnector.CommunicatoR{Sockets.TCPSocket}) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:55 [12] serve_repl(sock::Sockets.TCPSocket) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/communicating.jl:185 [13] serve(port_hint::Int64; multiclient::Bool, portfile::String) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/communicating.jl:152 [14] top-level scope @ /usr/local/lib/R/site-library/JuliaConnectoR/Julia/main.jl:23 [15] include(mod::Module, _path::String) @ Base ./Base.jl:419 [16] exec_options(opts::Base.JLOptions) @ Base ./client.jl:303 [17] _start() @ Base ./client.jl:522 in expression starting at string:1 Backtrace: ▆ 1. ├─fwildclusterboot::boottest(...) at test_tidy.R:38:4 2. └─fwildclusterboot:::boottest.lm(...) 3. └─fwildclusterboot:::run_bootstrap(...) 4. └─fwildclusterboot:::boot_algo_julia(...) 5. └─JuliaConnectoR::juliaEval("using StableRNGs") 6. └─JuliaConnectoR::juliaCall("RConnector.mainevalcmd", expr) 7. ├─base::tryCatch(...) 8. │ └─base (local) tryCatchList(expr, classes, parentenv, handlers) 9. │ └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]]) 10. │ └─base (local) doTryCatch(return(expr), name, parentenv, handler) 11. └─JuliaConnectoR:::doCallJulia(funName, jlargs) 12. └─JuliaConnectoR:::handleCallbacksAndOutput() ── Error ('test_tidy.R:89:3'): test tidiers with q > 1 ───────────────────────── Error: Evaluation in Julia failed. Original Julia error message: LoadError: ArgumentError: Package StableRNGs not found in current path. - Run `import Pkg; Pkg.add("StableRNGs")` to install the StableRNGs package. Stacktrace: [1] macro expansion @ ./loading.jl:1163 [inlined] [2] macro expansion @ ./lock.jl:223 [inlined] [3] require(into::Module, mod::Symbol) @ Base ./loading.jl:1144 [4] eval @ ./boot.jl:368 [inlined] [5] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) @ Base ./loading.jl:1428 [6] include_string (repeats 2 times) @ ./loading.jl:1438 [inlined] [7] mainevalcmd(str::String) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:150 [8] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ Base ./essentials.jl:729 [9] invokelatest(::Any, ::Any, ::Vararg{Any}) @ Base ./essentials.jl:726 [10] evaluate!(call::Main.RConnector.Call, communicator::Main.RConnector.CommunicatoR{Sockets.TCPSocket}) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:67 [11] evaluate_checked!(call::Main.RConnector.Call, communicator::Main.RConnector.CommunicatoR{Sockets.TCPSocket}) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:55 [12] serve_repl(sock::Sockets.TCPSocket) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/communicating.jl:185 [13] serve(port_hint::Int64; multiclient::Bool, portfile::String) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/communicating.jl:152 [14] top-level scope @ /usr/local/lib/R/site-library/JuliaConnectoR/Julia/main.jl:23 [15] include(mod::Module, _path::String) @ Base ./Base.jl:419 [16] exec_options(opts::Base.JLOptions) @ Base ./client.jl:303 [17] _start() @ Base ./client.jl:522 in expression starting at string:1 Backtrace: ▆ 1. ├─fwildclusterboot::mboottest(lm_fit, R = R, B = 999, clustid = "group_id1") at test_tidy.R:89:2 2. └─fwildclusterboot:::mboottest.lm(lm_fit, R = R, B = 999, clustid = "group_id1") 3. └─fwildclusterboot:::boot_algo_julia(...) 4. └─JuliaConnectoR::juliaEval("using StableRNGs") 5. └─JuliaConnectoR::juliaCall("RConnector.mainevalcmd", expr) 6. ├─base::tryCatch(...) 7. │ └─base (local) tryCatchList(expr, classes, parentenv, handlers) 8. │ └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]]) 9. │ └─base (local) doTryCatch(return(expr), name, parentenv, handler) 10. └─JuliaConnectoR:::doCallJulia(funName, jlargs) 11. └─JuliaConnectoR:::handleCallbacksAndOutput() ── Error ('test-error_warning.R:237:9'): errors and warnings q = 1 ───────────── Error: Evaluation in Julia failed. Original Julia error message: LoadError: ArgumentError: Package StableRNGs not found in current path. - Run `import Pkg; Pkg.add("StableRNGs")` to install the StableRNGs package. Stacktrace: [1] macro expansion @ ./loading.jl:1163 [inlined] [2] macro expansion @ ./lock.jl:223 [inlined] [3] require(into::Module, mod::Symbol) @ Base ./loading.jl:1144 [4] eval @ ./boot.jl:368 [inlined] [5] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) @ Base ./loading.jl:1428 [6] include_string (repeats 2 times) @ ./loading.jl:1438 [inlined] [7] mainevalcmd(str::String) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:150 [8] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ Base ./essentials.jl:729 [9] invokelatest(::Any, ::Any, ::Vararg{Any}) @ Base ./essentials.jl:726 [10] evaluate!(call::Main.RConnector.Call, communicator::Main.RConnector.CommunicatoR{Sockets.TCPSocket}) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:67 [11] evaluate_checked!(call::Main.RConnector.Call, communicator::Main.RConnector.CommunicatoR{Sockets.TCPSocket}) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:55 [12] serve_repl(sock::Sockets.TCPSocket) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/communicating.jl:185 [13] serve(port_hint::Int64; multiclient::Bool, portfile::String) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/communicating.jl:152 [14] top-level scope @ /usr/local/lib/R/site-library/JuliaConnectoR/Julia/main.jl:23 [15] include(mod::Module, _path::String) @ Base ./Base.jl:419 [16] exec_options(opts::Base.JLOptions) @ Base ./client.jl:303 [17] _start() @ Base ./client.jl:522 in expression starting at string:1 Backtrace: ▆ 1. ├─base::suppressWarnings(...) at test-error_warning.R:237:8 2. │ └─base::withCallingHandlers(...) 3. ├─testthat::expect_warning(...) 4. │ └─testthat:::expect_condition_matching(...) 5. │ └─testthat:::quasi_capture(...) 6. │ ├─testthat (local) .capture(...) 7. │ │ └─base::withCallingHandlers(...) 8. │ └─rlang::eval_bare(quo_get_expr(.quo), quo_get_env(.quo)) 9. ├─fwildclusterboot::boottest(...) 10. └─fwildclusterboot:::boottest.lm(...) 11. └─fwildclusterboot:::run_bootstrap(...) 12. └─fwildclusterboot:::boot_algo_julia(...) 13. └─JuliaConnectoR::juliaEval("using StableRNGs") 14. └─JuliaConnectoR::juliaCall("RConnector.mainevalcmd", expr) 15. ├─base::tryCatch(...) 16. │ └─base (local) tryCatchList(expr, classes, parentenv, handlers) 17. │ └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]]) 18. │ └─base (local) doTryCatch(return(expr), name, parentenv, handler) 19. └─JuliaConnectoR:::doCallJulia(funName, jlargs) 20. └─JuliaConnectoR:::handleCallbacksAndOutput() ── Error ('test-error_warning.R:1443:5'): error warning IV/WRE and q > 1 ─────── Error: Evaluation in Julia failed. Original Julia error message: LoadError: ArgumentError: Package StableRNGs not found in current path. - Run `import Pkg; Pkg.add("StableRNGs")` to install the StableRNGs package. Stacktrace: [1] macro expansion @ ./loading.jl:1163 [inlined] [2] macro expansion @ ./lock.jl:223 [inlined] [3] require(into::Module, mod::Symbol) @ Base ./loading.jl:1144 [4] eval @ ./boot.jl:368 [inlined] [5] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) @ Base ./loading.jl:1428 [6] include_string (repeats 2 times) @ ./loading.jl:1438 [inlined] [7] mainevalcmd(str::String) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:150 [8] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ Base ./essentials.jl:729 [9] invokelatest(::Any, ::Any, ::Vararg{Any}) @ Base ./essentials.jl:726 [10] evaluate!(call::Main.RConnector.Call, communicator::Main.RConnector.CommunicatoR{Sockets.TCPSocket}) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:67 [11] evaluate_checked!(call::Main.RConnector.Call, communicator::Main.RConnector.CommunicatoR{Sockets.TCPSocket}) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:55 [12] serve_repl(sock::Sockets.TCPSocket) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/communicating.jl:185 [13] serve(port_hint::Int64; multiclient::Bool, portfile::String) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/communicating.jl:152 [14] top-level scope @ /usr/local/lib/R/site-library/JuliaConnectoR/Julia/main.jl:23 [15] include(mod::Module, _path::String) @ Base ./Base.jl:419 [16] exec_options(opts::Base.JLOptions) @ Base ./client.jl:303 [17] _start() @ Base ./client.jl:522 in expression starting at string:1 Backtrace: ▆ 1. ├─testthat::expect_warning(...) at test-error_warning.R:1443:4 2. │ └─testthat:::expect_condition_matching(...) 3. │ └─testthat:::quasi_capture(...) 4. │ ├─testthat (local) .capture(...) 5. │ │ └─base::withCallingHandlers(...) 6. │ └─rlang::eval_bare(quo_get_expr(.quo), quo_get_env(.quo)) 7. ├─base::suppressMessages(...) 8. │ └─base::withCallingHandlers(...) 9. ├─fwildclusterboot::boottest(...) 10. └─fwildclusterboot:::boottest.ivreg(...) 11. └─fwildclusterboot:::boot_algo_julia(...) 12. └─JuliaConnectoR::juliaEval("using StableRNGs") 13. └─JuliaConnectoR::juliaCall("RConnector.mainevalcmd", expr) 14. ├─base::tryCatch(...) 15. │ └─base (local) tryCatchList(expr, classes, parentenv, handlers) 16. │ └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]]) 17. │ └─base (local) doTryCatch(return(expr), name, parentenv, handler) 18. └─JuliaConnectoR:::doCallJulia(funName, jlargs) 19. └─JuliaConnectoR:::handleCallbacksAndOutput() ── Error ('test-full-enumeration.R:268:13'): test full enumeration cases: r and julia ── Error: Evaluation in Julia failed. Original Julia error message: LoadError: ArgumentError: Package StableRNGs not found in current path. - Run `import Pkg; Pkg.add("StableRNGs")` to install the StableRNGs package. Stacktrace: [1] macro expansion @ ./loading.jl:1163 [inlined] [2] macro expansion @ ./lock.jl:223 [inlined] [3] require(into::Module, mod::Symbol) @ Base ./loading.jl:1144 [4] eval @ ./boot.jl:368 [inlined] [5] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) @ Base ./loading.jl:1428 [6] include_string (repeats 2 times) @ ./loading.jl:1438 [inlined] [7] mainevalcmd(str::String) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:150 [8] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ Base ./essentials.jl:729 [9] invokelatest(::Any, ::Any, ::Vararg{Any}) @ Base ./essentials.jl:726 [10] evaluate!(call::Main.RConnector.Call, communicator::Main.RConnector.CommunicatoR{Sockets.TCPSocket}) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:67 [11] evaluate_checked!(call::Main.RConnector.Call, communicator::Main.RConnector.CommunicatoR{Sockets.TCPSocket}) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:55 [12] serve_repl(sock::Sockets.TCPSocket) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/communicating.jl:185 [13] serve(port_hint::Int64; multiclient::Bool, portfile::String) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/communicating.jl:152 [14] top-level scope @ /usr/local/lib/R/site-library/JuliaConnectoR/Julia/main.jl:23 [15] include(mod::Module, _path::String) @ Base ./Base.jl:419 [16] exec_options(opts::Base.JLOptions) @ Base ./client.jl:303 [17] _start() @ Base ./client.jl:522 in expression starting at string:1 Backtrace: ▆ 1. ├─base::suppressWarnings(...) at test-full-enumeration.R:268:12 2. │ └─base::withCallingHandlers(...) 3. ├─fwildclusterboot::boottest(...) 4. └─fwildclusterboot:::boottest.lm(...) 5. └─fwildclusterboot:::run_bootstrap(...) 6. └─fwildclusterboot:::boot_algo_julia(...) 7. └─JuliaConnectoR::juliaEval("using StableRNGs") 8. └─JuliaConnectoR::juliaCall("RConnector.mainevalcmd", expr) 9. ├─base::tryCatch(...) 10. │ └─base (local) tryCatchList(expr, classes, parentenv, handlers) 11. │ └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]]) 12. │ └─base (local) doTryCatch(return(expr), name, parentenv, handler) 13. └─JuliaConnectoR:::doCallJulia(funName, jlargs) 14. └─JuliaConnectoR:::handleCallbacksAndOutput() ── Error ('test-global_vars.R:21:5'): global engine ──────────────────────────── Error: Evaluation in Julia failed. Original Julia error message: LoadError: ArgumentError: Package StableRNGs not found in current path. - Run `import Pkg; Pkg.add("StableRNGs")` to install the StableRNGs package. Stacktrace: [1] macro expansion @ ./loading.jl:1163 [inlined] [2] macro expansion @ ./lock.jl:223 [inlined] [3] require(into::Module, mod::Symbol) @ Base ./loading.jl:1144 [4] eval @ ./boot.jl:368 [inlined] [5] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) @ Base ./loading.jl:1428 [6] include_string (repeats 2 times) @ ./loading.jl:1438 [inlined] [7] mainevalcmd(str::String) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:150 [8] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ Base ./essentials.jl:729 [9] invokelatest(::Any, ::Any, ::Vararg{Any}) @ Base ./essentials.jl:726 [10] evaluate!(call::Main.RConnector.Call, communicator::Main.RConnector.CommunicatoR{Sockets.TCPSocket}) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:67 [11] evaluate_checked!(call::Main.RConnector.Call, communicator::Main.RConnector.CommunicatoR{Sockets.TCPSocket}) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:55 [12] serve_repl(sock::Sockets.TCPSocket) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/communicating.jl:185 [13] serve(port_hint::Int64; multiclient::Bool, portfile::String) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/communicating.jl:152 [14] top-level scope @ /usr/local/lib/R/site-library/JuliaConnectoR/Julia/main.jl:23 [15] include(mod::Module, _path::String) @ Base ./Base.jl:419 [16] exec_options(opts::Base.JLOptions) @ Base ./client.jl:303 [17] _start() @ Base ./client.jl:522 in expression starting at string:1 Backtrace: ▆ 1. ├─base::suppressWarnings(...) at test-global_vars.R:21:4 2. │ └─base::withCallingHandlers(...) 3. ├─fwildclusterboot::boottest(...) 4. └─fwildclusterboot:::boottest.lm(...) 5. └─fwildclusterboot:::run_bootstrap(...) 6. └─fwildclusterboot:::boot_algo_julia(...) 7. └─JuliaConnectoR::juliaEval("using StableRNGs") 8. └─JuliaConnectoR::juliaCall("RConnector.mainevalcmd", expr) 9. ├─base::tryCatch(...) 10. │ └─base (local) tryCatchList(expr, classes, parentenv, handlers) 11. │ └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]]) 12. │ └─base (local) doTryCatch(return(expr), name, parentenv, handler) 13. └─JuliaConnectoR:::doCallJulia(funName, jlargs) 14. └─JuliaConnectoR:::handleCallbacksAndOutput() ── Error ('test-method_equivalence.R:1422:5'): Do different, but equivalent ways to specify linear models lead to equivalent results? ── Error: Evaluation in Julia failed. Original Julia error message: LoadError: ArgumentError: Package StableRNGs not found in current path. - Run `import Pkg; Pkg.add("StableRNGs")` to install the StableRNGs package. Stacktrace: [1] macro expansion @ ./loading.jl:1163 [inlined] [2] macro expansion @ ./lock.jl:223 [inlined] [3] require(into::Module, mod::Symbol) @ Base ./loading.jl:1144 [4] eval @ ./boot.jl:368 [inlined] [5] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) @ Base ./loading.jl:1428 [6] include_string (repeats 2 times) @ ./loading.jl:1438 [inlined] [7] mainevalcmd(str::String) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:150 [8] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ Base ./essentials.jl:729 [9] invokelatest(::Any, ::Any, ::Vararg{Any}) @ Base ./essentials.jl:726 [10] evaluate!(call::Main.RConnector.Call, communicator::Main.RConnector.CommunicatoR{Sockets.TCPSocket}) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:67 [11] evaluate_checked!(call::Main.RConnector.Call, communicator::Main.RConnector.CommunicatoR{Sockets.TCPSocket}) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:55 [12] serve_repl(sock::Sockets.TCPSocket) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/communicating.jl:185 [13] serve(port_hint::Int64; multiclient::Bool, portfile::String) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/communicating.jl:152 [14] top-level scope @ /usr/local/lib/R/site-library/JuliaConnectoR/Julia/main.jl:23 [15] include(mod::Module, _path::String) @ Base ./Base.jl:419 [16] exec_options(opts::Base.JLOptions) @ Base ./client.jl:303 [17] _start() @ Base ./client.jl:522 in expression starting at string:1 Backtrace: ▆ 1. └─fwildclusterboot (local) create_models(clustid = "group_id1") at test-method_equivalence.R:1422:4 2. ├─base::assign(...) at test-method_equivalence.R:196:8 3. ├─base::suppressWarnings(...) 4. │ └─base::withCallingHandlers(...) 5. ├─fwildclusterboot::boottest(...) 6. └─fwildclusterboot:::boottest.lm(...) 7. └─fwildclusterboot:::run_bootstrap(...) 8. └─fwildclusterboot:::boot_algo_julia(...) 9. └─JuliaConnectoR::juliaEval("using StableRNGs") 10. └─JuliaConnectoR::juliaCall("RConnector.mainevalcmd", expr) 11. ├─base::tryCatch(...) 12. │ └─base (local) tryCatchList(expr, classes, parentenv, handlers) 13. │ └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]]) 14. │ └─base (local) doTryCatch(return(expr), name, parentenv, handler) 15. └─JuliaConnectoR:::doCallJulia(funName, jlargs) 16. └─JuliaConnectoR:::handleCallbacksAndOutput() ── Error ('test-new-bootstrap-variants.R:308:5'): variants 31 R vs Julia ─────── Error: Evaluation in Julia failed. Original Julia error message: LoadError: ArgumentError: Package StableRNGs not found in current path. - Run `import Pkg; Pkg.add("StableRNGs")` to install the StableRNGs package. Stacktrace: [1] macro expansion @ ./loading.jl:1163 [inlined] [2] macro expansion @ ./lock.jl:223 [inlined] [3] require(into::Module, mod::Symbol) @ Base ./loading.jl:1144 [4] eval @ ./boot.jl:368 [inlined] [5] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) @ Base ./loading.jl:1428 [6] include_string (repeats 2 times) @ ./loading.jl:1438 [inlined] [7] mainevalcmd(str::String) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:150 [8] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ Base ./essentials.jl:729 [9] invokelatest(::Any, ::Any, ::Vararg{Any}) @ Base ./essentials.jl:726 [10] evaluate!(call::Main.RConnector.Call, communicator::Main.RConnector.CommunicatoR{Sockets.TCPSocket}) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:67 [11] evaluate_checked!(call::Main.RConnector.Call, communicator::Main.RConnector.CommunicatoR{Sockets.TCPSocket}) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:55 [12] serve_repl(sock::Sockets.TCPSocket) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/communicating.jl:185 [13] serve(port_hint::Int64; multiclient::Bool, portfile::String) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/communicating.jl:152 [14] top-level scope @ /usr/local/lib/R/site-library/JuliaConnectoR/Julia/main.jl:23 [15] include(mod::Module, _path::String) @ Base ./Base.jl:419 [16] exec_options(opts::Base.JLOptions) @ Base ./client.jl:303 [17] _start() @ Base ./client.jl:522 in expression starting at string:1 Backtrace: ▆ 1. ├─base::suppressWarnings(...) at test-new-bootstrap-variants.R:308:4 2. │ └─base::withCallingHandlers(...) 3. ├─fwildclusterboot::boottest(...) 4. └─fwildclusterboot:::boottest.lm(...) 5. └─fwildclusterboot:::run_bootstrap(...) 6. └─fwildclusterboot:::boot_algo_julia(...) 7. └─JuliaConnectoR::juliaEval("using StableRNGs") 8. └─JuliaConnectoR::juliaCall("RConnector.mainevalcmd", expr) 9. ├─base::tryCatch(...) 10. │ └─base (local) tryCatchList(expr, classes, parentenv, handlers) 11. │ └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]]) 12. │ └─base (local) doTryCatch(return(expr), name, parentenv, handler) 13. └─JuliaConnectoR:::doCallJulia(funName, jlargs) 14. └─JuliaConnectoR:::handleCallbacksAndOutput() ── Error ('test-r-vs-julia.R:89:15'): test r against Julia I: stochastic tests ── Error: Evaluation in Julia failed. Original Julia error message: LoadError: ArgumentError: Package StableRNGs not found in current path. - Run `import Pkg; Pkg.add("StableRNGs")` to install the StableRNGs package. Stacktrace: [1] macro expansion @ ./loading.jl:1163 [inlined] [2] macro expansion @ ./lock.jl:223 [inlined] [3] require(into::Module, mod::Symbol) @ Base ./loading.jl:1144 [4] eval @ ./boot.jl:368 [inlined] [5] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) @ Base ./loading.jl:1428 [6] include_string (repeats 2 times) @ ./loading.jl:1438 [inlined] [7] mainevalcmd(str::String) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:150 [8] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ Base ./essentials.jl:729 [9] invokelatest(::Any, ::Any, ::Vararg{Any}) @ Base ./essentials.jl:726 [10] evaluate!(call::Main.RConnector.Call, communicator::Main.RConnector.CommunicatoR{Sockets.TCPSocket}) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:67 [11] evaluate_checked!(call::Main.RConnector.Call, communicator::Main.RConnector.CommunicatoR{Sockets.TCPSocket}) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:55 [12] serve_repl(sock::Sockets.TCPSocket) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/communicating.jl:185 [13] serve(port_hint::Int64; multiclient::Bool, portfile::String) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/communicating.jl:152 [14] top-level scope @ /usr/local/lib/R/site-library/JuliaConnectoR/Julia/main.jl:23 [15] include(mod::Module, _path::String) @ Base ./Base.jl:419 [16] exec_options(opts::Base.JLOptions) @ Base ./client.jl:303 [17] _start() @ Base ./client.jl:522 in expression starting at string:1 Backtrace: ▆ 1. ├─base::suppressWarnings(...) at test-r-vs-julia.R:89:14 2. │ └─base::withCallingHandlers(...) 3. ├─fwildclusterboot::boottest(...) 4. └─fwildclusterboot:::boottest.lm(...) 5. └─fwildclusterboot:::run_bootstrap(...) 6. └─fwildclusterboot:::boot_algo_julia(...) 7. └─JuliaConnectoR::juliaEval("using StableRNGs") 8. └─JuliaConnectoR::juliaCall("RConnector.mainevalcmd", expr) 9. ├─base::tryCatch(...) 10. │ └─base (local) tryCatchList(expr, classes, parentenv, handlers) 11. │ └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]]) 12. │ └─base (local) doTryCatch(return(expr), name, parentenv, handler) 13. └─JuliaConnectoR:::doCallJulia(funName, jlargs) 14. └─JuliaConnectoR:::handleCallbacksAndOutput() ── Error ('test-seed.R:38:7'): seed works for OLS ────────────────────────────── Error: Evaluation in Julia failed. Original Julia error message: LoadError: ArgumentError: Package StableRNGs not found in current path. - Run `import Pkg; Pkg.add("StableRNGs")` to install the StableRNGs package. Stacktrace: [1] macro expansion @ ./loading.jl:1163 [inlined] [2] macro expansion @ ./lock.jl:223 [inlined] [3] require(into::Module, mod::Symbol) @ Base ./loading.jl:1144 [4] eval @ ./boot.jl:368 [inlined] [5] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) @ Base ./loading.jl:1428 [6] include_string (repeats 2 times) @ ./loading.jl:1438 [inlined] [7] mainevalcmd(str::String) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:150 [8] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ Base ./essentials.jl:729 [9] invokelatest(::Any, ::Any, ::Vararg{Any}) @ Base ./essentials.jl:726 [10] evaluate!(call::Main.RConnector.Call, communicator::Main.RConnector.CommunicatoR{Sockets.TCPSocket}) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:67 [11] evaluate_checked!(call::Main.RConnector.Call, communicator::Main.RConnector.CommunicatoR{Sockets.TCPSocket}) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:55 [12] serve_repl(sock::Sockets.TCPSocket) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/communicating.jl:185 [13] serve(port_hint::Int64; multiclient::Bool, portfile::String) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/communicating.jl:152 [14] top-level scope @ /usr/local/lib/R/site-library/JuliaConnectoR/Julia/main.jl:23 [15] include(mod::Module, _path::String) @ Base ./Base.jl:419 [16] exec_options(opts::Base.JLOptions) @ Base ./client.jl:303 [17] _start() @ Base ./client.jl:522 in expression starting at string:1 Backtrace: ▆ 1. ├─base::suppressMessages(...) at test-seed.R:38:6 2. │ └─base::withCallingHandlers(...) 3. ├─fwildclusterboot::boottest(...) 4. └─fwildclusterboot:::boottest.lm(...) 5. └─fwildclusterboot:::run_bootstrap(...) 6. └─fwildclusterboot:::boot_algo_julia(...) 7. └─JuliaConnectoR::juliaEval("using StableRNGs") 8. └─JuliaConnectoR::juliaCall("RConnector.mainevalcmd", expr) 9. ├─base::tryCatch(...) 10. │ └─base (local) tryCatchList(expr, classes, parentenv, handlers) 11. │ └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]]) 12. │ └─base (local) doTryCatch(return(expr), name, parentenv, handler) 13. └─JuliaConnectoR:::doCallJulia(funName, jlargs) 14. └─JuliaConnectoR:::handleCallbacksAndOutput() ── Error ('test-tstat_equivalence.R:394:19'): t-stat equivalence OLS - WildBootTests ── Error: Evaluation in Julia failed. Original Julia error message: LoadError: ArgumentError: Package StableRNGs not found in current path. - Run `import Pkg; Pkg.add("StableRNGs")` to install the StableRNGs package. Stacktrace: [1] macro expansion @ ./loading.jl:1163 [inlined] [2] macro expansion @ ./lock.jl:223 [inlined] [3] require(into::Module, mod::Symbol) @ Base ./loading.jl:1144 [4] eval @ ./boot.jl:368 [inlined] [5] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) @ Base ./loading.jl:1428 [6] include_string (repeats 2 times) @ ./loading.jl:1438 [inlined] [7] mainevalcmd(str::String) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:150 [8] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ Base ./essentials.jl:729 [9] invokelatest(::Any, ::Any, ::Vararg{Any}) @ Base ./essentials.jl:726 [10] evaluate!(call::Main.RConnector.Call, communicator::Main.RConnector.CommunicatoR{Sockets.TCPSocket}) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:67 [11] evaluate_checked!(call::Main.RConnector.Call, communicator::Main.RConnector.CommunicatoR{Sockets.TCPSocket}) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:55 [12] serve_repl(sock::Sockets.TCPSocket) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/communicating.jl:185 [13] serve(port_hint::Int64; multiclient::Bool, portfile::String) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/communicating.jl:152 [14] top-level scope @ /usr/local/lib/R/site-library/JuliaConnectoR/Julia/main.jl:23 [15] include(mod::Module, _path::String) @ Base ./Base.jl:419 [16] exec_options(opts::Base.JLOptions) @ Base ./client.jl:303 [17] _start() @ Base ./client.jl:522 in expression starting at string:1 Backtrace: ▆ 1. ├─base::suppressWarnings(...) at test-tstat_equivalence.R:394:18 2. │ └─base::withCallingHandlers(...) 3. ├─fwildclusterboot::boottest(...) 4. └─fwildclusterboot:::boottest.lm(...) 5. └─fwildclusterboot:::run_bootstrap(...) 6. └─fwildclusterboot:::boot_algo_julia(...) 7. └─JuliaConnectoR::juliaEval("using StableRNGs") 8. └─JuliaConnectoR::juliaCall("RConnector.mainevalcmd", expr) 9. ├─base::tryCatch(...) 10. │ └─base (local) tryCatchList(expr, classes, parentenv, handlers) 11. │ └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]]) 12. │ └─base (local) doTryCatch(return(expr), name, parentenv, handler) 13. └─JuliaConnectoR:::doCallJulia(funName, jlargs) 14. └─JuliaConnectoR:::handleCallbacksAndOutput() ── Error ('test-tstat_equivalence.R:834:5'): t-stat equivalence OLS q > 1 ────── Error: Evaluation in Julia failed. Original Julia error message: LoadError: ArgumentError: Package StableRNGs not found in current path. - Run `import Pkg; Pkg.add("StableRNGs")` to install the StableRNGs package. Stacktrace: [1] macro expansion @ ./loading.jl:1163 [inlined] [2] macro expansion @ ./lock.jl:223 [inlined] [3] require(into::Module, mod::Symbol) @ Base ./loading.jl:1144 [4] eval @ ./boot.jl:368 [inlined] [5] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) @ Base ./loading.jl:1428 [6] include_string (repeats 2 times) @ ./loading.jl:1438 [inlined] [7] mainevalcmd(str::String) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:150 [8] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ Base ./essentials.jl:729 [9] invokelatest(::Any, ::Any, ::Vararg{Any}) @ Base ./essentials.jl:726 [10] evaluate!(call::Main.RConnector.Call, communicator::Main.RConnector.CommunicatoR{Sockets.TCPSocket}) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:67 [11] evaluate_checked!(call::Main.RConnector.Call, communicator::Main.RConnector.CommunicatoR{Sockets.TCPSocket}) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:55 [12] serve_repl(sock::Sockets.TCPSocket) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/communicating.jl:185 [13] serve(port_hint::Int64; multiclient::Bool, portfile::String) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/communicating.jl:152 [14] top-level scope @ /usr/local/lib/R/site-library/JuliaConnectoR/Julia/main.jl:23 [15] include(mod::Module, _path::String) @ Base ./Base.jl:419 [16] exec_options(opts::Base.JLOptions) @ Base ./client.jl:303 [17] _start() @ Base ./client.jl:522 in expression starting at string:1 Backtrace: ▆ 1. └─fwildclusterboot (local) wald_test(run_this_test = TRUE) at test-tstat_equivalence.R:834:4 2. ├─base::suppressWarnings(...) at test-tstat_equivalence.R:626:8 3. │ └─base::withCallingHandlers(...) 4. ├─fwildclusterboot::mboottest(...) 5. └─fwildclusterboot:::mboottest.lm(...) 6. └─fwildclusterboot:::boot_algo_julia(...) 7. └─JuliaConnectoR::juliaEval("using StableRNGs") 8. └─JuliaConnectoR::juliaCall("RConnector.mainevalcmd", expr) 9. ├─base::tryCatch(...) 10. │ └─base (local) tryCatchList(expr, classes, parentenv, handlers) 11. │ └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]]) 12. │ └─base (local) doTryCatch(return(expr), name, parentenv, handler) 13. └─JuliaConnectoR:::doCallJulia(funName, jlargs) 14. └─JuliaConnectoR:::handleCallbacksAndOutput() ── Error ('test-tstat_equivalence.R:915:5'): t-stat equivalence IV ───────────── Error: Evaluation in Julia failed. Original Julia error message: LoadError: ArgumentError: Package StableRNGs not found in current path. - Run `import Pkg; Pkg.add("StableRNGs")` to install the StableRNGs package. Stacktrace: [1] macro expansion @ ./loading.jl:1163 [inlined] [2] macro expansion @ ./lock.jl:223 [inlined] [3] require(into::Module, mod::Symbol) @ Base ./loading.jl:1144 [4] eval @ ./boot.jl:368 [inlined] [5] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) @ Base ./loading.jl:1428 [6] include_string (repeats 2 times) @ ./loading.jl:1438 [inlined] [7] mainevalcmd(str::String) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:150 [8] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ Base ./essentials.jl:729 [9] invokelatest(::Any, ::Any, ::Vararg{Any}) @ Base ./essentials.jl:726 [10] evaluate!(call::Main.RConnector.Call, communicator::Main.RConnector.CommunicatoR{Sockets.TCPSocket}) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:67 [11] evaluate_checked!(call::Main.RConnector.Call, communicator::Main.RConnector.CommunicatoR{Sockets.TCPSocket}) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/evaluating.jl:55 [12] serve_repl(sock::Sockets.TCPSocket) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/communicating.jl:185 [13] serve(port_hint::Int64; multiclient::Bool, portfile::String) @ Main.RConnector /usr/local/lib/R/site-library/JuliaConnectoR/Julia/communicating.jl:152 [14] top-level scope @ /usr/local/lib/R/site-library/JuliaConnectoR/Julia/main.jl:23 [15] include(mod::Module, _path::String) @ Base ./Base.jl:419 [16] exec_options(opts::Base.JLOptions) @ Base ./client.jl:303 [17] _start() @ Base ./client.jl:522 in expression starting at string:1 Backtrace: ▆ 1. └─fwildclusterboot (local) iv_test(run_this_test = TRUE) at test-tstat_equivalence.R:915:4 2. ├─base::suppressWarnings(...) at test-tstat_equivalence.R:880:8 3. │ └─base::withCallingHandlers(...) 4. ├─fwildclusterboot::boottest(...) 5. └─fwildclusterboot:::boottest.ivreg(...) 6. └─fwildclusterboot:::boot_algo_julia(...) 7. └─JuliaConnectoR::juliaEval("using StableRNGs") 8. └─JuliaConnectoR::juliaCall("RConnector.mainevalcmd", expr) 9. ├─base::tryCatch(...) 10. │ └─base (local) tryCatchList(expr, classes, parentenv, handlers) 11. │ └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]]) 12. │ └─base (local) doTryCatch(return(expr), name, parentenv, handler) 13. └─JuliaConnectoR:::doCallJulia(funName, jlargs) 14. └─JuliaConnectoR:::handleCallbacksAndOutput() [ FAIL 13 | WARN 13 | SKIP 0 | PASS 420 ] Error: Test failures Execution halted R CMD check generated the following check_fails: 1. rcmdcheck_tests_pass 2. rcmdcheck_reasonable_installed_size #### Test coverage with [covr](https://covr.r-lib.org/) ERROR: Test Coverage Failed #### Cyclocomplexity with [cyclocomp](https://github.com/MangoTheCat/cyclocomp) The following functions have cyclocomplexity >= 15: function | cyclocomplexity --- | --- get_cluster | 35 gtools_permutations | 34 boot_algo_fastnreliable | 27 boot_aggregate | 22 boot_algo_julia | 20 getBoottest_nthreads | 16 check_boottest_args_plus | 15 #### Static code analyses with [lintr](https://github.com/jimhester/lintr) [lintr](https://github.com/jimhester/lintr) found the following 41 potential issues: message | number of times --- | --- Avoid library() and require() calls in packages | 16 Lines should not be more than 80 characters. | 25
:heavy_multiplication_x: The following function name is duplicated in other packages: - - `pval` from CIPerm, dostats, dostats, GUIDE, learnstats, molic, mosaic, overlapptest, RPtests, timereg
|package |version | |:--------|:--------| |pkgstats |0.1.3.4 | |pkgcheck |0.1.1.19 |
Processing may not proceed until the items marked with :heavy_multiplication_x: have been resolved.
helske
commented
1 year ago Hi @s3alfisc, I reran the the automatic checks which still show some issues, but based on the discussion above it seems that the these are mainly are caused by problems in our side so I will proceed with seeking for reviewers.
Regarding the items marked with ✖️:
codemeta.json file, could you please create this for example with the codemetar (https://github.com/ropensci/codemetar) package by running codemetar::write_codemeta() when in the package root directory.helske
commented
1 year ago Thanks again for your submission @s3alfisc! In addition to the notes above, few more comments:
desc::desc_normalize() for your package which reformats the DESCRIPTION fields in standard way (e.g., orders the packages in Imports field alphabetically).testthat contain several consecutive empty lines; I suggest using only one to make the files bit more readable. The styler package might be helpful here.requireNamespace in tests, but given that those packages are already in declared in DESCRIPTION, you probably don't need this. Alternatively, you could replace those calls with testthat packages skip_if_not_installed which should properly indicate you that the test was skipped. helske
commented
1 year ago @ropensci-review-bot seeking reviewers
ropensci-review-bot
commented
1 year ago Please add this badge to the README of your package repository:
[](https://github.com/ropensci/software-review/issues/546)Furthermore, if your package does not have a NEWS.md file yet, please create one to capture the changes made during the review process. See https://devguide.ropensci.org/releasing.html#news
helske
commented
1 year ago @ropensci-review-bot assign @meghapsimatrix as reviewer
ropensci-review-bot
commented
1 year ago @meghapsimatrix added to the reviewers list. Review due date is 2023-04-17. Thanks @meghapsimatrix for accepting to review! Please refer to our reviewer guide.
rOpenSci’s community is our best asset. We aim for reviews to be open, non-adversarial, and focused on improving software quality. Be respectful and kind! See our reviewers guide and code of conduct for more.
ropensci-review-bot
commented
1 year ago @meghapsimatrix: If you haven't done so, please fill this form for us to update our reviewers records.
s3alfisc
commented
1 year ago Hi @helske - thanks for being the editor of this package, and for your first feedback. I will try to address all of your initial comments within the next days. Best, Alex
s3alfisc
commented
1 year ago @mpadge asked me to clarify that the review should be based on the "master" branch - the repo used to have a dedicated "ropensci" branch - to which this submission was linked - which I have deprecated a while back as it had gotten stale.
Besides, I have a paper draft for fwildclusterboot, which is not yet in a state to be shared widely, but which I think could be helpful to reviewers, as it gives details on some of the few small deviations of the algorithm implemented in fwildclusterboot vs the fast and wild paper. Should I send you this draft @meghapsimatrix?
I have implemented all the changes @helske has asked for, except for the readme header? Could you please clarify, @helske? I am unfortunately not sure what you'd like me to do =)
meghapsimatrix
commented
1 year ago Yes please send the draft. Thanks!
On Sat, Apr 1, 2023 at 6:39 AM Alexander Fischer @.***> wrote:
@mpadge https://github.com/mpadge asked me to clarify that the review should be based on the "master" branch - the repo used to have a dedicated "ropensci" branch - to which this submission was linked - which I have deprecated a while back as it had gotten stale.
Besides, I have a paper draft for fwildclusterboot, which is not yet in a state to be shared widely, but which I think could be helpful to reviewers, as it gives details on some of the few small deviations of the algorithm implemented in fwildclusterboot vs the fast and wild paper. Should I send you this draft @meghapsimatrix https://github.com/meghapsimatrix?
— Reply to this email directly, view it on GitHub https://github.com/ropensci/software-review/issues/546#issuecomment-1492945804, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7RFVDUKQSNA2UJQRYQLL3W7AHWTANCNFSM5Z2NNLMA . You are receiving this because you were mentioned.Message ID: @.***>
-- Megha Joshi, PhD she/her/hers meghapsimatrix.com
helske
commented
1 year ago Thanks Alex, looking good. Regarding the readme.Rmd, this line defines the title for the readme as well as for the pkgdown site:
https://github.com/s3alfisc/fwildclusterboot/blob/1783a83a39aedb38e1f25983b3a6ccadcd183436/README.Rmd#L18
i.e. these:
and
meghapsimatrix
commented
1 year ago Please check off boxes as applicable, and elaborate in comments below. Your review is not limited to these topics, as described in the reviewer guide
The package includes all the following forms of documentation:
URL, BugReports and Maintainer (which may be autogenerated via Authors@R).Estimated hours spent reviewing:
This is a great package with super fast implementation of cluster wild bootstrap. Thanks for the opportunity to review.
R CMD check
1 NOTE
❯ checking top-level files ... NOTE
Non-standard file/directory found at top level:
‘codemeta.json’Comments on vignettes and documentation and comments on trying to use the functions (organized based on sections of the pkgdown website https://s3alfisc.github.io/fwildclusterboot/index.html)
README
Get Started Vignette
The vignette could go over why it is important to run many bootstrap replications. And, what does many mean? Also, is there a point where the number of bootstraps is too excessive?
The vignette could also provide a high level overview of the algorithm that makes cluster wild bootstrapping run fast, especially in R. The fastness seems to be the core idea behind the package so some detail on that somewhere on the vignette might be good.
Under the The boottest() function section, the text says that you are creating a random dataset but the code shows that you are using the voters dataset.
The vignette could also go over when to use which weights. For example, if we have a small number of clusters, Rademacher weights may not be appropriate.
Function Reference
The functions could be grouped together in some logical way. Right now, the list of functions is organized alphabetically.
I like the tidy() implementation a lot. Small improvement would be to change the column names to snake_case and add which types of weights were used to bootstrap (e.g., Rademacher).
I am a bit confused about interpreting the plot from the plot() function. A bit more explanation of what the blue and red lines mean could help :)
A bit more explanation on the different bootstrap_type options would help too.
In the documentation page for boottest it says we can use either boottest's seed argument or set the seed globally using set.seed() and dqrng::dqset.seed(). I tried using the seed argument but was not able to run it. Adding the seed argument directly to the boottest function would be an improvement and make the set up more use-friendly rather than having to set seed using both set.seed()and dqrng::dqset.seed().
library(fwildclusterboot)
dqrng::dqset.seed(2352342)
set.seed(23325)
data(voters)
# estimate the regression model via lm
lm_fit <- lm(
proposition_vote ~ treatment + ideology1 + log_income + Q1_immigration ,
data = voters
)
boot_lm <- boottest(
lm_fit,
clustid = "group_id1",
param = "treatment",
B = 9999,
seed = 20230412
)Error: in boottest.lm(lm_fit, clustid = "group_id1", param ...:
'seed' is not a valid argument of function boottest.lm.
In addition: Warning message:
Please note that the seeding behavior for random number generation for `boottest()` has changed with `fwildclusterboot`
version 0.13.
It will no longer be possible to exactly reproduce results produced by versions lower than 0.13.
If your prior results were produced under sufficiently many bootstrap iterations, none of your conclusions will change. For
more details about this change, please read the notes in
[news.md](https://cran.r-project.org/web/packages/fwildclusterboot/news/news.html).
This warning is displayed once per session.
Called from: stop_up(my_call, res)Vignettes fwildclusterboot and fixest
helske
commented
1 year ago Thanks for the review @meghapsimatrix! I apologize that unfortunately it seems that the review template link in my email was incorrect, as your review used the basic template used for non-statistical-software packages. Would you mind filling the correct one, which can be found here: https://stats-devguide.ropensci.org/pkgreview.html#pkgrev-template?
@s3alfisc I am still seeking for another reviewer, you might want to hold on with non-trivial software updates and replies until we've gotten a second review as well.
meghapsimatrix
commented
1 year ago Sorry about that! Will fill out the correct one!
On Fri, Apr 14, 2023 at 12:47 AM Jouni Helske @.***> wrote:
Thanks for the review @meghapsimatrix https://github.com/meghapsimatrix! I apologize that unfortunately it seems that the review template link in my email was incorrect, as your review used the basic template used for non-statistical-software packages. Would you mind filling the correct one, which can be found here: https://stats-devguide.ropensci.org/pkgreview.html#pkgrev-template?
@s3alfisc https://github.com/s3alfisc I am still seeking for another reviewer, you might want to hold on with non-trivial software updates and replies until we've gotten a second review as well.
— Reply to this email directly, view it on GitHub https://github.com/ropensci/software-review/issues/546#issuecomment-1507954363, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7RFVFQX2ZASTXK2U3X6RLXBDQFNANCNFSM5Z2NNLMA . You are receiving this because you were mentioned.Message ID: @.***>
-- Megha Joshi, PhD she/her/hers meghapsimatrix.com
s3alfisc
commented
1 year ago Thanks for your review @meghapsimatrix! I will start incorporating your feedback over the next days and will also reply in more detail =)
ropensci-review-bot
commented
1 year ago :calendar: @meghapsimatrix you have 2 days left before the due date for your review (2023-04-17).
s3alfisc
commented
1 year ago Hi @meghapsimatrix, first of all, thanks for your review! It thought it easiest to address it point by point =)
fwildclusterboot and fixest vignette - I mostly have it because I would frequently get questions on how to use fixest multiple estimation capabilities with boottest, and thought it might be good to have something to point to. set.seed() and dqrng.dqset.seed() in a function overwrites the global seed state, which is likely not super dangerous in the way users might interact with fwildclusterboot, but considered bad practice. I have an entire discussion with myself here =) I should definitely delete the seed argument you found in the vignette though! Best, Alex
meghapsimatrix
commented
1 year ago Here is the correct template filled out :) @s3alfisc thanks for addressing my comments :) Great job on this super useful package :)
Please check off boxes as applicable, and elaborate in comments below. Your review is not limited to these topics, as described in the reviewer guide
The following standards currently deemed non-applicable (through tags of @srrstatsNA) could potentially be applied to future versions of this software: (Please specify)
Please also comment on any standards which you consider either particularly well, or insufficiently, documented.
- G1.0 is well documented. The authors have made clear connections to academic literature.
- G.1.1 is also well documented. The authors provide citations to the novel-ness of the algorithm in general and in R.
- G.1.3 Some statistical terminology could be expanded upon more. For example, vignettes use terms like restricted and unrestricted bootstraps which could use bit more explanation :)
- RE4.18 and 4.3 to 4.5 Summary output and tidy outputs are well done.
For packages aiming for silver or gold badges:
- Compliance with a good number of standards beyond those identified as minimally necessary. Numbers of standards: G : 50 / 68; RE : 20 / 48; Total : 70 / 116
- Demonstrating excellence in compliance with multiple standards from at least two broad sub-categories.The package demonstrates excellence in compliance with G.1.0, G.1.1, RE4.18 and RE4.3 to 4.5
- Generality of usage beyond one single envisioned use case. The package supports ordinary least squares (OLS), fixed effects, and instrumental variables (IV) regression models. Subcluster bootstrapping, multiway bootstrapping, confidence intervals calculations are also supported.
- Internal aspects of package structure and design. Algorithms are written in a way that is efficient, flexible, generalizable, and accurate. Right now the package works for OLS and IV models. More types of models, like multilevel models, could be incorporated.
The package includes all the following forms of documentation:
Estimated hours spent reviewing: 10 hours
I already provided narrative comments which have been addressed :) Thanks!!
s3alfisc
commented
1 year ago Hi Megha, I have opened a (yet incomplete PR) on the dev branch to incorporate your feedback. The idea is to add a "FAQ" vignette in which I will try to explain some key concepts without too much technical jargon. I have also added you as a package contributor!
meghapsimatrix
commented
1 year ago Thank you!
On Sat, Apr 29, 2023 at 7:43 AM Alexander Fischer @.***> wrote:
Hi Megha, I have opened a (yet incomplete PR) on the dev branch https://github.com/s3alfisc/fwildclusterboot/pull/114to incorporate your feedback. The idea is to add a "FAQ" vignette in which I will try to explain some key concepts without too much technical jargon. I have also added you as a package contributor!
— Reply to this email directly, view it on GitHub https://github.com/ropensci/software-review/issues/546#issuecomment-1528779722, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7RFVCVCPCXUXZ4C25XFF3XDUEFLANCNFSM5Z2NNLMA . You are receiving this because you were mentioned.Message ID: @.***>
-- Megha Joshi, PhD she/her/hers meghapsimatrix.com
helske
commented
1 year ago @ropensci-review-bot assign @markean as reviewer
ropensci-review-bot
commented
1 year ago @markean added to the reviewers list. Review due date is 2023-05-29. Thanks @markean for accepting to review! Please refer to our reviewer guide.
rOpenSci’s community is our best asset. We aim for reviews to be open, non-adversarial, and focused on improving software quality. Be respectful and kind! See our reviewers guide and code of conduct for more.
ropensci-review-bot
commented
1 year ago @markean: If you haven't done so, please fill this form for us to update our reviewers records.
ropensci-review-bot
commented
1 year ago :calendar: @markean you have 2 days left before the due date for your review (2023-05-29).
markean
commented
1 year ago Hi @s3alfisc. Thank you for submitting your interesting package. I have attached my review below, which is based on the latest GitHub version. Hope it is helpful!
Please check off boxes as applicable, and elaborate in comments below. Your review is not limited to these topics, as described in the reviewer guide
The following standards currently deemed non-applicable (through tags of @srrstatsNA) could potentially be applied to future versions of this software: (Please specify)
Please also comment on any standards which you consider either particularly well, or insufficiently, documented.
I have checked the standards via the srr package by running srr::srr_report(). The author has put all the tags in a single file and just stated that many standards have been applied (e.g., Done, Yes, Checked, etc.), which makes it rather difficult to see how each standard has been reflected in the code. It would be easier to review if the tags were placed in relevant locations of the package. Below I have some comments on the compliance with standards.
G2.3b: It seems many character inputs are case sensitive. For example, see below.
library(fwildclusterboot)
lm_fit <- lm(
proposition_vote ~ treatment + ideology1 + log_income + Q1_immigration,
data = voters
)
boot <- boottest(lm_fit,
B = 1000, param = "treatment", clustid = "group_id1",
type = "NORM"
)Error: in boottest.lm(lm_fit, B = 1000, param = "...:
Argument 'type' must be a
single character equal to
'rademacher', 'mammen',
'norm', 'gamma' or 'webb'.
Problem: the value 'NORM'
doesn't match any choice
(note that no partial
matching is performed).RE1.0: Which parameter can be specified as a formula? Do you have any example?
RE1.1: Should it be tagged as 'srrstatsNA'?
RE3.0, RE3.1: Apart from the code quality, can the error/warning message be suppressed? This is not explicitly stated.
RE5.0, RE6.1, RE6.2, RE6.3, RE7.0: No response can be found in the srr-stats-standards.R file. Are they all not applicable?
G5.8a, G5.8b: Why are they not applicable? Are they tested?
RE4.8 ~ RE7.4 (at the bottom of the srr-stats-standards.R file): No response can be found.
For packages aiming for silver or gold badges:
The package includes all the following forms of documentation:
I think adding 'Details' sections in the generic boottest() function would improve general accessability. Some high-level descriptions, such as what each method does depending on the object class or what assumptions are made, would be helpful. See, for example, https://r-pkgs.org/man.html#title-description-details. Although references are introduced in the documentation, many users would not be willing to search for the articles. Additionally, the reference section could adopt a coherent bibliographical style.
It seems there is no way to gracefully interrupt or abort the execution of the boottest() function in an R session.
This can be an issue especially in the bootstrap context where the iteration number could be very large.
It can be difficult address this issue when interfacing with other languages or APIs in R, but I was wondering if this has been considered in the package design.
In the plot() method, it needs to be explained what the blue and red lines indicate either in the plot itself or in the documentation. Additionally, since it is a generic method, it would be good if the method supports other arguments. It does not accept some common arguments, which contradicts the usage of ... in the documentation.
See below.
library(fwildclusterboot)
lm_fit <- lm(
proposition_vote ~ treatment + ideology1 + log_income + Q1_immigration,
data = voters
)
boot <- boottest(lm_fit,
B = 9999,
param = "treatment",
clustid = "group_id1"
)
plot(boot, main = "My Title")Error: in plot.boottest(boot, main = "Title"):
'main' is not a valid argument of function plot.boottest.The argument R in the boottest() function currently seems to only accept a numeric vector. Many existing R packages for hypothesis testing (e.g., the car package with the hypothesis.matrix argument in the linearHypothesis() function or the multcomp package with the linfct argument in the glht() function) allow users to specify a hypothesis symbolically (or at least using a character vector). Can this feature be added in the package? I believe new users to this package would expect this functionality.
Can the seed mechanism be unified using only the set.seed() function? Many users would expect reproducibility by simply using set.seed(), not dqrng::dqset.seed().
The use of rlang for producing informative messages to the console is very good. However, unexpected line breaks occur for lengthy messages. This seems to be caused by manually applied or typed margin columns. Although it is aesthetically pleasing in the source code, I would recommend fixing this issue for general users. See below for example.
library(fwildclusterboot)
lm_fit <- lm(
proposition_vote ~ treatment + ideology1 + log_income +
Q1_immigration,
data = voters
)
boot <- boottest(lm_fit,
B = 9999,
clustid = c("group_id1", "group_id2"),
param = c("treatment", "ideology1"),
R = c(1, 1, 1),
r = 2
)Error in `process_R()`:
! The constraints vector must either be NULL or a numeric of
the same length as the `param` input vector.Estimated hours spent reviewing: 12 hours
s3alfisc
commented
1 year ago Hi @markean,
Thanks a lot for your thorough and very helpful review! One key takeaway from your and @meghapsimatrix's review is that I should assume less specialized knowledge from my package's users and explain multiple concepts in more detail. I will definitely work on that!
I am sorry about not placing the standard tags in the code - I tried this at some point, but it soon appeared to me that my code would be clustered with tags if I handled it that way. Maybe I should have just done so anyways, and am happy to do it if it helps you assess how the package complies with standards, @markean.
Regarding the standards that you mentioned in more detail:
RE1.0: Which parameter can be specified as a formula? Do you have any example?
The param and clustid arguments can be specified as formula. E.g. you can run boottest() as
library(fwildclusterboot)
data(voters)
fit <- lm(proposition_vote ~ treatment, data = voters)
boottest(fit, param = ~treatment, clustid = ~group_id1, B = 999)RE1.1: Should it be tagged as 'srrstatsNA'? Yes, you are right!
RE3.0, RE3.1: Apart from the code quality, can the error/warning message be suppressed? This is not explicitly stated.
No, this is currently not possible, and I will put a verbose argument on my to do list. Great suggestion, thanks!
RE5.0, RE6.1, RE6.2, RE6.3, RE7.0: No response can be found in the srr-stats-standards.R file. Are they all not applicable? Yes, they are either not applicable, or I did not fully understand what they are asking for... 😅 I will make sure to add them to the non-applicable section and add a comment.
G5.8a, G5.8b: Why are they not applicable? Are they tested?
Regarding the zero length data, I thought it would be not applicable, as data is practically never provided to boottest(). For all other non-supported data types, I assumed that I should allow anything that passes through lm() / model.matrix(). The key bottleneck is for models with fixed effect estimation, where I internally transform all model fixed effects to a factor. I have currently one open bug, which occurs when there is a date variable specified in the model. In this case, the creation of the model.frame() of the clustering variable fails. I will have to fix this bug, and add a particular test case. Are there any other special variable types you think I should be testing?
RE4.8 ~ RE7.4 (at the bottom of the srr-stats-standards.R file): No response can be found. Yes - for some cases, I did not even try to provide a response, as they appeared too "out of scope". I.e. for "RE6.3 Where a model object is used to generate a forecast (for example, through a predict() method), the default plot method should provide clear visual distinction between modelled (interpolated) and forecast (extrapolated) values." I did not provide an explanation for applicability or non-applicability, because prediction is clearly so much out of scope. For reasons of housekeeping, it would probably have been better if I had added a "out of scope" or "no forecasting functionality" comment. I will make sure to add these comments.
I think adding 'Details' sections in the generic boottest() function would improve general accessability.
This is a great suggestion, I will make sure to add this.
It seems there is no way to gracefully interrupt or abort the execution of the boottest() function in an R session.
In fact I currently don't know how to do this. The "main" algorithm is fully vectorized. The c++ code that cannot be interrupted is either a matrix multiplication via rcpp::eigen or the creation of the G X B bootstrap weights matrix. For the heteroskedastic bootstrap, there are some c++ for loops, and I think that rcpp provides options to (more or less, or even fully) gracefully abort. I will read up on this!
In the plot() method, it needs to be explained what the blue and red lines indicate either in the plot itself or in the documentation. Additionally, since it is a generic method, it would be good if the method supports other arguments. It does not accept some common arguments, which contradicts the usage of ... in the documentation.
All great points, I'll work on that.
wildmeta) and because the algorithm in the fast and wild paper is quite cumbersome :D But based on the algorithms in the new-ish MacKinnon et al papers, it should be one day of work to implement a Wald test (note that I am always too optimistic with these things). Btw, note that a Wald test can be run via the mboottest() functions, but only through WildBootTests.jl. Can the seed mechanism be unified using only the set.seed() function? Many users would expect reproducibility by simply using set.seed(), not dqrng::dqset.seed().
Unfortunately, I don't think so. Generation of rademacher, webb and normal weights via dqrng::dqset.seed() is significantly faster than via e.g. stats::sample() (see the benchmark I have once posted here). But unfortunately, the dqrng package does not support Mammen weights. though I think this option will be added soon. So I think that I could fully switch to dqrng for the random sampling for the cluster bootstrap. For the heteroskedastic bootstrap, I sample "directly in rcpp", which does not allow me to call drqrng (at least not without jumping through some hoops and calling it's c/c++ code directly, which I am afraid might be beyond my abilities). So for now, I don't think I can change this, or at least not without significant time investment. One solution could be the following: I could introduce a fwildclusterboot::set_seed() function, that calls set.seed() and dqrng::dqset.seed() and explicitly tells users that both functions are being called. This + very explicit documentation might be a good option?
rlang error message line breaks - I think I introduced these to pass the goodpractice line length checks 😅 There's probably a way to stay within the 80 character limit and not have these annoying breaks in the console. I'll do some googling / ask chatGPT =) So, this was a very long response - sorry about that 😄. Thanks so much for your review @markean! I will try to address all your (and @meghapsimatrix's) feedback swiftly (I plan to work on it over the next weekend) and will ping you once there are updates / once I am finished.
Best, Alex
mpadge
commented
1 year ago @s3alfisc Perspective from the rOpenSci stats team is that standards documentation should definitely be done inline, at each point in the code at which compliance actually occurs. This enables both easy understanding of compliance, and far easier long-term maintenance for you, as you yourself then don't have to constantly switch back-and-forth between two separate locations for code and compliance documentation. We have also updated our check system to ensure that documentation of stats compliance is not concentrated in a single file, so your package may well fail that check if run again now? (If you want to see, it's probably best to set up our pkgcheck-action workflow in your repo, rather than cluttering the review here with extra checks.) Hope that helps :smiley:
s3alfisc
commented
1 year ago Thanks @mpadge! This is the first thing I'll tackle then :)
s3alfisc
commented
1 year ago Just to give an update - I am working on incorporating feedback into the package (see the dev branch). I hope that I will be done by the end of next weekend.
jhollist
commented
9 months ago @s3alfisc I am currently serving as the rOpenSci EIC and just checking in on things.
How's it coming on the updates to your package from the review process?
s3alfisc
commented
9 months ago Hi @jhollist , thanks for checking up on this.
The honest status update is that I have, to some degree, given up on adressing all of the comments I received out of the review process. The main hurdle are the inline standards documentation via roxygen tags. To some degree, I feel that many of them do not really apply to the package, as fwildclusterboot is not a package for regression fitting but for regression inference. And then with the tags that do apply, it seems like an incredible amount of work to put them in every single spot of the package where they apply. I've started this multiple times but have repeatedly given up.
On the other hand, I did incorporate many of the other points I got feedback on:
plot.boottest().boottest() that links to the detailed vignettes. I'm not really sure what the best way forward will be. I have quite a bit of time off until the New Year, so chances are I pull myself together and make sure all the tags can be found in the codebase. To put some pressure on me - how about we keep this review open until Jan 2, and if I haven't made sufficient progress until then, we close the review?
Best, Alex
ldecicco-USGS
commented
7 months ago @helske , do you have any feedback for @s3alfisc ? I'm also going to tag @mpadge concerning the feedback about difficulties in the inline standards documentation via roxygen tags. I have no experience to offer any help on that topic.
s3alfisc
commented
7 months ago Hi, reposting for visibility a message I sent to @mpadge on the topic:
In my memory, the main thing that I still needed to do for the
fwildclusterbootreview was to include the srr tags into the fwildclusterboot codebase. For every tag that was relevant, I placed it in at least one point in the codebase - I decided not to place the same tags in each boottest() method, e.g. onlyboottest_fixestis properly tagged whereasboottest_lmandboottest_lfeare not. For quite a few tags, I could not find the "one" spot in the codebase, so I left them in the srr file with a comment. For example for a tag as "never compare ints with bools", I could not find a proper spot. I just don't think I do so! Otherwise, many tags are just not relevant - as fwildclusterboot is a post regression method, NA values need to be handled by the regression package which fwildclusterboot supports; or all tags relating predictions. Beyond my main takeaway from the second review was that I needed to fix the standards and improve the documentation of statistical terms. https://github.com/ropensci/software-review/issues/546#issuecomment-1566285672 For this, I have significantly revised the vignettes at some point, added details sections to the docs, etc. There are other things that I am not perfectionist enough to fix, i.e. a consistent citation format for all references, and then things that I simply cannot fix.
helske
commented
7 months ago @mpadge please correct me if I'm wrong, but my impression on the standard tags is that it is fine to have some general tags relevant for the whole package for example in the main package documentation, e.g., in fwildclusterboot-package.R which could contain some overall info on the package accessible by ?fwildclusterboot (similar text as in the DESCRIPTION). For non-applicable standards, you could also include these in the above file as well, with a short description on why they do not apply, which I see you have already done in the srr-stats-standards.R.
Other than than the placement of the standards, if you feel that you have adequately reacted to all points by the reviewers, then we can ask them to take a look and see if they feel ready to approve the package.
mpadge
commented
7 months ago @ldecicco-USGS and @helske I've had a look through the code, and from my side at least would now be happy with the way that the standards have been documented in-line within the code. I've opened a short TODO list for @s3alfisc at https://github.com/s3alfisc/fwildclusterboot/issues/149. Alex, please report back once you've addressed those, and all should be good to proceed. Thanks :+1:
s3alfisc
commented
6 months ago I'll tackle all of your suggestions this weekend. Thanks for the feedback @mpadge! I'll report here once I think I am done =)
Submitting Author Name: Alexander Fischer Submitting Author Github Handle: !--author1-->@s3alfisc<!--end-author1-- Other Package Authors Github handles: (comma separated, delete if none) @droodman Repository: https://github.com/s3alfisc/fwildclusterboot/tree/master Version submitted: 0.9.1 (in ropensci branch) Submission type: Stats Badge grade: silver Editor: !--editor-->@helske<!--end-editor-- Reviewers: @meghapsimatrix, @markean
Due date for @meghapsimatrix: 2023-04-17 Due date for @markean: 2023-05-29Archive: TBD Version accepted: TBD Language: en
Pre-submission Inquiry
General Information
Who is the target audience and what are scientific applications of this package? The target audience is academic social scientists (economics, political science, sociology). fwildclusterboot should be used whenever regression errors are "clustered" into few groups, in which case inference based on asymptotic approximations might fail.
Paste your responses to our General Standard G1.1 here, describing whether your software is:
fwildclusterbootis the first R-implementation of the 'fast' wild cluster bootstrap algorithm outlined in Roodman et al (2019, Stata Journal) and implemented in Stata via the boottest package. Recently, @droodman has also ported the algorithm to Julia. Besides implementing the "fast & wild" algorithm in native R,fwildclusterbootalso allows to callWildBootTests.jlvia theJuliaConnectoRpackage. The "fast" algorithm is significantly faster than any other R-implementations of the wild cluster bootstrap to date (e.g. see here).Yes.
Badging
At a minimal level, I strive for a silver badge.
Technical checks
Confirm each of the following by checking the box.
autotestchecks on the package, and ensured no tests fail.srr_stats_pre_submit()function confirms this package may be submitted.pkgcheck()function confirms this package may be submitted - alternatively, please explain reasons for any checks which your package is unable to pass.pkgcheck::pkgcheck()currently fails on my machine with the following error message: "Error in normalizePath(path.expand(path), winslash, mustWork).: path[1]="C:/.../fwildclusterboot/NA" The system cannot find the specified file. It also fails on github-actions on theropenscibranch.Also, note that the CMD checks currently fail on github actions as the
gtoolspackage has been orphaned - I am monitoring the situation, but it looks like the package will not be removed from CRAN.This package:
Publication options
Code of conduct