search

r-lib / testthat

An R πŸ“¦ to make testing πŸ˜€
https://testthat.r-lib.org
Other
869 stars 313 forks source link

JunitReporter fails badly on warning outside of test_that #1913

Open pawelru opened 7 months ago

pawelru commented 7 months ago

Hi! I wanted to share an use case in which JunitReporter throws and others don't.

I had the following (pseudo-code)

new_foo <- \(...) {...}

old_foo <- \(...) {
  lifecycle::deprecate_warn(...)
  new_foo(...)
}

In test file:

x <- old_foo(...)

test_that("foo is working", {
  ...
})

When using JunitReporter and only(!) during check it fails as follows:

  Error in UseMethod("xml_add_child") : 
    no applicable method for 'xml_add_child' applied to an object of class "NULL"
  Calls: test_check ... o_apply -> lapply -> FUN -> <Anonymous> -> <Anonymous>
  Execution halted

This line in particular: https://github.com/r-lib/testthat/blob/07c5cd99c3098b9f228258fe7a3df5a3e9a7a2b8/R/reporter-junit.R#L106-L111

For some reason self$suite is NULL when calling $add_result() method. My guess is it's because it's outside of test_that.

We have addressed this internally by changing x <- old_foo() into x <- new_foo() but I think it would be good to share it with you as this was not a problem for a default reporter (haven't checked all of them) as well as within TEST (as opposed to CHECK).

hadley commented 4 months ago

Would you mind making a very simple reprex package that illustrates the problem? That will make it faster for me to track down and fix when I'm next working on testthat.

pawelru commented 4 months ago 
library(usethis)
unlink("foo", recursive = TRUE)
create_package("foo")
#> βœ” Creating 'foo/'
#> βœ” Setting active project to '/private/var/folders/px/_50zh3ns2nd4pmd727cmwlc00000gn/T/RtmpOLXaLg/reprex-8b202694fcdc-irate-booby/foo'
#> βœ” Creating 'R/'
#> βœ” Writing 'DESCRIPTION'
#> Package: foo
#> Title: What the Package Does (One Line, Title Case)
#> Version: 0.0.0.9000
#> Authors@R (parsed):
#>     * First Last <first.last@example.com> [aut, cre] (YOUR-ORCID-ID)
#> Description: What the package does (one paragraph).
#> License: `use_mit_license()`, `use_gpl3_license()` or friends to pick a
#>     license
#> Encoding: UTF-8
#> Roxygen: list(markdown = TRUE)
#> RoxygenNote: 7.3.1
#> βœ” Writing 'NAMESPACE'
#> βœ” Setting active project to '<no active project>'
setwd("foo")
# a few steps to make R CMD CHECK less cluttered with notes
use_mit_license()
#> βœ” Setting active project to '/private/var/folders/px/_50zh3ns2nd4pmd727cmwlc00000gn/T/RtmpOLXaLg/reprex-8b202694fcdc-irate-booby/foo'
#> βœ” Adding 'MIT + file LICENSE' to License
#> βœ” Writing 'LICENSE'
#> βœ” Writing 'LICENSE.md'
#> βœ” Adding '^LICENSE\\.md$' to '.Rbuildignore'
use_package_doc()
#> βœ” Writing 'R/foo-package.R'
use_lifecycle()
#> βœ” Adding 'lifecycle' to Imports field in DESCRIPTION
#> β€’ Refer to functions with `lifecycle::fun()`
#> βœ” Adding '@importFrom lifecycle deprecated' to 'R/foo-package.R'
#> βœ” Writing 'NAMESPACE'
#> βœ” Creating 'man/figures/'
#> βœ” Copied SVG badges to 'man/figures/'
#> β€’ Add badges in documentation topics by inserting one of:
#>   #' `r lifecycle::badge('experimental')`
#>   #' `r lifecycle::badge('superseded')`
#>   #' `r lifecycle::badge('deprecated')`

# R/ directory
write('new_foo <- function() "new foo"', file = "R/new_foo.R")
write('old_foo <- function() {lifecycle::deprecate_warn("0.0.1", "old_foo()", "new_foo()"); new_foo()}', file = "R/old_foo.R")

# testthat
use_testthat()
#> βœ” Adding 'testthat' to Suggests field in DESCRIPTION
#> βœ” Adding '3' to Config/testthat/edition
#> βœ” Creating 'tests/testthat/'
#> βœ” Writing 'tests/testthat.R'
#> β€’ Call `use_test()` to initialize a basic test file and open it for editing.
write('x <- old_foo()', file = "tests/testthat/test-old_foo.R") # <- THIS LINE!!!
write('test_that("foo is working", {expect_true(TRUE)})', file = "tests/testthat/test-old_foo.R", append = TRUE)
# overwrite default reporter
write('library(testthat); library(foo)', file = "tests/testthat.R")
write('test_check("foo", reporter = JunitReporter$new())', file = "tests/testthat.R", append = TRUE)

# TEST -> OK: warnings are expected
devtools::test()
#> β„Ή Testing foo
#> βœ” | F W  S  OK | Context
#> 
#> ⠏ |          0 | old_foo                                                        
#> βœ” |   1      1 | old_foo
#> ────────────────────────────────────────────────────────────────────────────────
#> Warning ('test-old_foo.R:1:1'): (code run outside of `test_that()`)
#> `old_foo()` was deprecated in foo 0.0.1.
#> β„Ή Please use `new_foo()` instead.
#> Backtrace:
#>     β–†
#>  1. └─foo:::old_foo() at test-old_foo.R:1:1
#> ────────────────────────────────────────────────────────────────────────────────
#> 
#> ══ Results ═════════════════════════════════════════════════════════════════════
#> [ FAIL 0 | WARN 1 | SKIP 0 | PASS 1 ]

# R CMD CHECK - FAIL
rcmdcheck::rcmdcheck(args = "--no-manual")
#> ── R CMD build ─────────────────────────────────────────────────────────────────
#> * checking for file β€˜.../DESCRIPTION’ ... OK
#> * preparing β€˜foo’:
#> * checking DESCRIPTION meta-information ... OK
#> * checking for LF line-endings in source and make files and shell scripts
#> * checking for empty or unneeded directories
#> * building β€˜foo_0.0.0.9000.tar.gz’
#> 
#> ── R CMD check ─────────────────────────────────────────────────────────────────
#> * using log directory β€˜/private/var/folders/px/_50zh3ns2nd4pmd727cmwlc00000gn/T/RtmpARWDUn/file112d36216a04c/foo.Rcheck’
#> * using R version 4.3.2 (2023-10-31)
#> * using platform: aarch64-apple-darwin20 (64-bit)
#> * R was compiled by
#>     Apple clang version 14.0.0 (clang-1400.0.29.202)
#>     GNU Fortran (GCC) 12.2.0
#> * running under: macOS Ventura 13.5
#> * using session charset: UTF-8
#> * using option β€˜--no-manual’
#> * checking for file β€˜foo/DESCRIPTION’ ... OK
#> * this is package β€˜foo’ version β€˜0.0.0.9000’
#> * package encoding: UTF-8
#> * checking package namespace information ... OK
#> * checking package dependencies ... OK
#> * checking if this is a source package ... OK
#> * checking if there is a namespace ... OK
#> * checking for executable files ... OK
#> * checking for hidden files and directories ... OK
#> * checking for portable file names ... OK
#> * checking for sufficient/correct file permissions ... OK
#> * checking whether package β€˜foo’ can be installed ... OK
#> * checking installed package size ... OK
#> * checking package directory ... OK
#> * checking DESCRIPTION meta-information ... OK
#> * checking top-level files ... OK
#> * checking for left-over files ... OK
#> * checking index information ... OK
#> * checking package subdirectories ... OK
#> * checking R files for non-ASCII characters ... OK
#> * checking R files for syntax errors ... OK
#> * checking whether the package can be loaded ... OK
#> * checking whether the package can be loaded with stated dependencies ... OK
#> * checking whether the package can be unloaded cleanly ... OK
#> * checking whether the namespace can be loaded with stated dependencies ... OK
#> * checking whether the namespace can be unloaded cleanly ... OK
#> * checking loading without being on the library search path ... OK
#> * checking dependencies in R code ... OK
#> * checking S3 generic/method consistency ... OK
#> * checking replacement functions ... OK
#> * checking foreign function calls ... OK
#> * checking R code for possible problems ... OK
#> * checking Rd files ... OK
#> * checking Rd metadata ... OK
#> * checking Rd cross-references ... OK
#> * checking for missing documentation entries ... OK
#> * checking for code/documentation mismatches ... OK
#> * checking Rd \usage sections ... OK
#> * checking Rd contents ... OK
#> * checking for unstated dependencies in examples ... WARNING
#> no parsed files found
#> * checking examples ... NONE
#> * checking for unstated dependencies in β€˜tests’ ... OK
#> * checking tests ...
#>   Running β€˜testthat.R’
#>  ERROR
#> Running the tests in β€˜tests/testthat.R’ failed.
#> Complete output:
#>   > library(testthat); library(foo)
#>   > test_check("foo", reporter = JunitReporter$new())
#>   Error in UseMethod("xml_add_child") : 
#>     no applicable method for 'xml_add_child' applied to an object of class "NULL"
#>   Calls: test_check ... o_apply -> lapply -> FUN -> <Anonymous> -> <Anonymous>
#>   Execution halted
#> * DONE
#> 
#> Status: 1 ERROR, 1 WARNING
#> See
#>   β€˜/private/var/folders/px/_50zh3ns2nd4pmd727cmwlc00000gn/T/RtmpARWDUn/file112d36216a04c/foo.Rcheck/00check.log’
#> for details.
#> ── R CMD check results ───────────────────────────────────── foo 0.0.0.9000 ────
#> Duration: 6.5s
#> 
#> ❯ checking tests ...
#>   See below...
#> 
#> ❯ checking for unstated dependencies in examples ... WARNING
#>   no parsed files found
#> 
#> ── Test failures ───────────────────────────────────────────────── testthat ────
#> 
#> > library(testthat); library(foo)
#> > test_check("foo", reporter = JunitReporter$new())
#> Error in UseMethod("xml_add_child") : 
#>   no applicable method for 'xml_add_child' applied to an object of class "NULL"
#> Calls: test_check ... o_apply -> lapply -> FUN -> <Anonymous> -> <Anonymous>
#> Execution halted
#> 
#> 1 error βœ– | 1 warning βœ– | 0 notes βœ”

# session Info
sessionInfo()
#> R version 4.3.2 (2023-10-31)
#> Platform: aarch64-apple-darwin20 (64-bit)
#> Running under: macOS Ventura 13.5
#> 
#> Matrix products: default
#> BLAS:   /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libRblas.0.dylib 
#> LAPACK: /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libRlapack.dylib;  LAPACK version 3.11.0
#> 
#> locale:
#> [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
#> 
#> time zone: Europe/Zurich
#> tzcode source: internal
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#> [1] foo_0.0.0.9000 testthat_3.2.1 usethis_2.2.2 
#> 
#> loaded via a namespace (and not attached):
#>  [1] xfun_0.41         htmlwidgets_1.6.4 devtools_2.4.5    remotes_2.4.2.1  
#>  [5] processx_3.8.3    callr_3.7.3       vctrs_0.6.5       tools_4.3.2      
#>  [9] ps_1.7.6          curl_5.2.0        tibble_3.2.1      fansi_1.0.6      
#> [13] pkgconfig_2.0.3   R.oo_1.26.0       xopen_1.0.0       desc_1.4.3       
#> [17] lifecycle_1.0.4   R.cache_0.16.0    compiler_4.3.2    stringr_1.5.1    
#> [21] brio_1.1.4        httpuv_1.6.14     htmltools_0.5.7   yaml_2.3.8       
#> [25] later_1.3.2       pillar_1.9.0      crayon_1.5.2      urlchecker_1.0.1 
#> [29] whisker_0.4.1     ellipsis_0.3.2    R.utils_2.12.3    cachem_1.0.8     
#> [33] sessioninfo_1.2.2 mime_0.12         styler_1.10.2     digest_0.6.34    
#> [37] stringi_1.8.3     purrr_1.0.2       rprojroot_2.0.4   fastmap_1.1.1    
#> [41] cli_3.6.2         magrittr_2.0.3    pkgbuild_1.4.3    utf8_1.2.4       
#> [45] withr_3.0.0       prettyunits_1.2.0 waldo_0.5.2       promises_1.2.1   
#> [49] roxygen2_7.3.1    rmarkdown_2.25    R.methodsS3_1.8.2 memoise_2.0.1    
#> [53] shiny_1.8.0       evaluate_0.23     knitr_1.45        rcmdcheck_1.4.0  
#> [57] miniUI_0.1.1.1    profvis_0.3.8     rlang_1.1.3       Rcpp_1.0.12      
#> [61] xtable_1.8-4      glue_1.7.0        xml2_1.3.6        reprex_2.1.0     
#> [65] pkgload_1.3.4     rstudioapi_0.15.0 R6_2.5.1          fs_1.6.3

Created on 2024-02-13 with reprex v2.1.0

hadley commented 4 months ago

Thanks!