rstudio / shinytest

Automated testing for shiny apps
https://rstudio.github.io/shinytest/
Other
225 stars 55 forks source link

Feature request: `testApp(..., update = FALSE)` #418

Open bersbersbers opened 2 years ago

bersbersbers commented 2 years ago

The package I develop is under version control, and quite honestly, I prefer git's and my IDE's diffing routines over shinytests. Therefore, instead of using the built-in diff viewer (which compares expected to current) I very often issue testApp followed by an automatic snapshotUpdate (to compare the working-copy expected with the previous expected).

What would help me quite a bit is if I could, instead of running testApp(); snapshotUpdate(), run testApp(update = TRUE). Why? For two reasons:

  1. When running many tests, I get earlier feedback if test 1 is run, then updated; test 2 is run, then updated, ... Today, I can only run tests 1-10, and then update tests 1-10 (I am aware of the testnames parameter to run tests individually, but this means I have to duplicate shinytests logic for test discovery, shinytest:::findTests not being public).
  2. When running a subset of tests using shinytest::snapshotUpdate(testnames = ...), it is sometimes unclear for which tests I need to run the update: I may not want to run it on all tests, for example; and I shall not call it on tests which passed to avoid

    shinytest::snapshotUpdate("inst/shiny", testnames = c("mytest")) Error: Current result directory not found: inst/shiny/tests/shinytest/mytest-current

My idea would be to add update = FALSE as an argument to shinytest::testApp and add https://github.com/rstudio/shinytest/blob/d0f6d737919b542c6a1e8bbd0ea05f9e4b3f5334/R/test-app.R#L62-L66

67    if (update) snapshotUpdate(path$dir, testnames = testname, quiet = quiet, suffix = suffix)
bersbersbers commented 2 years ago

This works well for me:

diff --git a/R/test-app.R b/R/test-app.R
index 7436560..1793160 100644
--- a/R/test-app.R
+++ b/R/test-app.R
@@ -14,6 +14,7 @@
 #' @param interactive If there are any differences between current results and
 #'   expected results, provide an interactive graphical viewer that shows the
 #'   changes and allows the user to accept or reject the changes.
+#' @param updateExpected Update snapshots in case of differences after testing.
 #' @param suffix An optional suffix for the expected results directory. For
 #'   example, if the suffix is `"mac"`, the expected directory would be
 #'   `mytest-expected-mac`.
@@ -30,6 +31,7 @@ testApp <- function(
   quiet = FALSE,
   compareImages = TRUE,
   interactive = is_interactive(),
+  updateExpected = !interactive,
   suffix = NULL
 )
 {
@@ -64,11 +66,25 @@ testApp <- function(
     }
     env <- new.env(parent = .GlobalEnv)
     source(testname, local = env)
+
+    # Update expected results
+    if (updateExpected) {
+      snapshotUpdate(
+        path$dir,
+        testnames = sub("\\.[rR]$", "", testname),
+        quiet = TRUE,
+        suffix = suffix
+      )
+    }
   })

   gc()
   if (!quiet) message("")  # New line

+  if (updateExpected) {
+    return() # Nothing to compare
+  }
+
   # Compare all results
   snapshotCompare(
     path$dir,

I set the default to !interactive so I can call it using the same way as the released version, but I am fine with any default value once it's released.

I'll test this a bit and would come up with a PR later, but feel free to take it earlier.