712 - `shinytest2` Run all example apps

[averissimo]

Pull Request

Part of #712

Changes description

• Adds tests that iterate on each documentation file and runs the examples apps by mocking interactive and shinyApp functions.
  • Checks if there are no errors nor validation errors (with exceptions)
• Implements https://github.com/insightsengineering/teal.modules.clinical/pull/983 on this repository

Changes from https://github.com/insightsengineering/teal.modules.clinical/pull/983

• Adds:
  • Regex rules define "accepted" validation errors
• Fixes:
  • Reverts to use library instead of pkgload::load_all due to problems with system.file call that cannot find package files.

diff -u teal.modules.clinical/tests/testthat/test-examples.R teal.modules.general/tests/testthat/test-examples.R 
--- teal.modules.clinical/tests/testthat/test-examples.R    2024-04-12 10:32:33.100707738 +0200
+++ teal.modules.general/tests/testthat/test-examples.R 2024-04-12 10:26:27.645642183 +0200
@@ -38,12 +38,7 @@ with_mocked_app_bindings <- function(code) {
   # change to `print(shiny__shinyApp(...))` and remove allow warning once fixed
   mocked_shinyApp <- function(ui, server, ...) { # nolint object_name_linter.
     functionBody(server) <- bquote({
-      pkgload::load_all(
-        .(normalizePath(file.path(testthat::test_path(), "..", ".."))),
-        export_all = FALSE,
-        attach_testthat = FALSE,
-        warn_conflicts = FALSE
-      )
+      library(.(testthat::testing_package()), character.only = TRUE)
       .(functionBody(server))
     })
     print(do.call(shiny__shinyApp, append(x = list(ui = ui, server = server), list(...))))
@@ -56,16 +51,34 @@ with_mocked_app_bindings <- function(code) {
     app_driver <- shinytest2::AppDriver$new(
       x,
       shiny_args = args,
+      timeout = 20 * 1000,
+      load_timeout = 30 * 1000,
       check_names = FALSE, # explicit check below
       options = options() # https://github.com/rstudio/shinytest2/issues/377
     )
     on.exit(app_driver$stop(), add = TRUE)
-    app_driver$wait_for_idle(timeout = 20000)
+    app_driver$wait_for_idle()

     # Simple testing
     ## warning in the app does not invoke a warning in the test
     ## https://github.com/rstudio/shinytest2/issues/378
     app_logs <- subset(app_driver$get_logs(), location == "shiny")[["message"]]
+
+    # Check if the teal app has content (indicator of a Shiny App fatal error)
+    if (identical(trimws(app_driver$get_text("#teal-main_ui_container")), "")) {
+      tryCatch(
+        app_driver$wait_for_idle(duration = 2000), # wait 2 seconds for session to disconnect
+        error = function(err) {
+          stop(
+            sprintf(
+              "Teal Application is empty. An Error may have occured:\n%s",
+              paste0(subset(app_driver$get_logs(), location == "shiny")[["message"]], collapse = "\n")
+            )
+          )
+        }
+      )
+    }
+
     # allow `Warning in file(con, "r")` warning coming from pkgload::load_all()
     if (any(grepl("Warning in.*", app_logs) & !grepl("Warning in file\\(con, \"r\"\\)", app_logs))) {
       warning(
@@ -79,9 +92,17 @@ with_mocked_app_bindings <- function(code) {
     ## Throw an error instead of a warning (default `AppDriver$new(..., check_names = TRUE)` throws a warning)
     app_driver$expect_unique_names()

+    err_el <- Filter(
+      function(x) {
+        allowed_errors <- getOption("test_examples.discard_error_regex", "")
+        identical(allowed_errors, "") || !grepl(allowed_errors, x)
+      },
+      app_driver$get_html(".shiny-output-error")
+    )
+
     ## shinytest2 captures app crash but teal continues on error inside the module
     ## we need to use a different way to check if there are errors
-    if (!is.null(err_el <- app_driver$get_html(".shiny-output-error"))) {
+    if (!is.null(err_el) && length(err_el) > 0) {
       stop(sprintf("Module error is observed:\n%s", err_el))
     }

@@ -110,11 +131,14 @@ with_mocked_app_bindings <- function(code) {

 strict_exceptions <- c(
   # https://github.com/r-lib/gtable/pull/94
-  "tm_g_barchart_simple.Rd",
-  "tm_g_ci.Rd",
-  "tm_g_ipp.Rd",
-  "tm_g_pp_adverse_events.Rd",
-  "tm_g_pp_vitals.Rd"
+  "tm_outliers.Rd",
+  "tm_g_response.Rd",
+  "tm_a_pca.Rd"
+)
+
+discard_validation_regex <- list(
+  "tm_file_viewer.Rd" = "Please select a file\\.",
+  "tm_g_distribution.Rd" = "Please select a test"
 )

 for (i in rd_files()) {
@@ -122,11 +146,18 @@ for (i in rd_files()) {
     paste0("example-", basename(i)),
     {
       testthat::skip_on_cran()
+      skip_if_too_deep(5)
       if (basename(i) %in% strict_exceptions) {
         op <- options()
         withr::local_options(opts_partial_match_old)
         withr::defer(options(op))
       }
+      # Allow for specific validation errors for individual examples
+      withr::local_options(
+        list(
+          "test_examples.discard_error_regex" = discard_validation_regex[[basename(i)]]
+        )
+      )
       with_mocked_app_bindings(
         # suppress warnings coming from saving qenv https://github.com/insightsengineering/teal.code/issues/194
         suppress_warnings(```

[github-actions[bot]]

Unit Tests Summary

  1 files   22 suites   10m 40s :stopwatch: 147 tests 147 :white_check_mark: 0 :zzz: 0 :x: 475 runs  475 :white_check_mark: 0 :zzz: 0 :x:

Results for commit b13ce16a.

:recycle: This comment has been updated with latest results.

[github-actions[bot]]

Unit Test Performance Difference

Test Suite	$Status$	Time on main	$±Time$	$±Tests$	$±Skipped$	$±Failures$	$±Errors$
examples	👶		$+4.95$	$+40$	$0$	$0$	$0$
shinytest2-tm_g_distribution	💚	$56.15$	$-1.01$	$0$	$0$	$0$	$0$
shinytest2-tm_outliers	💚	$87.00$	$-2.35$	$0$	$0$	$0$	$0$
shinytest2-tm_t_crosstable	💚	$27.88$	$-1.26$	$0$	$0$	$0$	$0$
shinytest2-tm_variable_browser	💚	$50.88$	$-1.95$	$0$	$0$	$0$	$0$

Additional test case details

| Test Suite | $Status$ | Time on `main` | $±Time$ | Test Case | |:-----|:----:|:----:|:----:|:-----| | examples | 👶 | | $+1.63$ | example_add_facet_labels.Rd | | examples | 👶 | | $+0.05$ | example_call_fun_dots.Rd | | examples | 👶 | | $+0.01$ | example_create_sparklines.Rd | | examples | 👶 | | $+0.01$ | example_establish_updating_selection.Rd | | examples | 👶 | | $+0.01$ | example_get_scatterplotmatrix_stats.Rd | | examples | 👶 | | $+0.01$ | example_include_css_files.Rd | | examples | 👶 | | $+0.01$ | example_is_tab_active_js.Rd | | examples | 👶 | | $+0.01$ | example_plot_var_summary.Rd | | examples | 👶 | | $+0.01$ | example_rADAE.Rd | | examples | 👶 | | $+0.01$ | example_rADLB.Rd | | examples | 👶 | | $+0.01$ | example_rADRS.Rd | | examples | 👶 | | $+0.01$ | example_rADSL.Rd | | examples | 👶 | | $+0.01$ | example_rADTTE.Rd | | examples | 👶 | | $+0.01$ | example_remove_outliers_from.Rd | | examples | 👶 | | $+0.01$ | example_render_single_tab.Rd | | examples | 👶 | | $+0.01$ | example_render_tab_header.Rd | | examples | 👶 | | $+0.01$ | example_render_tab_table.Rd | | examples | 👶 | | $+0.01$ | example_render_tabset_panel_content.Rd | | examples | 👶 | | $+0.01$ | example_shared_params.Rd | | examples | 👶 | | $+0.01$ | example_teal.modules.general.Rd | | examples | 👶 | | $+0.13$ | example_tm_a_pca.Rd | | examples | 👶 | | $+0.10$ | example_tm_a_regression.Rd | | examples | 👶 | | $+0.09$ | example_tm_data_table.Rd | | examples | 👶 | | $+0.15$ | example_tm_file_viewer.Rd | | examples | 👶 | | $+0.05$ | example_tm_front_page.Rd | | examples | 👶 | | $+0.10$ | example_tm_g_association.Rd | | examples | 👶 | | $+0.10$ | example_tm_g_bivariate.Rd | | examples | 👶 | | $+0.99$ | example_tm_g_distribution.Rd | | examples | 👶 | | $+0.11$ | example_tm_g_response.Rd | | examples | 👶 | | $+0.14$ | example_tm_g_scatterplot.Rd | | examples | 👶 | | $+0.22$ | example_tm_g_scatterplotmatrix.Rd | | examples | 👶 | | $+0.10$ | example_tm_missing_data.Rd | | examples | 👶 | | $+0.14$ | example_tm_outliers.Rd | | examples | 👶 | | $+0.58$ | example_tm_t_crosstable.Rd | | examples | 👶 | | $+0.12$ | example_tm_variable_browser.Rd | | examples | 👶 | | $+0.01$ | example_validate_input.Rd | | examples | 👶 | | $+0.01$ | example_var_missings_info.Rd | | examples | 👶 | | $+0.01$ | example_var_summary_table.Rd | | examples | 👶 | | $+0.01$ | example_variable_type_icons.Rd | | examples | 👶 | | $+0.01$ | example_varname_w_label.Rd | | shinytest2-tm_g_distribution | 💚 | $32.48$ | $-1.12$ | e2e_tm_g_distribution_module_is_initialised_with_the_specified_defaults |

Results for commit f8c339608ff7f6e9fc696775f4f7c1ce32ff31b2

♻️ This comment has been updated with latest results.

[averissimo]

I find it somehow fishy :fish: that the test only take around 5s, it takes longer on my local machine.

However, the TESTING_DEPTH env var is defined.

[kartikeyakirar]

@averissimo I have not investigated the issue, but I am receiving a warning message. Can you check this?

[averissimo]

Very weird @kartikeyakirar