search

gavinsimpson / gratia

ggplot-based graphics and useful functions for GAMs fitted using the mgcv package
https://gavinsimpson.github.io/gratia/
Other
206 stars 28 forks source link

JOSS review: failing tests #303

Closed dill closed 3 days ago

dill commented 3 months ago

Running tests for the package on a Debian sid system (details at the bottom), I get failures on visual diffs of the plots.

I'm not familiar with vdiffr so not sure if this is expected or not?

Looking at these, I don't see any differences in some plots (see below also). I thought it was an mgcv/scam/ggplot2 version issue but re-installed from CRAN and got the following:

✔ | F W  S  OK | Context
✔ |         82 | add-functions [2.4s]                                           
✔ |          4 | bam-ar                                                         
✔ |          4 | basis-size                                                     
✖ | 2    1  29 | basis [6.1s]                                                   
────────────────────────────────────────────────────────────────────────────────
Failure (test-basis.R:83:3): basis() works with bivariate tprs smooths
Snapshot of `testcase` to 'basis/draw-basis-works-with-a-bivariate-tprs-contour.svg' has changed
Run `testthat::snapshot_review('basis/')` to review changes
Backtrace:
    ▆
 1. └─gratia:::expect_doppelganger(...) at test-basis.R:83:3
 2.   └─vdiffr::expect_doppelganger(title, fig, ...) at tests/testthat/helper-vdiffr.R:39:5
 3.     ├─base::withCallingHandlers(...)
 4.     └─testthat::expect_snapshot_file(...)

Failure (test-basis.R:104:3): basis() works with bivariate te smooths
Snapshot of `testcase` to 'basis/draw-basis-works-with-a-bivariate-te-contour.svg' has changed
Run `testthat::snapshot_review('basis/')` to review changes
Backtrace:
    ▆
 1. └─gratia:::expect_doppelganger(...) at test-basis.R:104:3
 2.   └─vdiffr::expect_doppelganger(title, fig, ...) at tests/testthat/helper-vdiffr.R:39:5
 3.     ├─base::withCallingHandlers(...)
 4.     └─testthat::expect_snapshot_file(...)
────────────────────────────────────────────────────────────────────────────────
✔ |         14 | by-variables                                                   
✖ | 3       29 | compare-smooths [3.1s]                                         
────────────────────────────────────────────────────────────────────────────────
Failure (test-compare-smooths.R:37:3): draw.compare_smooths() can plot a comparison of smooths
Snapshot of `testcase` to 'compare-smooths/compare-smooths-all-smooths.svg' has changed
Run `testthat::snapshot_review('compare-smooths/')` to review changes
Backtrace:
    ▆
 1. └─gratia:::expect_doppelganger(...) at test-compare-smooths.R:37:3
 2.   └─vdiffr::expect_doppelganger(title, fig, ...) at tests/testthat/helper-vdiffr.R:39:5
 3.     ├─base::withCallingHandlers(...)
 4.     └─testthat::expect_snapshot_file(...)

Failure (test-compare-smooths.R:38:3): draw.compare_smooths() can plot a comparison of smooths
Snapshot of `testcase` to 'compare-smooths/compare-smooths-set-nrow-ncol.svg' has changed
Run `testthat::snapshot_review('compare-smooths/')` to review changes
Backtrace:
    ▆
 1. └─gratia:::expect_doppelganger(...) at test-compare-smooths.R:38:3
 2.   └─vdiffr::expect_doppelganger(title, fig, ...) at tests/testthat/helper-vdiffr.R:39:5
 3.     ├─base::withCallingHandlers(...)
 4.     └─testthat::expect_snapshot_file(...)

Failure (test-compare-smooths.R:39:3): draw.compare_smooths() can plot a comparison of smooths
Snapshot of `testcase` to 'compare-smooths/compare-smooths-bottom-legend.svg' has changed
Run `testthat::snapshot_review('compare-smooths/')` to review changes
Backtrace:
    ▆
 1. └─gratia:::expect_doppelganger(...) at test-compare-smooths.R:39:3
 2.   └─vdiffr::expect_doppelganger(title, fig, ...) at tests/testthat/helper-vdiffr.R:39:5
 3.     ├─base::withCallingHandlers(...)
 4.     └─testthat::expect_snapshot_file(...)
────────────────────────────────────────────────────────────────────────────────
✔ |         20 | concurvity                                                     
✔ |      1  44 | confint-methods [1.1s]                                         
✔ |        100 | data-sim                                                       
✔ |         99 | data-slice                                                     
✔ |         36 | datagen                                                        
✖ | 2      222 | derivatives [7.1s]                                             
────────────────────────────────────────────────────────────────────────────────
Failure (test-derivatives.R:441:3): derivatives() works for factor by smooths issue 47
Snapshot of `testcase` to 'derivatives/draw-issue-47-derivatives-for-gamm-factor-by.svg' has changed
Run `testthat::snapshot_review('derivatives/')` to review changes
Backtrace:
    ▆
 1. └─gratia:::expect_doppelganger(...) at test-derivatives.R:441:3
 2.   └─vdiffr::expect_doppelganger(title, fig, ...) at tests/testthat/helper-vdiffr.R:39:5
 3.     ├─base::withCallingHandlers(...)
 4.     └─testthat::expect_snapshot_file(...)

Failure (test-derivatives.R:477:3): derivatives() works for fs smooths issue 57 and 301
Snapshot of `testcase` to 'derivatives/draw-issue-57-derivatives-for-factor-by.svg' has changed
Run `testthat::snapshot_review('derivatives/')` to review changes
Backtrace:
    ▆
 1. └─gratia:::expect_doppelganger(...) at test-derivatives.R:477:3
 2.   └─vdiffr::expect_doppelganger(title, fig, ...) at tests/testthat/helper-vdiffr.R:39:5
 3.     ├─base::withCallingHandlers(...)
 4.     └─testthat::expect_snapshot_file(...)
────────────────────────────────────────────────────────────────────────────────
✔ |         40 | diagnostic-plots [3.3s]                                        
✖ | 2       30 | difference-smooths [5.9s]                                      
────────────────────────────────────────────────────────────────────────────────
Failure (test-difference-smooths.R:76:3): difference_smooths() works for a gamm model
Snapshot of `testcase` to 'difference-smooths/draw-difference-smooths-gamm.svg' has changed
Run `testthat::snapshot_review('difference-smooths/')` to review changes
Backtrace:
    ▆
 1. └─gratia:::expect_doppelganger(...) at test-difference-smooths.R:76:3
 2.   └─vdiffr::expect_doppelganger(title, fig, ...) at tests/testthat/helper-vdiffr.R:39:5
 3.     ├─base::withCallingHandlers(...)
 4.     └─testthat::expect_snapshot_file(...)

Failure (test-difference-smooths.R:129:3): difference_smooths() works for a bivariate gam
Snapshot of `testcase` to 'difference-smooths/draw-difference-smooths-bivariate-gam-contours.svg' has changed
Run `testthat::snapshot_review('difference-smooths/')` to review changes
Backtrace:
    ▆
 1. └─gratia:::expect_doppelganger(...) at test-difference-smooths.R:129:3
 2.   └─vdiffr::expect_doppelganger(title, fig, ...) at tests/testthat/helper-vdiffr.R:39:5
 3.     ├─base::withCallingHandlers(...)
 4.     └─testthat::expect_snapshot_file(...)
────────────────────────────────────────────────────────────────────────────────
✔ |         12 | distribution-helpers                                           
✖ | 3       66 | draw-gam [15.8s]                                               
────────────────────────────────────────────────────────────────────────────────
Failure (test-draw-gam.R:47:3): draw.gam works for m_gamm4
Snapshot of `testcase` to 'draw-gam/draw-gam-m-gamm4.svg' has changed
Run `testthat::snapshot_review('draw-gam/')` to review changes
Backtrace:
    ▆
 1. └─gratia:::expect_doppelganger("draw_gam m_gamm4", plt) at test-draw-gam.R:47:3
 2.   └─vdiffr::expect_doppelganger(title, fig, ...) at tests/testthat/helper-vdiffr.R:39:5
 3.     ├─base::withCallingHandlers(...)
 4.     └─testthat::expect_snapshot_file(...)

Failure (test-draw-gam.R:91:3): draw.gam works for dlnm_m
Snapshot of `testcase` to 'draw-gam/draw-gam-dlnm-m.svg' has changed
Run `testthat::snapshot_review('draw-gam/')` to review changes
Backtrace:
    ▆
 1. └─gratia:::expect_doppelganger("draw_gam dlnm_m", plt) at test-draw-gam.R:91:3
 2.   └─vdiffr::expect_doppelganger(title, fig, ...) at tests/testthat/helper-vdiffr.R:39:5
 3.     ├─base::withCallingHandlers(...)
 4.     └─testthat::expect_snapshot_file(...)

Failure (test-draw-gam.R:201:3): draw.gam works for a quadvariate smooth
Snapshot of `testcase` to 'draw-gam/draw-gam-quadvar-te.svg' has changed
Run `testthat::snapshot_review('draw-gam/')` to review changes
Backtrace:
    ▆
 1. └─gratia:::expect_doppelganger("draw_gam quadvar te", plt) at test-draw-gam.R:201:3
 2.   └─vdiffr::expect_doppelganger(title, fig, ...) at tests/testthat/helper-vdiffr.R:39:5
 3.     ├─base::withCallingHandlers(...)
 4.     └─testthat::expect_snapshot_file(...)
────────────────────────────────────────────────────────────────────────────────
Maximum number of failures exceeded; quitting at end of file.
ℹ Increase this number with (e.g.) `testthat::set_max_fails(Inf)` 

══ Results ═════════════════════════════════════════════════════════════════════
Duration: 48.7 s

── Skipped tests (2) ───────────────────────────────────────────────────────────
• On Linux (1): test-confint-methods.R:199:3
• This needs fixing as something in scam changed (1): test-basis.R:53:3

── Failed tests ────────────────────────────────────────────────────────────────
Failure (test-basis.R:83:3): basis() works with bivariate tprs smooths
Snapshot of `testcase` to 'basis/draw-basis-works-with-a-bivariate-tprs-contour.svg' has changed
Run `testthat::snapshot_review('basis/')` to review changes
Backtrace:
    ▆
 1. └─gratia:::expect_doppelganger(...) at test-basis.R:83:3
 2.   └─vdiffr::expect_doppelganger(title, fig, ...) at tests/testthat/helper-vdiffr.R:39:5
 3.     ├─base::withCallingHandlers(...)
 4.     └─testthat::expect_snapshot_file(...)

Failure (test-basis.R:104:3): basis() works with bivariate te smooths
Snapshot of `testcase` to 'basis/draw-basis-works-with-a-bivariate-te-contour.svg' has changed
Run `testthat::snapshot_review('basis/')` to review changes
Backtrace:
    ▆
 1. └─gratia:::expect_doppelganger(...) at test-basis.R:104:3
 2.   └─vdiffr::expect_doppelganger(title, fig, ...) at tests/testthat/helper-vdiffr.R:39:5
 3.     ├─base::withCallingHandlers(...)
 4.     └─testthat::expect_snapshot_file(...)

Failure (test-compare-smooths.R:37:3): draw.compare_smooths() can plot a comparison of smooths
Snapshot of `testcase` to 'compare-smooths/compare-smooths-all-smooths.svg' has changed
Run `testthat::snapshot_review('compare-smooths/')` to review changes
Backtrace:
    ▆
 1. └─gratia:::expect_doppelganger(...) at test-compare-smooths.R:37:3
 2.   └─vdiffr::expect_doppelganger(title, fig, ...) at tests/testthat/helper-vdiffr.R:39:5
 3.     ├─base::withCallingHandlers(...)
 4.     └─testthat::expect_snapshot_file(...)

Failure (test-compare-smooths.R:38:3): draw.compare_smooths() can plot a comparison of smooths
Snapshot of `testcase` to 'compare-smooths/compare-smooths-set-nrow-ncol.svg' has changed
Run `testthat::snapshot_review('compare-smooths/')` to review changes
Backtrace:
    ▆
 1. └─gratia:::expect_doppelganger(...) at test-compare-smooths.R:38:3
 2.   └─vdiffr::expect_doppelganger(title, fig, ...) at tests/testthat/helper-vdiffr.R:39:5
 3.     ├─base::withCallingHandlers(...)
 4.     └─testthat::expect_snapshot_file(...)

Failure (test-compare-smooths.R:39:3): draw.compare_smooths() can plot a comparison of smooths
Snapshot of `testcase` to 'compare-smooths/compare-smooths-bottom-legend.svg' has changed
Run `testthat::snapshot_review('compare-smooths/')` to review changes
Backtrace:
    ▆
 1. └─gratia:::expect_doppelganger(...) at test-compare-smooths.R:39:3
 2.   └─vdiffr::expect_doppelganger(title, fig, ...) at tests/testthat/helper-vdiffr.R:39:5
 3.     ├─base::withCallingHandlers(...)
 4.     └─testthat::expect_snapshot_file(...)

Failure (test-derivatives.R:441:3): derivatives() works for factor by smooths issue 47
Snapshot of `testcase` to 'derivatives/draw-issue-47-derivatives-for-gamm-factor-by.svg' has changed
Run `testthat::snapshot_review('derivatives/')` to review changes
Backtrace:
    ▆
 1. └─gratia:::expect_doppelganger(...) at test-derivatives.R:441:3
 2.   └─vdiffr::expect_doppelganger(title, fig, ...) at tests/testthat/helper-vdiffr.R:39:5
 3.     ├─base::withCallingHandlers(...)
 4.     └─testthat::expect_snapshot_file(...)

Failure (test-derivatives.R:477:3): derivatives() works for fs smooths issue 57 and 301
Snapshot of `testcase` to 'derivatives/draw-issue-57-derivatives-for-factor-by.svg' has changed
Run `testthat::snapshot_review('derivatives/')` to review changes
Backtrace:
    ▆
 1. └─gratia:::expect_doppelganger(...) at test-derivatives.R:477:3
 2.   └─vdiffr::expect_doppelganger(title, fig, ...) at tests/testthat/helper-vdiffr.R:39:5
 3.     ├─base::withCallingHandlers(...)
 4.     └─testthat::expect_snapshot_file(...)

Failure (test-difference-smooths.R:76:3): difference_smooths() works for a gamm model
Snapshot of `testcase` to 'difference-smooths/draw-difference-smooths-gamm.svg' has changed
Run `testthat::snapshot_review('difference-smooths/')` to review changes
Backtrace:
    ▆
 1. └─gratia:::expect_doppelganger(...) at test-difference-smooths.R:76:3
 2.   └─vdiffr::expect_doppelganger(title, fig, ...) at tests/testthat/helper-vdiffr.R:39:5
 3.     ├─base::withCallingHandlers(...)
 4.     └─testthat::expect_snapshot_file(...)

Failure (test-difference-smooths.R:129:3): difference_smooths() works for a bivariate gam
Snapshot of `testcase` to 'difference-smooths/draw-difference-smooths-bivariate-gam-contours.svg' has changed
Run `testthat::snapshot_review('difference-smooths/')` to review changes
Backtrace:
    ▆
 1. └─gratia:::expect_doppelganger(...) at test-difference-smooths.R:129:3
 2.   └─vdiffr::expect_doppelganger(title, fig, ...) at tests/testthat/helper-vdiffr.R:39:5
 3.     ├─base::withCallingHandlers(...)
 4.     └─testthat::expect_snapshot_file(...)

Failure (test-draw-gam.R:47:3): draw.gam works for m_gamm4
Snapshot of `testcase` to 'draw-gam/draw-gam-m-gamm4.svg' has changed
Run `testthat::snapshot_review('draw-gam/')` to review changes
Backtrace:
    ▆
 1. └─gratia:::expect_doppelganger("draw_gam m_gamm4", plt) at test-draw-gam.R:47:3
 2.   └─vdiffr::expect_doppelganger(title, fig, ...) at tests/testthat/helper-vdiffr.R:39:5
 3.     ├─base::withCallingHandlers(...)
 4.     └─testthat::expect_snapshot_file(...)

Failure (test-draw-gam.R:91:3): draw.gam works for dlnm_m
Snapshot of `testcase` to 'draw-gam/draw-gam-dlnm-m.svg' has changed
Run `testthat::snapshot_review('draw-gam/')` to review changes
Backtrace:
    ▆
 1. └─gratia:::expect_doppelganger("draw_gam dlnm_m", plt) at test-draw-gam.R:91:3
 2.   └─vdiffr::expect_doppelganger(title, fig, ...) at tests/testthat/helper-vdiffr.R:39:5
 3.     ├─base::withCallingHandlers(...)
 4.     └─testthat::expect_snapshot_file(...)

Failure (test-draw-gam.R:201:3): draw.gam works for a quadvariate smooth
Snapshot of `testcase` to 'draw-gam/draw-gam-quadvar-te.svg' has changed
Run `testthat::snapshot_review('draw-gam/')` to review changes
Backtrace:
    ▆
 1. └─gratia:::expect_doppelganger("draw_gam quadvar te", plt) at test-draw-gam.R:201:3
 2.   └─vdiffr::expect_doppelganger(title, fig, ...) at tests/testthat/helper-vdiffr.R:39:5
 3.     ├─base::withCallingHandlers(...)
 4.     └─testthat::expect_snapshot_file(...)

[ FAIL 12 | WARN 0 | SKIP 2 | PASS 831 ]
══ Terminated early ════════════════════════════════════════════════════════════
>

Example "bad" plots:

old: draw-gam-m-gamm4

new: draw-gam-m-gamm4 new

another with an actual difference

old: draw-gam-dlnm-m

new: draw-gam-dlnm-m new

Finally, some information, about my session (post-running devtools::test()):

> session_info()
─ Session info ───────────────────────────────────────────────────────────────
 setting  value
 version  R version 4.4.1 (2024-06-14)
 os       Debian GNU/Linux 12 (bookworm)
 system   x86_64, linux-gnu
 ui       X11
 language en_GB:en
 collate  en_GB.UTF-8
 ctype    en_GB.UTF-8
 tz       Europe/London
 date     2024-07-29
 pandoc   2.17.1.1 @ /usr/bin/pandoc

─ Packages ───────────────────────────────────────────────────────────────────
 ! package      * version    date (UTC) lib source
   boot           1.3-30     2024-02-26 [1] CRAN (R 4.4.0)
   brio           1.1.5      2024-04-24 [1] CRAN (R 4.4.0)
   cachem         1.0.8      2023-05-01 [1] CRAN (R 4.4.0)
   cli            3.6.3      2024-06-21 [1] CRAN (R 4.4.1)
   colorspace     2.1-1      2024-07-26 [1] CRAN (R 4.4.1)
   desc           1.4.3      2023-12-10 [1] CRAN (R 4.4.0)
   devtools     * 2.4.5      2022-10-11 [1] CRAN (R 4.4.0)
   digest         0.6.35     2024-03-11 [1] CRAN (R 4.4.0)
   dplyr        * 1.1.4      2023-11-17 [1] CRAN (R 4.4.0)
   ellipsis       0.3.2      2021-04-29 [1] CRAN (R 4.4.0)
   evaluate       0.23       2023-11-01 [1] CRAN (R 4.4.0)
   fansi          1.0.6      2023-12-08 [1] CRAN (R 4.4.0)
   farver         2.1.2      2024-05-13 [1] CRAN (R 4.4.0)
   fastmap        1.1.1      2023-02-24 [1] CRAN (R 4.4.0)
   fs             1.6.4      2024-04-25 [1] CRAN (R 4.4.0)
   gamm4        * 0.2-6      2020-04-03 [1] CRAN (R 4.4.0)
   generics       0.1.3      2022-07-05 [1] CRAN (R 4.4.0)
   ggokabeito     0.1.0      2021-10-18 [1] CRAN (R 4.4.0)
   ggplot2      * 3.5.1      2024-04-23 [1] CRAN (R 4.4.1)
   glue           1.7.0      2024-01-09 [1] CRAN (R 4.4.0)
 P gratia       * 0.9.2.9005 2024-07-29 [?] Github (gavinsimpson/gratia@d7c32c3)
   gtable         0.3.5      2024-04-22 [1] CRAN (R 4.4.0)
   htmltools      0.5.8.1    2024-04-04 [1] CRAN (R 4.4.0)
   htmlwidgets    1.6.4      2023-12-06 [1] CRAN (R 4.4.0)
   httpuv         1.6.15     2024-03-26 [1] CRAN (R 4.4.0)
   isoband        0.2.7      2022-12-20 [1] CRAN (R 4.4.0)
   labeling       0.4.3      2023-08-29 [1] CRAN (R 4.4.0)
   later          1.3.2      2023-12-06 [1] CRAN (R 4.4.0)
   lattice        0.22-6     2024-03-20 [1] CRAN (R 4.4.0)
   lifecycle      1.0.4      2023-11-07 [1] CRAN (R 4.4.0)
   lme4         * 1.1-35.3   2024-04-16 [1] CRAN (R 4.4.0)
   magrittr       2.0.3      2022-03-30 [1] CRAN (R 4.4.0)
   MASS           7.3-60.2   2024-04-26 [1] CRAN (R 4.4.0)
   Matrix       * 1.7-0      2024-04-26 [1] CRAN (R 4.4.0)
   memoise        2.0.1      2021-11-26 [1] CRAN (R 4.4.0)
   mgcv         * 1.9-1      2023-12-21 [1] CRAN (R 4.4.1)
   mime           0.12       2021-09-28 [1] CRAN (R 4.4.0)
   miniUI         0.1.1.1    2018-05-18 [1] CRAN (R 4.4.0)
   minqa          1.2.6      2023-09-11 [1] CRAN (R 4.4.0)
   munsell        0.5.1      2024-04-01 [1] CRAN (R 4.4.0)
   mvnfast        0.2.8      2023-02-23 [1] CRAN (R 4.4.0)
   nlme         * 3.1-164    2023-11-27 [1] CRAN (R 4.4.0)
   nloptr         2.0.3      2022-05-26 [1] CRAN (R 4.4.0)
   patchwork      1.2.0      2024-01-08 [1] CRAN (R 4.4.0)
   pillar         1.9.0      2023-03-22 [1] CRAN (R 4.4.0)
   pkgbuild       1.4.4      2024-03-17 [1] CRAN (R 4.4.0)
   pkgconfig      2.0.3      2019-09-22 [1] CRAN (R 4.4.0)
   pkgload        1.3.4      2024-01-16 [1] CRAN (R 4.4.0)
   profvis        0.3.8      2023-05-02 [1] CRAN (R 4.4.0)
   promises       1.3.0      2024-04-05 [1] CRAN (R 4.4.0)
   purrr          1.0.2      2023-08-10 [1] CRAN (R 4.4.0)
   R6             2.5.1      2021-08-19 [1] CRAN (R 4.4.0)
   RColorBrewer   1.1-3      2022-04-03 [1] CRAN (R 4.4.0)
   Rcpp           1.0.13     2024-07-17 [1] CRAN (R 4.4.1)
   remotes        2.5.0      2024-03-17 [1] CRAN (R 4.4.0)
   rlang          1.1.4      2024-06-04 [1] CRAN (R 4.4.0)
   rprojroot      2.0.4      2023-11-05 [1] CRAN (R 4.4.0)
   rstudioapi     0.16.0     2024-03-24 [1] CRAN (R 4.4.0)
   scales         1.3.0      2023-11-28 [1] CRAN (R 4.4.0)
   scam         * 1.2-17     2024-06-19 [1] CRAN (R 4.4.1)
   sessioninfo    1.2.2      2021-12-06 [1] CRAN (R 4.4.0)
   shiny          1.8.1.1    2024-04-02 [1] CRAN (R 4.4.0)
   stringi        1.8.4      2024-05-06 [1] CRAN (R 4.4.0)
   stringr        1.5.1      2023-11-14 [1] CRAN (R 4.4.0)
   testthat     * 3.2.1.1    2024-04-14 [1] CRAN (R 4.4.0)
   tibble       * 3.2.1      2023-03-20 [1] CRAN (R 4.4.0)
   tidyr          1.3.1      2024-01-24 [1] CRAN (R 4.4.0)
   tidyselect     1.2.1      2024-03-11 [1] CRAN (R 4.4.0)
   urlchecker     1.0.1      2021-11-30 [1] CRAN (R 4.4.0)
   usethis      * 2.2.3      2024-02-19 [1] CRAN (R 4.4.0)
   utf8           1.2.4      2023-10-22 [1] CRAN (R 4.4.0)
   vctrs          0.6.5      2023-12-01 [1] CRAN (R 4.4.0)
   vdiffr         1.0.7      2023-09-22 [1] CRAN (R 4.4.0)
   viridisLite    0.4.2      2023-05-02 [1] CRAN (R 4.4.0)
   waldo          0.5.2      2023-11-02 [1] CRAN (R 4.4.0)
   withr          3.0.0      2024-01-16 [1] CRAN (R 4.4.0)
   xtable         1.8-4      2019-04-21 [1] CRAN (R 4.4.0)

 [1] /home/dill/R/x86_64-pc-linux-gnu-library/4.4
 [2] /usr/local/lib/R/site-library
 [3] /usr/lib/R/site-library
 [4] /usr/lib/R/library

 P ── Loaded and on-disk path mismatch.

──────────────────────────────────────────────────────────────────────────────
>
gavinsimpson commented 3 months ago

This is a bit of an infelicity with vdiffr; despite using SVG for the comparisons (so it's all just plain text comparisons), a myriad of system differences can cause minor differences (or major ones - sign differences on eigenvectors is one relevant example because I use thin plate splines in a lot of the tests). As such, these tests are most useful for me running tests locally to identify when something changes rather than for general use. However, you can't just do that with R packages (unless I shipped the package with no tests on CRAN). On GitHub I have managed to skip tests that cause problems on Ubuntu, Windows, or their macOS X runners, and likewise for CRAN, but I do this through skip_on_ci() and skip_on_cran() and a few skip_on_os() calls, but I suspect a lot of the skipped tests on CI are failing on Debian for trivial reasons that would need me to add skip_on_os("linux").

The first plot comparison you show is a typical example of a failing test that will more than likely be in a decimal place of a coordinate of one of the graphical elements that is just not visible. The second plot comparison you show is most likely differences in the underlying fitted GAM between Debian on an Intel CPU (if you are running this on the laptop you had at ISEC) and my M3 Mac. There's not much I can about either of these kinds of failures.

I could perhaps do something more in https://github.com/gavinsimpson/gratia/blob/d7c32c3e52d152006c7db60eeb03b9f640bc2aa0/tests/testthat/helper-vdiffr.R to simply stop any vdiffr tests from running unless they're on my system (conditional on the system not being CRAN or a CI) but I'm not sure how that would affect checks on CRAN's winbuilder service and Simon Urbanek's MacOS X check server.

dill commented 3 months ago

Aha! No worries, I was unaware of this issue! That seems like a totally reasonable approach to me given the constraints. I'm happy for this issue to be closed for the purposes of the review.