unstable behavior of expect_equal on two numeric arrays depending on `tolerance` value

[qmarcou]

Hi, I've just stumbled upon an issue I can't diagnose. testthat::expect_equal unexpectedly returns "Error: act (actual) not equal to exp (expected). actual != expected but don't know how to show the difference", for some tolerance values, and displays errors in another format for other tolerance values. Below .14 of tolerance I get an error, and below .0098 I get a correctly displayed error.

Here is a MWE:

act <- c(1.728489e-05, 1.728662e-01, 2.230096e-01)
exp <- c(1.020453e-06, 1.694780e-01, 2.211992e-01)

testthat::expect_equal(
              object = act,
              expected = exp,
              tolerance = .01
            )
# Error: `act` (`actual`) not equal to `exp` (`expected`).
# actual != expected but don't know how to show the difference

testthat::expect_equal(
          object = act,
          expected = exp,
          tolerance = .014
        )
# no error

testthat::expect_equal(
          object = act,
          expected = exp,
          tolerance = .0098
        )
# Error: `act` (`actual`) not equal to `exp` (`expected`).

#  `actual`: 0.000 0.173 0.223
#`expected`: 0.000 0.169 0.221

Is it due to a switch between relative vs absolute tolerance mentioned in the docs? I would have expected to have the same error report for any too strict tolerance, and the first error is not informative.

Thanks for your help and this great package!

sessioninfo::session_info()
─ Session info ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 setting  value
 version  R version 4.3.3 (2024-02-29)
 os       Ubuntu 22.04.4 LTS
 system   x86_64, linux-gnu
 ui       X11
 language (EN)
 collate  en_US.UTF-8
 ctype    en_US.UTF-8
 tz       Europe/Paris
 date     2024-04-19
 pandoc   2.9.2.1 @ /usr/bin/pandoc

─ Packages ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 package     * version date (UTC) lib source
 brio          1.1.4   2023-12-10 [1] CRAN (R 4.3.3)
 cli           3.6.2   2023-12-11 [1] CRAN (R 4.3.3)
 crayon        1.5.2   2022-09-29 [1] CRAN (R 4.3.3)
 desc          1.4.3   2023-12-10 [1] CRAN (R 4.3.3)
 diffobj       0.3.5   2021-10-05 [1] CRAN (R 4.3.3)
 fansi         1.0.6   2023-12-08 [1] CRAN (R 4.3.3)
 glue          1.7.0   2024-01-09 [1] CRAN (R 4.3.3)
 jsonlite      1.8.8   2023-12-04 [1] CRAN (R 4.3.3)
 lifecycle     1.0.4   2023-11-07 [1] CRAN (R 4.3.3)
 magrittr      2.0.3   2022-03-30 [1] CRAN (R 4.3.3)
 pillar        1.9.0   2023-03-22 [1] CRAN (R 4.3.3)
 pkgconfig     2.0.3   2019-09-22 [1] CRAN (R 4.3.3)
 pkgload       1.3.4   2024-01-16 [1] CRAN (R 4.3.3)
 R6            2.5.1   2021-08-19 [1] CRAN (R 4.3.3)
 rematch2      2.1.2   2020-05-01 [1] CRAN (R 4.3.3)
 rlang         1.1.3   2024-01-10 [1] CRAN (R 4.3.3)
 rprojroot     2.0.4   2023-11-05 [1] CRAN (R 4.3.3)
 sessioninfo   1.2.2   2021-12-06 [1] CRAN (R 4.3.3)
 testthat      3.2.1.1 2024-04-14 [1] CRAN (R 4.3.3)
 tibble        3.2.1   2023-03-20 [1] CRAN (R 4.3.3)
 utf8          1.2.4   2023-10-22 [1] CRAN (R 4.3.3)
 vctrs         0.6.5   2023-12-01 [1] CRAN (R 4.3.3)
 waldo         0.5.2   2023-11-02 [1] CRAN (R 4.3.3)

[qmarcou]

Ok this actually seems to be a waldo related issue:

waldo::compare(
                  act,
                  exp,
                  tolerance = .01)

# old != new but don't know how to show the difference

waldo::compare(
                  act,
                  exp,
                  tolerance = .014)
# ✔ No differences

waldo::compare(
                  act,
                  exp,
                  tolerance = .0098)
# `old`: 0.000 0.173 0.223
# `new`: 0.000 0.169 0.221

A similar is open there, see r-lib/waldo#190. On a not too dissimilar theme I think r-lib/waldo#188 is also an issue for using testthat reliably.

I leave this issue open hoping it will help the ones open on waldo's repo being picked up.

[hadley]

Root issue is https://github.com/r-lib/waldo/issues/187, I think.