rstudio / shinytest2

https://rstudio.github.io/shinytest2/
Other
106 stars 18 forks source link

Snapshot of `file` to '...' has changed #281

Open nick-youngblut opened 2 years ago

nick-youngblut commented 2 years ago

I created a simple snapshot test:

library(shinytest2)

test_that("{shinytest2} recording: startup", {
  app <- AppDriver$new(variant = platform_variant(), name = "startup", seed = 3812, 
      height = 976, width = 1619)
  app$expect_screenshot()
})

...which passes when I run shinytest2::test_app() (mac-OS 12.6). However, my GitHub action on macos-latest fails:

✔ | F W S  OK | Context
    date, intersect, setdiff, union

⠏ |         0 | shinytest2                                                      
⠋ | 1       0 | shinytest2                                                      
✖ | 1       0 | shinytest2 [11.0s]
────────────────────────────────────────────────────────────────────────────────
Failure (test-shinytest2.R:7): {shinytest2} recording: startup
Snapshot of `file` to 'shinytest2/startup-001.png' has changed
Error: Error: Test failures
Run `testthat::snapshot_review('shinytest2/')` to review changes
Execution halted
Backtrace:
 1. app$expect_screenshot()
      at test-shinytest2.R:7:2
 2. shinytest2:::app_expect_screenshot_and_variant(...)
 3. shinytest2:::app_expect_screenshot(self, private, ...)
 4. shinytest2:::app__expect_snapshot_file(...)
 6. testthat::expect_snapshot_file(...)

My check-app.yaml file:

on:
  push:
    branches: [main, master]
  pull_request:
    branches: [main, master]

name: my_shiny_app

jobs:
  test-app:
    runs-on: ${{ matrix.os }}
    name: ${{ matrix.os }} (${{ matrix.r }})
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, macos-latest]
        r: [4.2.1]
    env:
      GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}
      R_KEEP_PKG_SOURCE: yes
    steps:
      - uses: actions/checkout@v2
      - uses: r-lib/actions/setup-pandoc@v2
      - uses: r-lib/actions/setup-r@v2
        with:
          r-version: ${{ matrix.r }}
          http-user-agent: ${{ matrix.config.http-user-agent }}
          use-public-rspm: true
      - uses: r-lib/actions/setup-renv@v2
      - uses: rstudio/shinytest2/actions/test-app@v1
        with:
          path: "."

The CI test OS: macOS 12.6.1

If I had to guess on what the problem is, I would bet on the png width slightly changing. It appears that the png width doesn't stay constant in all conditions, even though width= is set via AppDriver$new()

schloerke commented 1 year ago

Related: https://github.com/rstudio/shinytest2/issues/171