search

rstudio / shinycoreci-apps

SUPERSEDED - Apps for Shiny continuous integration testing
Other
36 stars 4 forks source link

GHA: Mac-R4.1's minor font discrepancy test failures #191

Open MadhulikaTanuboddi opened 2 years ago

MadhulikaTanuboddi commented 2 years ago

For almost a week, we have been noticing that most of the tests fromMac 4.1.1 build are failing due to very subtle font styling changes. check the following shinytest snapshots

mac-4 1-google_fonts_Plots_x-axis mac-4 1-fonts-download-plots-x-axis mac-4 1-fonts mac-4 1-301-bs-themes

I noticed that when the build started failing, the Mac OS we are using in the GHA virtual environment is 10.15.17 whereas when it passed around 10 days ago, the virtual environment was 11.6 (I noticed that until Oct 19, with Mac 4.1 builds, we did not see the font download issues when the GHA was using Mac OS 11.6 (Big sur) --> https://github.com/rstudio/shinycoreci-apps/runs/3936390773?check_suite_focus=true)

If you notice a pattern from the above screenshots, most of the tests are failing are mostly with plots and in addition with x-axis values/text font diff. Not sure if this is a relevant information though.

Mac-4 1 1-build-virtual_environment mac4 1 1-passing-build_Oct_19 Mac-4 1 1-failing-builds_10 15

It is also surprising that only Mac R4.1 is being affected by this and not the other Mac build (R4.0, R3.6, 3.5 and 3.4).

Here is just a quick log when I ranshinycoreci::fix_all_gha_branches command on my R console

Apps:
* 001-hello                           : mytest     ; macOS-4.1
* 004-mpg                             : mytest     ; macOS-4.1
* 006-tabsets                         : mytest     ; macOS-4.1
* 014-onflushed                       : mytest     ; macOS-4.1
* 026-shiny-inline                    : mytest     ; macOS-4.1
* 027-absolutely-positioned-panels    : mytest     ; macOS-4.1
* 033-conditionalpanel-demo           : mytest     ; macOS-4.1
* 050-kmeans-example                  : mytest     ; macOS-4.1
* 054-nvd3-line-chart-output          : mytest     ; macOS-4.1
* 062-submitbutton-demo               : mytest     ; macOS-4.1
* 063-superzip-example                : mytest     ; macOS-4.1, Wndws-3.5
* 108-module-output                   : mytest     ; macOS-4.1
* 117-shinythemes                     : mytest     ; Wndws-3.5
* 119-namespaced-conditionalpanel-demo: mytest     ; macOS-4.1
* 122-async-outputs                   : mytest     ; macOS-4.1
* 128-plot-dim-error                  : mytest     ; macOS-4.1
* 130-output-null                     : mytest     ; macOS-4.1
* 131-renderplot-args                 : mytest     ; macOS-4.1
* 136-plot-cache                      : mytest     ; macOS-4.1
* 137-plot-cache-key                  : mytest     ; macOS-4.1
* 143-async-plot-caching              : mytest     ; macOS-4.1
* 149-onRender                        : mytest     ; macOS-4.1
* 192-reactlog-hello                  : mytest     ; macOS-4.1
* 194-plot-cache-key                  : mytest     ; macOS-4.1
* 213-tab-panels                      : mytest     ; macOS-3.4
* 216-quosures                        : mytest     ; macOS-4.1
* 301-bs-themes                       : custom4    ; macOS-4.1
* 301-bs-themes                       : default3   ; macOS-4.1
* 301-bs-themes                       : default4   ; macOS-4.1
* 302-bootswatch-themes               : cerulean4  ; macOS-4.1
* 302-bootswatch-themes               : cosmo4     ; macOS-4.1
* 302-bootswatch-themes               : cyborg4    ; macOS-4.1
* 302-bootswatch-themes               : darkly4    ; macOS-4.1
* 302-bootswatch-themes               : journal4   ; macOS-4.1
* 302-bootswatch-themes               : litera4    ; macOS-4.1
* 302-bootswatch-themes               : lumen4     ; macOS-4.1
* 302-bootswatch-themes               : lux4       ; macOS-4.1
* 302-bootswatch-themes               : materia4   ; macOS-4.1
* 302-bootswatch-themes               : minty4     ; macOS-4.1
* 302-bootswatch-themes               : pulse4     ; macOS-4.1
* 302-bootswatch-themes               : sandstone4 ; macOS-4.1
* 302-bootswatch-themes               : simplex4   ; macOS-4.1
* 302-bootswatch-themes               : sketchy4   ; macOS-4.1
* 302-bootswatch-themes               : slate4     ; macOS-4.1
* 302-bootswatch-themes               : solar4     ; macOS-4.1
* 302-bootswatch-themes               : spacelab4  ; macOS-4.1
* 302-bootswatch-themes               : superhero4 ; macOS-4.1
* 302-bootswatch-themes               : yeti4      ; macOS-4.1

https://user-images.githubusercontent.com/5369655/139123523-23208ea0-c3bd-4830-80ad-8a98ead194c8.mp4

MadhulikaTanuboddi commented 2 years ago

The changes are very subtle that they could be accepted. But, I want to understand the reason for these consistent failures (i mean if they are due to the Mac OS downgrade from 11.6 to 10.15 and R4.1 with Mac?)

MadhulikaTanuboddi commented 2 years ago

Thanks to @schloerke. Ideally, all the times, the Mac runs should be running with 10.15. Because it is listed as the macos-latest as per https://github.com/actions/virtual-environments. So, it is surprising that few builds ago on the tests even ran on Mac 11.6. Looks like they are testing and slowly rolling-over the changes. https://github.com/actions/virtual-environments/issues/4060

We originally thought we could set the macos-latest to 11.6 via https://github.com/rstudio/shinycoreci-apps/pull/192. But, since it will be default latest version anyways from Nov 3, we decided to wait. That means, the test will continue to be flaky until then.

MadhulikaTanuboddi commented 2 years ago

Last week, we merged https://github.com/rstudio/shinycoreci-apps/pull/192. Because, from https://github.com/actions/virtual-environments/issues/4060, it looks like the latest version would still be 10.15 until mid-December