Difference in json file generation between devtools::test() and devtools::check()

[LamaTe]

Hi, thanks for this nice package. I am working on a shiny app in form of a package which I normally test using shinytest combined with testthat. I have a minimalistic shiny test (to investigate the error) consisting of a single screenshot right after starting the app through recordTest(). The test passes when running testApp(), devtools::test() etc.

However, I ran into a weird error which only occurs when running devtools::check(), always failing the test: The json file that is generated suddenly varies but not in terms of 'actual' values but in terms of ordering - i.e. certain inputs appear in a different order than the one of the recorded test. Hence, the test 'fails' and devtools::check() does not pass.

Attached, you can find the expected test and the current test json files - They only vary in their ordering of input values. expected_and_current_json.zip

I am out of ideas as to what causes this behaviour - any hint/help is greatly appreciated! Thanks in advance!

[schloerke]

@LamaTe & @bersbersbers Could I have the output of Sys.getlocale() and sessioninfo::session_info() from your testing machine with shinytest and shiny loaded? My local: "en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8"

The first two values correspond to LC_COLLATE and LC_CTYPE. These values are responsible for sorting order.

Upstream, the values from Shiny have already been sorted here: https://github.com/rstudio/shiny/commit/cfc0194c0072211d473f6ce14123d698b0cd84d1 .

Looking at the diff in the attached json files, the sorting is done, but the rules are different.

Using the keys in your example for the example below, I can recreate the out-of-order sorting:

x <- c("Data_train_csv", "Data_train_header", "Data_train_header_xlsx",
"Data_train_quote", "Data_train_sep", "Data_train_sheet", "Data_train_type",
"Data_train_xlsx", "Help", "Learner_Learners_Tab", "Learner_add",
"Predict_data_csv", "Predict_data_header", "Predict_data_header_xlsx",
"Predict_data_quote", "Predict_data_sep", "Predict_data_sheet",
"Predict_data_type", "Predict_data_xlsx", "Predict_predict",
"Task_backend", "TrainFit_Base", "TrainFit_resample", "navbar")

# 'navbar' is in the middle
sort(x)
#>  [1] "Data_train_csv"           "Data_train_header"       
#>  [3] "Data_train_header_xlsx"   "Data_train_quote"        
#>  [5] "Data_train_sep"           "Data_train_sheet"        
#>  [7] "Data_train_type"          "Data_train_xlsx"         
#>  [9] "Help"                     "Learner_add"             
#> [11] "Learner_Learners_Tab"     "navbar"                  
#> [13] "Predict_data_csv"         "Predict_data_header"     
#> [15] "Predict_data_header_xlsx" "Predict_data_quote"      
#> [17] "Predict_data_sep"         "Predict_data_sheet"      
#> [19] "Predict_data_type"        "Predict_data_xlsx"       
#> [21] "Predict_predict"          "Task_backend"            
#> [23] "TrainFit_Base"            "TrainFit_resample"

# 'navbar' is in the middle
withr::with_locale(list(LC_CTYPE = "en_US.UTF-8", LC_COLLATE = "en_US.UTF-8"), sort(x))
#>  [1] "Data_train_csv"           "Data_train_header"       
#>  [3] "Data_train_header_xlsx"   "Data_train_quote"        
#>  [5] "Data_train_sep"           "Data_train_sheet"        
#>  [7] "Data_train_type"          "Data_train_xlsx"         
#>  [9] "Help"                     "Learner_add"             
#> [11] "Learner_Learners_Tab"     "navbar"                  
#> [13] "Predict_data_csv"         "Predict_data_header"     
#> [15] "Predict_data_header_xlsx" "Predict_data_quote"      
#> [17] "Predict_data_sep"         "Predict_data_sheet"      
#> [19] "Predict_data_type"        "Predict_data_xlsx"       
#> [21] "Predict_predict"          "Task_backend"            
#> [23] "TrainFit_Base"            "TrainFit_resample"

# 'navbar' is at the end
withr::with_locale(list(LC_CTYPE = "C", LC_COLLATE = "C"), sort(x))
#>  [1] "Data_train_csv"           "Data_train_header"       
#>  [3] "Data_train_header_xlsx"   "Data_train_quote"        
#>  [5] "Data_train_sep"           "Data_train_sheet"        
#>  [7] "Data_train_type"          "Data_train_xlsx"         
#>  [9] "Help"                     "Learner_Learners_Tab"    
#> [11] "Learner_add"              "Predict_data_csv"        
#> [13] "Predict_data_header"      "Predict_data_header_xlsx"
#> [15] "Predict_data_quote"       "Predict_data_sep"        
#> [17] "Predict_data_sheet"       "Predict_data_type"       
#> [19] "Predict_data_xlsx"        "Predict_predict"         
#> [21] "Task_backend"             "TrainFit_Base"           
#> [23] "TrainFit_resample"        "navbar"

^{Created on 2021-09-29 by the reprex package (v2.0.0)}

I can't determine where the new locale value is being set. Are you by chance manually setting your locale value in your .Rprofile?

---

Maybe shiny should use something like devtools:::sort_ci() where sorting is done under a consistent sort order. https://github.com/r-lib/devtools/blob/d5901f8b1b986cfc623a3615e511015ad468ab0f/R/utils.R#L23-L25

[schloerke]

Please try using remotes::install_github("rstudio/shiny#3515") (and restart your R session) to see if the bug is fixed. Thank you!

[bersbersbers]

@LamaTe & @bersbersbers Could I have the output of Sys.getlocale() from your testing machine? Mine: "en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8"

Sure!

"LC_CTYPE=en_US.UTF-8;LC_NUMERIC=C;LC_TIME=en_US.UTF-8;LC_COLLATE=en_US.UTF-8;LC_MONETARY=en_US.UTF-8;LC_MESSAGES=en_US.UTF-8;LC_PAPER=en_US.UTF-8;LC_NAME=C;LC_ADDRESS=C;LC_TELEPHONE=C;LC_MEASUREMENT=en_US.UTF-8;LC_IDENTIFICATION=C"

Looking at the diff in the attached json files, the sorting is done, but the rules are different.

Yes, I can reproduce your case.

I am not sure this is the exact same problem that I see, but it's definitely related. The keys that are involved for me are (simplified) x <- c("Xc_height", "X_col", "Xc_width", "X_row") # current and y <- c("X_col", "X_row", "Xc_height", "Xc_width") # expected You can see how x is sorted when ignoring underscores, while y is sorted as if "_" < "c". But I fail to make them actually sort differently in R using the locale.

I can't determine where the new locale value is being set. Are you by chance manually setting your locale value in your .Rprofile?

In my case, I have at least an idea. I had someone run tests on their German Mac and noticed that tests were failing because messages were translated (English expected, German current). So I was looking for a portable way to set the message language, which I haven't found except through locales. I settled on this - note that LC_MESSAGES does not exist on Windows, available locales are generally different between platforms; and I did not know C at the time:

  for (category in c("LC_ALL", "LC_MESSAGES")) {
    for (locale in c("en_US.UTF-8", "en_US.utf8", "en_US", "en", "English")) {
      if (suppressWarnings(Sys.setlocale(category, locale)) != "") {
        break
      }
    }
  }

This is code that I run in some set_env() which I run within app.R, which consists solely of this:

mypackage:::set_env()
shinyApp(mypackage:::package_ui(), mypackage:::package_server)

And then there is mypackage::run_tests which basically does shinytest::testApp(system.file("shiny", package = "mypackage")) and is annotated with @examples run_tests(testnames = c("tabs_home")). This is how R CMD check gets to run this code.

Edit: Still, I don't see why devtools::check() would work any different from any other way of starting shinytest. Maybe it's not only locales that influence sort order?

2nd edit: This seems to be true, it can be platform-dependent, among other things:

Some platforms may not respect the locale and always sort in numerical order of the bytes in an 8-bit locale, or in Unicode code-point order for a UTF-8 locale (and may not sort in the same order for the same language in different character sets).

https://www.rdocumentation.org/packages/base/versions/3.6.2/topics/Comparison

I hope this describes my issue sufficiently well. I can of course experiment further with setting fewer locales (such as not setting LC_ALL when LC_MESSAGES is available) and/or try to convert my app into a reprex, but you can image this is not a 5-minute thing :) And there's rstudio/shiny#3515 that I can test.

Maybe shiny should use something like devtools:::sort_ci() where sorting is done under a consistent sort order. https://github.com/r-lib/devtools/blob/d5901f8b1b986cfc623a3615e511015ad468ab0f/R/utils.R#L23-L25

This would make sense, I guess.

[bersbersbers]

It seems like rstudio/shiny#3515 does not fix the issue for me. Regenerating all tests as usual does not change anything (this is expected - I am setting the same locale already). Running tests within devtools::check() reproduces the same incorrect order I saw before, underlining my point that sort order is influenced not only by the locale. The question is, what - and why is it different in check()?

[bersbersbers]

Hmmm... I wonder if withr::with_collate() is doing what we are thinking it does:

x <- c("X_col", "X_row", "Xc_height", "Xc_width")
> withr::with_collate("C", order(x))
[1] 1 2 3 4
> withr::with_collate("en_US.UTF-8", order(x))
[1] 1 2 3 4

By contrast:

$ printf "X_col\nX_row\nXc_height\nXc_width\n" | LC_COLLATE=C sort
X_col
X_row
Xc_height
Xc_width

$ printf "X_col\nX_row\nXc_height\nXc_width\n" | LC_COLLATE=en_US.UTF-8 sort
Xc_height
X_col
Xc_width
X_row

[wch]

Here's what I get when I run the same commands from the terminal. The results of sort are the same in both cases.

$  printf "X_col\nX_row\nXc_height\nXc_width\n" | LC_COLLATE=C sort
X_col
X_row
Xc_height
Xc_width

$ printf "X_col\nX_row\nXc_height\nXc_width\n" | LC_COLLATE=en_US.UTF-8 sort
X_col
X_row
Xc_height
Xc_width

In other words, on my computer (US English Mac), LC_COLLATE does not affect how _ and c are sorted.

The locale is respected from R when using accented letters:

> x <- c("a", "A", "å", "b", "B")
> withr::with_collate("C", sort(x))
[1] "A" "B" "a" "b" "å"
> withr::with_collate("en_US.UTF-8", sort(x))
[1] "a" "A" "å" "b" "B"

One of the mysteries for me is why your computer changes the sorting of _ and c depending on locale.

For reference, here is my info after running the code example above (using withr):

> sessioninfo::session_info()
─ Session info ───────────────────────────────────────────────────────────────
 setting  value                       
 version  R version 4.1.1 (2021-08-10)
 os       macOS Big Sur 10.16         
 system   x86_64, darwin17.0          
 ui       X11                         
 language (EN)                        
 collate  en_US.UTF-8                 
 ctype    en_US.UTF-8                 
 tz       America/Chicago             
 date     2021-09-29                  

─ Packages ───────────────────────────────────────────────────────────────────
 package     * version date       lib source           
 cli           3.0.1   2021-07-17 [1] standard (@3.0.1)
 sessioninfo   1.1.1   2018-11-05 [1] standard (@1.1.1)
 withr         2.4.2   2021-04-18 [1] standard (@2.4.2)

[1] /Library/Frameworks/R.framework/Versions/4.1/Resources/library

[bersbersbers]

Agreed - same output in R here:

> x <- c("a", "A", "å", "b", "B")
> withr::with_collate("C", sort(x))
[1] "A" "B" "a" "b" "å"
> withr::with_collate("en_US.UTF-8", sort(x))
[1] "a" "A" "å" "b" "B" 

My OS is very different:

> sessioninfo::session_info()
─ Session info ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 setting  value                       
 version  R version 4.1.1 (2021-08-10)
 os       openSUSE Leap 15.2          
 system   x86_64, linux-gnu           
 ui       X11                         
 language (EN)                        
 collate  en_US.UTF-8                 
 ctype    en_US.UTF-8                 
 tz       Europe/Berlin               
 date     2021-09-29                  

─ Packages ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 package     * version date       lib source        
 cli           3.0.1   2021-07-17 [1] CRAN (R 4.1.0)
 glue          1.4.2   2020-08-27 [1] CRAN (R 4.1.0)
 import        1.2.0   2020-09-24 [1] CRAN (R 4.1.0)
 sessioninfo   1.1.1   2018-11-05 [1] CRAN (R 4.1.0)
 withr         2.4.2   2021-04-18 [1] CRAN (R 4.1.0)

[1] /data2/bers/opt/R/4.1/library
[2] /usr/lib64/R/library

(import and glue being used in my .Rprofile, but I tried without it, with the same result.)

One of the mysteries for me is why your computer changes the sorting of _ and c depending on locale.

That's not a complete mystery, I think - compare https://stackoverflow.com/a/43081689

What is a mystery to me is how in devtools::check(), sort is affected in a way that is

• unexpected/unwanted,
• irreproducible (in R) otherwise and, at the same time,
• reproducible in bash.

[schloerke]

What is a mystery to me is how in devtools::check(), sort is affected in a way that is

• unexpected/unwanted,
• irreproducible (in R) otherwise and, at the same time,
• reproducible in bash.

@bersbersbers Do you mind opening an Issue in http://github.com/r-lib/devtools and tagging this Issue? Thank you!

[wch]

For the record: On my Linux machine, I see that the command-line sort order of c and _ is affected by locale and I get the same results as @bersbersbers. This is unlike on my Mac that I tested on earlier, where the locale did not affect it. But it turns out this is just because the sort programs on Linux and Mac are different. See: https://unix.stackexchange.com/a/591205 https://blog.zhimingwang.org/macos-lc_collate-hunt

So the sort program isn't a good benchmark to use cross-platform; best to stick with R for testing sort order, since that's what we care about.

[bersbersbers]

So the sort program isn't a good benchmark to use cross-platform; best to stick with R for testing sort order, since that's what we care about.

Agreed - just keep in mind that the unexpected sort order that we are receiving is identical to the one we get from sort, for reasons still to be figured out.

I posted https://github.com/r-lib/devtools/issues/2377 including a reprex (which, by the way, does not include any locale manipulations from my side).

[bersbersbers]

A summary from the (long) other issue that may be interesting for @LamaTe is that it is expected and documented that R CMD check sets LC_COLLATE=C before running. So differences between native runs and R CMD check are to be expected if your native environment uses a difference LC_COLLATE. One might think that using Sys.setlocale would help, but it can make things even worse.

In my case, there were initial differences regarding sorting of capital letters, which I fixed by setting LC_ALL manually. But for some reason unknown to me, sorting of underscores differs as a function of the system-environment LC_COLLATE even if one sets a within-R locale just before the call to sort:

$ env -i LC_COLLATE=  R -q -e 'Sys.setlocale("LC_ALL", "en_US.UTF-8"); sort(c("_a", "a", "b", "A", "B"), method = "shell")'
> Sys.setlocale("LC_ALL", "en_US.UTF-8"); sort(c("_a", "a"))
[1] "LC_CTYPE=en_US.UTF-8;LC_NUMERIC=C;LC_TIME=en_US.UTF-8;LC_COLLATE=en_US.UTF-8;LC_MONETARY=en_US.UTF-8;LC_MESSAGES=C;LC_PAPER=C;LC_NAME=C;LC_ADDRESS=C;LC_TELEPHONE=C;LC_MEASUREMENT=C;LC_IDENTIFICATION=C"
[1] "_a" "a"  "A"  "b"  "B" 

$ env -i LC_COLLATE=C R -q -e 'Sys.setlocale("LC_ALL", "en_US.UTF-8"); sort(c("_a", "a", "b", "A", "B"), method = "shell")'
> Sys.setlocale("LC_ALL", "en_US.UTF-8"); sort(c("_a", "a"))
[1] "LC_CTYPE=en_US.UTF-8;LC_NUMERIC=C;LC_TIME=en_US.UTF-8;LC_COLLATE=en_US.UTF-8;LC_MONETARY=en_US.UTF-8;LC_MESSAGES=C;LC_PAPER=C;LC_NAME=C;LC_ADDRESS=C;LC_TELEPHONE=C;LC_MEASUREMENT=C;LC_IDENTIFICATION=C"
[1] "a"  "_a" "A"  "b"  "B"

So the problem I was trying to avoid was actually caused, just in a different way, by me trying to avoid it. I believe the handling of underscores is a bug, but maybe there is a good reason things are the way they are.

If you are sorting yourself, you may chose method = "radix" to achieve identical sorting. Otherwise, forcing LC_COLLATE to C (instead of en_US.UTF-8) solves the problem for me, although it did cause me having to regenerate all my existing tests because of differences in sorting of capital letters.

Edit: I have to take this back: LC_COLLATE=C leads to unacceptable ordering of user-facing strings in the app, so I cannot force LC_COLLATE=C on the whole app.

Maybe shinytest could come up with a default-off option for stable sorting (e.g., using method = radix) in snapshotInit and/or snapshot. One could then amend the new-test wizard to full this default-off option with an "on" value so that newly generated tests will benefit from it without changing existing tests.

[LamaTe]

Hi all, Thank you for the swift replies & help.

Please try using remotes::install_github("rstudio/shiny#3515") (and restart your R session) to see if the bug is fixed. Thank you!

@schloerke Using the suggested shiny#3515 indeed solves the problem for me - awesome! I am, however, not setting the locale value in my .Rprofile manually.

Just for reference, here are my specs: Locale:
> Sys.getlocale() [1] "LC_CTYPE=en_GB.UTF-8;LC_NUMERIC=C;LC_TIME=de_DE.UTF-8;LC_COLLATE=en_GB.UTF-8;LC_MONETARY=de_DE.UTF-8;LC_MESSAGES=en_GB.UTF-8;LC_PAPER=de_DE.UTF-8;LC_NAME=C;LC_ADDRESS=C;LC_TELEPHONE=C;LC_MEASUREMENT=de_DE.UTF-8;LC_IDENTIFICATION=C"

Session:

sessioninfo::session_info() ─ Session info ───────────────────────────────────────────────────── setting value
version R version 4.1.1 (2021-08-10) os Manjaro Linux
system x86_64, linux-gnu
ui RStudio
language (EN)
collate en_GB.UTF-8
ctype en_GB.UTF-8
tz Europe/Amsterdam
date 2021-09-30

─ Packages ─────────────────────────────────────────────────────────

package version date lib source
assertthat 0.2.1 2019-03-21 [1] CRAN (R 4.0.3) base64enc 0.1-3 2015-07-28 [1] CRAN (R 4.0.3) callr 3.7.0 2021-04-20 [1] CRAN (R 4.0.4) cli 3.0.1 2021-07-17 [1] CRAN (R 4.1.0) crayon 1.4.1 2021-02-08 [1] CRAN (R 4.0.3) curl 4.3.2 2021-06-23 [1] CRAN (R 4.1.0) debugme 1.1.0 2017-10-22 [1] CRAN (R 4.0.3) digest 0.6.27 2020-10-24 [1] CRAN (R 4.0.3) ellipsis 0.3.2 2021-04-29 [1] CRAN (R 4.1.0) fastmap 1.1.0 2021-01-25 [1] CRAN (R 4.0.3) htmltools 0.5.2 2021-08-25 [1] CRAN (R 4.1.0) httpuv 1.6.3 2021-09-09 [1] CRAN (R 4.1.1) httr 1.4.2 2020-07-20 [1] CRAN (R 4.0.3) jsonlite 1.7.2 2020-12-09 [1] CRAN (R 4.0.3) later 1.3.0 2021-08-18 [1] CRAN (R 4.1.0) lifecycle 1.0.0 2021-02-15 [1] CRAN (R 4.0.4) magrittr 2.0.1 2020-11-17 [1] CRAN (R 4.0.3) mime 0.11 2021-06-23 [1] CRAN (R 4.1.0) parsedate 1.2.1 2021-04-20 [1] CRAN (R 4.0.4) pingr 2.0.1 2020-06-22 [1] CRAN (R 4.0.3) png 0.1-7 2013-12-03 [1] CRAN (R 4.0.3) processx 3.5.2 2021-04-30 [1] CRAN (R 4.1.0) promises 1.2.0.1 2021-02-11 [1] CRAN (R 4.0.4) ps 1.6.0 2021-02-28 [1] CRAN (R 4.0.4) R6 2.5.1 2021-08-19 [1] CRAN (R 4.1.0) Rcpp 1.0.7 2021-07-07 [1] CRAN (R 4.1.0) rematch 1.0.1 2016-04-21 [1] CRAN (R 4.0.3) rlang 0.4.11 2021-04-30 [1] CRAN (R 4.1.0) sessioninfo 1.1.1 2018-11-05 [1] CRAN (R 4.0.3) shiny 1.6.0 2021-01-25 [1] CRAN (R 4.0.3) shinytest * 1.5.0 2021-01-13 [1] CRAN (R 4.1.1) showimage 1.0.0 2018-01-24 [1] CRAN (R 4.0.3) testthat 3.0.4 2021-07-01 [1] CRAN (R 4.1.0) webdriver 1.0.6 2021-01-12 [1] CRAN (R 4.0.3) withr 2.4.2 2021-04-18 [1] CRAN (R 4.0.4) xtable 1.8-4 2019-04-21 [1] CRAN (R 4.0.3)

[1] /home/laurens/RProjects/dev_pack_mlr3shiny [2] /usr/lib/R/library

@bersbersbers - Thank you for the helpful discussion. Hope you also managed to get a better lasting solution.

[bersbersbers]

I have fixed my part of this issue through adding Sys.setenv, see https://github.com/r-lib/devtools/issues/2377#issuecomment-931026796 for more information. I am not sure if Sys.setenv needs to be added to rstudio/shiny#3515 - I leave this for others to decide (but I guess yes!)

[stla]

Hello,

I have a similar issue: https://community.rstudio.com/t/order-difference-between-shinytest-expected-and-current-json-files/145549

However I have the latest version of Shiny. Is it due to my locale?

> Sys.getlocale()
[1] "LC_COLLATE=French_Belgium.utf8;LC_CTYPE=French_Belgium.utf8;LC_MONETARY=French_Belgium.utf8;LC_NUMERIC=C;LC_TIME=French_Belgium.utf8"

[bersbersbers]

Can you share two strings that change order?

[stla]

Hello @bersbersbers

When I directly run testApp, then the first key of the JSON object is data_experimentId. When I run it via testthat, the first key is dataProcess.

[stla]

I've just tried:

library(shinytest)
collation <- Sys.getlocale("LC_COLLATE")
Sys.setlocale("LC_COLLATE", "C")
shinytest::testApp("../")
Sys.setlocale("LC_COLLATE", collation)

and:

test_that("Shiny test", {
  skip_if_not_installed("shinytest")
  library(shinytest)
  collation <- Sys.getlocale("LC_COLLATE")
  Sys.setlocale("LC_COLLATE", "C")
  expect_pass(
    testApp(
      system.file("app", package = "shinyCircularDichroism"),
      quiet = TRUE,
      compareImages = TRUE
    )
  )
  Sys.setlocale("LC_COLLATE", collation)
})

but the problem remains.

[bersbersbers]

What OS are you on?

Another guess: your call to Sys.setlocale may not be doing anything. Check my loop snippet in https://github.com/rstudio/shinytest/issues/409#issuecomment-930403069 - I had been doing this exactly because different OSes use different locale strings, and a non-existent locale string makes Sys.setlocale fail silently. So I try setting multiple ones in a row, checking the return value and stopping as soon as one works. Nasty hack, but I am not aware of any cross-platform way of setting locales.

[stla]

I'm on Windows. Will read your comment, thanks.

[bersbersbers]

Read also https://github.com/r-lib/devtools/issues/2377#issuecomment-931026796 (TLDR: try adding a Sys.setenv).

[stla]

Thanks it works with Sys.setenv !!