shiny test errors

[cpsievert]

Hi @johndharrison,

Thanks for this wonderful package! I'm hoping to use this and testthat to create tests for the animint package. Specifically, my goal is to serve local files, access the DOM via RSelenium, and verify certain elements have certain properties (hopefully all within one R session). Your vignette on testing shiny apps is clearly related to my goal, but it isn't clear to me how the test knows how to serve the shiny app. For example, how do you go about serving the shiny app at http://127.0.0.1:6012? Do you have to do this in a separate R session? Related to my question, when I run

library(testthat)
library(devtools)
install_github("johndharrison/RSelenium")
library(RSelenium)
test_dir(paste0(find.package("RSelenium"), "/apps/shinytestapp/tests/"), filter = 'basic', reporter = "Tap")

I get:

1..3
# Context basic 
not ok 1 can connect to app 
  appTitle not equal to "Shiny Test App"
  1 string mismatches:
  x[1]: "Shiny Test App"
  y[1]: "Problem loading page"

not ok 2 controls are present 
  subscript out of bounds
  1: expect_equal(appCtrlLabels[[1]], "Select controls required:") at test-basic.r:24
  2: expect_that(object, equals(expected, label = expected.label, ...), info = info, label = label)
  3: condition(object)
  4: compare(expected, actual, ...)
  5: compare.character(expected, actual, ...)
  6: identical(x, y) 
not ok 3 tabs are present 
  subscript out of bounds
  1: expect_equal(appTabLabels[[1]], "Plots") at test-basic.r:34
  2: expect_that(object, equals(expected, label = expected.label, ...), info = info, label = label)
  3: condition(object)
  4: compare(expected, actual, ...)
  5: compare.character(expected, actual, ...)
  6: identical(x, y) 

[johndharrison]

Hi @cpsievert thanks for the kind words with regards to the package. Having worked with sports data I like what you have done with your sports related packages. It would be great to see more sports related packages on CRAN.

Unfortunately we have to deal with R's single threadedness so a second R process will need to drive the app. This can be done from within the R session making a call to system for example:

system("R -e \"shiny::runApp(paste0(find.package('RSelenium'), '/apps/shinytestapp'), port = 6012)\" &", intern = FALSE, wait = FALSE)

Now the test shiny app should be available at 127.0.0.1:6012.

So the test assumes that the app is running locally on port 6012.

[cpsievert]

Thanks! Looks like we have a lot in common ;)

And thanks for the solution. I think this will work just fine for my purposes. By the way, I don't think you want the & part. I ran this instead and it worked:

system("R -e \"shiny::runApp(paste0(find.package('RSelenium'), '/apps/shinytestapp'), port = 6012)\"", intern = FALSE, wait = FALSE)

[johndharrison]

Yeah the '&' was only needed on windows (system calls are cross platform problematic) on linux I needed to remove the ampersand. Let me know if you need any assistance when you are writing tests for animint I would be more than happy to lend a hand.

[cpsievert]

Ah, gotcha. I suppose I could use .Platform$OS.type to workaround that. One more question. Do you know of an elegant way to kill the process serving the local files in the background?

[johndharrison]

Lol I was afraid you were going to ask me about killing the process. On linux (Mac I would guess also) it would be easy just ps aux

> system('ps aux|grep "shiny::runApp"')
john      3491  0.1  1.6 254744 69876 ?        S    09:03   0:03 /usr/lib/R/bin/exec/R -e shiny::runApp(paste0(find.package('RSelenium'),~+~'/apps/shinytestapp'),~+~port~+~=~+~6012)
john      3647  0.0  0.0   4408   612 ?        S    09:35   0:00 sh -c ps aux|grep "shiny::runApp"
john      3649  0.0  0.0  13596   892 ?        S    09:35   0:00 grep shiny::runApp

That will give you the process id (3491 in this case). Then you could issue kill -TERM 3491

or if you are feeling very certain:

pkill -f shiny::runApp

so for example:

> system('ps aux|grep "shiny::runApp"')
john      3796 29.6  1.5 251044 66188 ?        S    09:54   0:01 /usr/lib/R/bin/exec/R -e shiny::runApp(paste0(find.package('RSelenium'),~+~'/apps/shinytestapp'),~+~port~+~=~+~6012)
john      3805  0.0  0.0   4408   608 ?        S    09:54   0:00 sh -c ps aux|grep "shiny::runApp"
john      3807  0.0  0.0  13592   892 ?        S    09:54   0:00 grep shiny::runApp
> system('pkill -f shiny::runApp')
> system('ps aux|grep "shiny::runApp"')
john      3812  0.0  0.0   4408   608 ?        S    09:54   0:00 sh -c ps aux|grep "shiny::runApp"
john      3814  0.0  0.0  13596   892 ?        S    09:54   0:00 grep shiny::runApp

On windows it maybe easiest to get the process id by checking what ports are in use:

netstat -ano

and killing the process that is listening on port 6012. You may need a shell script to do this. I am not sure you could do it from R in Windows.

For example in R if I try to kill an open notepad:

> system("taskkill /f /im notepad.exe")
Warning message:
running command 'taskkill /f /im notepad.exe' had status 322 

whereas from the windows command line

C:\Users\john>taskkill /f /im notepad.exe
SUCCESS: The process "notepad.exe" with PID 6060 has been terminated.

[cpsievert]

Wow! Thanks for great, quick answer! I might end up using a different port for each test. Would that be a terrible thing to do?

[johndharrison]

Does each test have a different app? If they are using the same app once the app is running it can be used for all tests. testthat executes helper files first. Helper files start with helper and loaded before any tests are run. So maybe the app could be initiated in a helper file?

[cpsievert]

I won't be using shiny, but instead using servr::httd() to serve files produced by animint. And yes, there will be different plots in different html files that I want to test.

I'll have to brush up on testhat. Hopefully I can generate every file I need using these helper files, serve everything under one port and run tests from there. Thanks for the help!

[johndharrison]

servr is like SimpleHTTPServer in python? I use this to run tests on RSelenium. The tests are located https://github.com/johndharrison/RSelenium/tree/master/inst/tests . In that case you would just need one instance of the http server running and it could be used for all tests. So for example when I am testing RSelenium I have a http server (SimpleHTTPServer in this case but servr would also serve the same purpose Im guessing) serving these files https://github.com/SeleniumHQ/selenium/tree/master/common/src/web . on port 3000. I have a simple loadPage function then that allows the http files to be referenced

htmlSrc <- Sys.getenv("SEL_TEST_DIR")
loadPage <- function(pgStr){
  paste0("file://", file.path(htmlSrc, paste0(pgStr, ".html")))
}

for example loadPage("simpleTest") will reference

> loadPage("simpleTest")
[1] "file:///home/john/git/selenium/common/src/web/simpleTest.html"

[cpsievert]

On the README yihui says it is like python -m SimpleHTTPServer "to a degree". Looking at the source code, it uses httpuv::runServer() (shiny::runApp() uses that as well). Anyway, I'll try this out and let you know how it goes :pray:

[johndharrison]

No worries I will mark this as closed. If there are any issues with the testing feel free to open another issue.

Cheers

[cpsievert]

So I managed to implement a basic test here. It's a bit confusing to me why remDr$open(silent = TRUE) prompts my browser to open. Is there is a way to avoid this?

[johndharrison]

Hi Carson,

Can you give sessionInfo() just so I can try to match your setup for further conversation and testing.

Cheers

[cpsievert]

Sure thing. From a clean session, I just ran:

library(devtools)
install_github("tdhock/animint", ref = "carson-test")
setwd(system.file("tests", package = "animint"))
source(system.file("tests", "testthat.R", package = "animint"))

I see:

Loading required package: RJSONIO
Loading required package: ggplot2
Need help? Try the ggplot2 mailing list: http://groups.google.com/group/ggplot2.
Loading required package: proto
Loading required package: grid
Loading required package: plyr
Loading required package: maps
Loading required package: reshape2
Loading required package: MASS
Loading required package: hexbin
Loading required package: lattice
Loading required package: scales
Loading required package: RCurl
Loading required package: bitops
Loading required package: caTools
Loading required package: XML

R version 3.1.0 (2014-04-10) -- "Spring Dance"
Copyright (C) 2014 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin13.1.0 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> servr::httd(port=4848)
serving the directory . at http://localhost:4848
labels : ..........

During this time, Firefox opens a new window and I can see it navigating to the different pages. And finally:

sessionInfo()

R version 3.1.0 (2014-04-10)
Platform: x86_64-apple-darwin13.1.0 (64-bit)

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] grid      stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] RSelenium_1.2.4    XML_3.98-1.1       caTools_1.16       RCurl_1.95-4.1     bitops_1.0-6       servr_0.1          animint_2014.5.17  scales_0.2.3      
 [9] hexbin_1.26.3      lattice_0.20-29    MASS_7.3-31        reshape2_1.2.2     maps_2.3-6         plyr_1.8.1         proto_0.3-10       ggplot2_0.9.3.1.99
[17] RJSONIO_1.2-0.1    testthat_0.8.1     devtools_1.5.0.99 

loaded via a namespace (and not attached):
 [1] colorspace_1.2-4   dichromat_2.0-0    digest_0.6.4       evaluate_0.5.3     gtable_0.1.2       httpuv_1.3.0       httr_0.3           labeling_0.2      
 [9] memoise_0.1        munsell_0.4.2      parallel_3.1.0     RColorBrewer_1.0-5 Rcpp_0.11.1        stringr_0.6.2      tools_3.1.0        whisker_0.3-2  

[johndharrison]

So is the motivation to provide a test suite for your package? For example something that would integrate with continuous integration setups? I can suggest some remedies for your problem described above. For example you can run firefox headlessly using Xvfb. You can switch to chrome and issue a --no-startup-window arg on startup. You can use phantomjs. However ultimately for package testing I would advocate using an external provider to take care of the selenium servers. SauceLabs provides free testing for open source projects like animint. They also have SauceConnect which allows you to test local running apps with externally driven browsers. I use SauceLabs to test RSelenium. On the readme you can see the SauceLabs build status https://saucelabs.com/u/rselenium0.

This will enable you to test animint against many OS/browser combos https://saucelabs.com/platforms. You can create a free account https://saucelabs.com/opensauce.

[johndharrison]

If you would like to manage your own selenium server/ OS/ browser etc. Then it maybe easiest to use phantomjs. http://phantomjs.org/download.html contains the download files for Mac. Ensure the phantomjs bin is in your path. Then you can drive phantomjs simply by asking for it with the browserName option. So rather then remDr <- remoteDriver() use remDr <- remoteDriver(browserName = "phantomjs"). The phantomjs browser is headless so you wont have any window popping up.

You can run chrome and firefox etc headlessly. I will be producing a vignette on how to do this. However they would normally be running on a VPS.

[cpsievert]

Yes, I only have a few simple checks right now, but the plan is to do some more. Saucelabs looks interesting, but (at least for now) it might be overkill since I intend on doing some simple checks that certain SVG elements are what we expect them to be. I've looked into using Travis CI and phantomjs, but to be honest, it seemed complicated (I'm much more comfortable with R than JS). In fact, that was a big reason why I was relieved to find your package :)

I think for now switching to chrome and issuing a --no-startup-window is a good enough solution. I believe most of the interactive testing was done on Firefox anyway. Plus, who needs those other browsers :laughing:

[cpsievert]

Oh, that's good to know there is support for phantomjs. Keep up the good work :thumbsup:

[johndharrison]

You would issue a --no-startup-window using chrome as follows:

cprof <- list(chromeOptions = list(args = list("--no-startup-window")))
remDr <- remoteDriver(browserName = 'chrome', extraCapabilities = cprof)
remDr$open()

Unfortunately it appears from https://code.google.com/p/selenium/issues/detail?id=5351 that the chrome browser needs to be started open

Much like --disable-javascript, the chromedriver will not work if you use --no-startup-window. It needs to launch a window to establish the connection with the AutomationProxy.

[johndharrison]

I will run your test on RSelenium's sauceLab account and provide the code used so you can see what is entailed.

[cpsievert]

Thanks for the heads-up. I guess I'll look into remDr <- remoteDriver(browserName = "phantomjs") and possibly setup a Travis CI hook.

[johndharrison]

OK no worries. Have a look into using phantomjs for now. Probably more important to get some tests etc running. When you need to there are options for headless testing. There is also a more up-to date version of the vignette Driving OS/Browsers local and remote which details testing with SauceLabs and running phantomjs etc.