rstudio / rstudio

RStudio is an integrated development environment (IDE) for R
https://posit.co/products/open-source/rstudio/
Other
4.58k stars 1.08k forks source link

allow authoring of automation tests from within RStudio #14676

Closed kevinushey closed 2 weeks ago

kevinushey commented 3 weeks ago

Intent

This PR makes it possible to author, and run, RStudio automation / integration tests from within RStudio itself. The intent is to easy to:

The ultimate goal is to make it easy enough to author automation tests that they can be included as part of pull requests when appropriate, with those tests run on Jenkins as well.

The video below gives an example of me running the automation tests for a RStudio Desktop instance, from within my development RStudio Server instance.

https://github.com/rstudio/rstudio/assets/1976582/15847cbc-b9cd-4e1a-83a2-4541ce4e2aa5

Approach

This PR builds upon the Chrome DevTools Protocol, which exposes a websocket API for driving a Chromium instance. This is what gets used under the hood by tools like Selenium; however, we're creating and accessing the websocket by hand ourselves. This is all done through the R websocket package (maintained by us).

For non-interactive use, RStudio can now be launched with the --run-automation flag. This runs RStudio in a "headless" mode, where it will then launch a separate instance of RStudio with the --automation-agent flag, which will then instruct that automation agent to expose some extra hooks to make automation testing easier. (Currently, this just means it'll be easier to list application commands, and execute application commands, from Javascript.)

Automated Tests

An important question... who tests the automated tests?

QA Notes

We should consult with QA whether they'd like to author integration tests here, or continue authoring them in the separate rstudio-ide-automation repository. To further confirm, this PR only deals with the tests themselves; it does not touch any of the existing machinery for running automated tests, which should continue to live in rstudio-ide-automation.

Documentation

I should write a small README discussing the various SessionAutomation tools.

Checklist

kevinushey commented 3 weeks ago

I also think we'll want to bundle the automation tests with the IDE itself, so it can be easily run as-is. (Or, if not, make it easy to check out and run automation tests from a remote source?)