Testing using cypress

[da-h]

Description

This PR implements a baseline for automated testing of visdom using cypress.

Motivation and Context

Following the discussion in #849, this PR introduces some basic tests for many features of visdom.

In more detail:

• As basic setup tests, there are tests for
  • for the connection to the server
  • the possibilty to reconnect using the ui
  • the env / tree selection bar (does it open correctly, can we (de)select environment & clear everything)
• All Pane-Types (except the EmbeddingPane) are tested. The implemented tests are
  • Showing one ore more windows of the same example (to test repeatability)
  • Drag & drop- rearrangement of all pane types on the visdom dashboard
  • Resizing of panes & resetting the size using double-click
  • Closing of paness
• Basic filtering function tests for the regex-filter field
• Image Pane tests:
  • movement tests using mouse-drag or mouse-wheel
  • zoom using ctrl + wheel
  • reset of the viewpoint using double-click
  • download-button test
  • image history functionality with a slider
  • image grid functionality
• All examples from demo.py have been split into seperate files and functions to enable triggering them seperately. There is a test for each example.
  • For most plots and many other examples, there are visual regression tests that compare the screenshots of the output before & after any code change.
• Callback tests
  • properties with callbacks
  • image callbacks including mouse- & key-presses
  • text callbacks

Additionally:

• I found some bugs during the creation of the tests. (Comments in code). I will submit issues for anyone interested solving these.
• Added a data-cy-attribute to the filter input field to better match it in the test
• fixed a missing import in js/PropertiesPane.js
• adapted CONTRIBUTING.md and PULL_REQUEST_TEMPLATE.md to show the use of the new commands

How Has This Been Tested?

Who tests the tester? ;)

If you want to try it, do:

1. start a visdom server on port 8098
2. run npm run test:gui
3. click on screenshots.init.js (otherwise the visual regression tests in screenshots.js will fail)

Screenshots (if appropriate):

Types of changes

• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

• [x] My code follows the code style of this project.
• [x] My change requires a change to the documentation.
• [x] I have updated the documentation accordingly.
• [x] I have tested the changes using npm test. (Please check the CONTRIBUTING.MD file for details.)

[JackUrb]

WOW I love the cypress testing suite here. Huge kudos for getting it set up! I'd love to get this into CI for the project in some form, which I know @mariobehling was hoping we could add at some point.

I think I'm running into some issues though, and I'm not sure quite why: Some tests fail with missing content And others aren't getting the exact value they're looking for but most frequently tests fail because "Timed out retrying after 4000ms: Expected to find element: .rc-tree-select-selection__clear, but never found it.", but I think that has to do with a cascading failure after the env window isn't closed.

[da-h]

Too bad I got sloppy at the end. Sorry for the rough mistakes. -.-

Please check my resubmission above. :)

There were a few places that needed to be revisited:

• There was a syntax error in the asynchronous call (run task with option asyncrun: true): a space was missing between the arguments in some cases.
• The test for the graph-visualization was simply wrong. I fixed the wrong target value.
• The screenshots have been removed in between npm run test:init and npm run test. I have added a hook (after:screenshot) that saves the screenshots somewhere safe when running init-scripts, i.e. cypress/screenshots_init instead of cypress/screenshots.
• The screenshots of the npm run test:init script did not use the exact same visual spot for taking the screenshot. This should be fixed now.
• I haven't noticed that the initialization tests have been running also during the normal tests and thus have overwritten the screenshots. To solve this, I have adapted the command npm run test to ignore any *.init.js files. I also added a dedicated npm run test:visual to only run a visual regression test.
• There was a concurrency problem during file system write in the cypress task numDifferentPixels. I have switched to the synced filesystem commands there.
• ImagePane had an unneeded > symbol, which I removed.
• The visual regression test was inconsistent for the compare-views, but only every now and then.
  • One reason was that the tests for the compare view did use parallel run-commands (asyncrun: true).
  • The other reason was that the dict iteration in servery.py did not use consistent orders.
  • I have fixed both problems by sorting the keys in server.py before iterating (see 2nd last commit) & also ensuring that the tests show up in correct order (by prepending numbers before starting the envs).

[da-h]

In total, these are the issues I found during testing or / and which I will file in next week in the issue tracker:

• missing Embedding functionality testing (see below)
• inconsistent resize of the audio pane (sometimes it resizes to a bigger, sometimes to a smaller window)
• and also wrong resize for the ImagePane (not the same it started with)
• ImagePane requires multiple rerenders before showing the image correctly
• sometimes the env selection is covered by its selection tree
• the image callback result for mouseclicks seems to be off by 1 on the y-axis
• another concurrency bug exists during text callbacks. these require a short delay in between inputs. otherwise the events can change position

[da-h]

Here are a few more notes:

• I left out here the Embedding-Functionality. Since there is another project involved here, it's probably best to look at that separately.
• CI integration should not be difficult to integrate from here on. As far as i can see, there is either the conventional integration via github workflows, or via the github app cypress, which remains free of charge for less than 500 tests per month. However, they also seem to offer the possibility to test open source projects completely free of charge in case they surpass the 500 tests and apply for their program.
• I included a very small bugfix in server.py to fix the occurring error in the image_history test.

[da-h]

Unfortunately there were still two small bugs in it. I have to admit, this whole PR was much more fiddly than I expected. :)

1. The first problem that occurred especially with a completely fresh server instance was that the tree-select structure is not loaded expanded if a new folder is created by an env update. As far as I can see this is a bug of TreeSelect, because this actually gets the argument treeDefaultExpandAll. I did not see this problem before, as I didn't restart the server too often during cypress testing. I solved the problem by having the test script open all closed folders in the selection.
2. The second problem occurred during CI testing. The x_download tests got inconsistent results. I guess this is due to varying download speeds / buffering speeds.
   I removed these tests from the test-list.

I will repush the CI and the changed files later today.

[JackUrb]

I have to admit, this whole PR was much more fiddly than I expected

No worries @da-h - you took on a pretty impressive set of changes rather quickly. I'm always happy to review and help - there's no rush!

[da-h]

Here are the most recent changes:

• removed npm test requirement from PULL_REQUEST_TEMPLATE.md
• added automatic pull request checks that use the new cypress test cases
• merged the "Update static js files"-bot into the new workflow

The new workflow "process-changes" does the following:

1. build & install everything based on the proposed PR state
   • in doing so, it also commits the build files, if any *.js files changed
2. each sub-step uses an action prepare to download / or load cached data to speed-up the actions overall. (caching of pip and npm downloads)
3. the visual regression test consists of two steps: first it uses the base branch to create screenshots (in the future this could be cached as well), the next step compares the same screenshots with that of the pr-branch
4. in parallel to the visual regression test, there is a functional cypress test

In total, the process looks like this:

I have tested the new CI using these three PRs

• one that checks if a valid change passes: https://github.com/da-h/visdom/pull/4 (
• one that checks if a invalid visual change fails: https://github.com/da-h/visdom/pull/5
• one that checks if a syntax error also fails: https://github.com/da-h/visdom/pull/6

[da-h]

It seems there is one last error in fetching from a feature branch if $GITHUB_HEAD_REF is coming from a fork (instead of origin).

[da-h]

As far as I can tell, this PR is now ready for the detailed review. Sorry again for the constant back and forth. As I said, this PR was quite a bumpy ride after all.

The last two commits change the process a bit, introducing the committing of build files in a seperate job.

With one exception, the errors in the PR-checks below are only due to missing files on master that are introduced in this very PR. As shown in the example PRs these errors do not show up in succeeding PRs.
The remaining error in the functional test is a bit mysterious as I cannot reproduce it, neither on my machine, nor on the PR-tests linked above.

[da-h]

As a final test, I tried to test against the changes in #849. (See the respective test-PR in my own fork)

Unfortunately, two CI-runs of the same pr yield inconsistent results, while it passes locally on my maschine.

See:

• https://github.com/da-h/visdom/actions/runs/2234258610 (one failed functional test )
• https://github.com/da-h/visdom/actions/runs/2234331698 (three failed functional tests)

[da-h]

After having read a bit about other people complaining about inconsistent cypress results, especially in github action runs (e.g. here), i think I have finally found the solution for the problem. It was a user error -- what else.

First, & just for future reference. I have tested many suggestions (from the links above) without success:

• headed browser in github action
• missing mouseover event for cypress .click() events (seemed to be a common problem for angular-apps)
• using .trigger('click') instead of .click() which triggers on the original dom state, in contrast to click() which may fail on dom changes during click event itself
• explicit wait after actions that may take longer to process

The error was:

• i used .click().click() instead of .dblclick() in one test
• in the other test cy.get('input[value="on"]') was inconsistent, while cy.get('input[type="checkbox"]') is not (which i do not understand)

I set the PR thus again on ready, and turn again at the CI-failure-success wheel of fortune. ;) The argument for a (this time) truly finished PR are the following consistent results of the functional tests here, here and here.

[da-h]

PS: As explained above, the visual regression tests are expected to fail for now due to missing files in the current master branch.

[da-h]

In the newest revision, I reverted to the original workflow update-js-build-files.yml.

I had hoped to merge the two workflows, but it is not so easy or at least not intended to commit to the original branch automatically when a PR is based on a branch of a fork. So the current suggestion is to leave the static-build-files-bot as it was (i.e. push-events initiate the bot on a per-branch basis) and additionally run the cypress tests on an incoming PR.

The advantage of all these additional tests is at least that the cypress CI tests have now been run several more times and i am now more sure that they are consistent. :)

[da-h]

🥳