Automatic performance and pixels regression test with webgfx-tests

[takahirox]

Hubs has two tests. One is code based unit tests (npm run test) and another one is smoke test.

I think it would be helpful if we have automatic performance and pixels regression test which helps us detect performance or output pixels degradation.

I tried webgfx-tests @fernandojsg made for WebGL application. It can be used as regression test, and also it can compare the performance across the platforms/browsers.

Let me share what it is and how to run for Hubs. If it looks good to you folks, I'd like to move forward to integrate this test in Hubs release or CI test.

---

Google document version

Hubs + webgfx-tests

What webgfx-tests is

webgfx-tests is a Performance testing tool for WebGL application. It measures runtime performance and also compares rendered pixels with reference image.

• Repository: https://github.com/MozillaReality/webgfx-tests
• Tool Author: Fernando https://github.com/fernandojsg
• Demo Video: https://www.youtube.com/watch?v=3f-jANLuJ7I
• Article: https://fernandojsg.com/project/webgfx-tests/

Features

• Ability to execute any application deterministically without modifications
• Compare performance but also rendered pixels
• Multiplatform: desktop and standalone devices such as XR headsets
• Support any browser: Chrome, Firefox, Firefox Reality, Safari, Oculus Browser, ...
• Constant condition test across platforms/browsers by hooking Math.random(), resize canvas size to specified one, and running for specified frames
• Record and replay inputs and events
• Command line tool (easy to automate)
• Results in a format easy to use for analysis

What it can’t do (yet)

• CPU profiling
• Test application which doesn’t use requestAnimationFrame()
• Start the test from a certain event as trigger (e.g. completion of downloading assets)
• Measure a certain event (e.g. parsing glTF or uploading texture)

Use cases

• Run the tests every release or commit to detect performance or functionality degradation
• Compare the performance across the platforms or browsers to check whether the platform/browser specific issues exist or not

Architecture

• Testing server: Loads local application files or external sites, inserts test code which hooks WebGL, Canvas, requestAnimationFrame, and Math.random and sends the information, status, and results to WebSocket server
• WebSocket server: Receives the information from Testing server and stores them to local storage
• Terminal and Browsers: By running command on terminal, webgfx-tests boots up the Testing server, the WebSocket server, and browsers, then the browser loads application including test code inserted by Testing server from Testing server and starts tests.

(Maybe the arrows between the Test server and local application files/external sites were pointing in the opposite direction… The Test server loads them and inserts the test code to them.)

Test

A browser loads an application from the Testing server. The testing server inserts test code to application. The test code hooks WebGL, Canvas, and requestAnimationFrame to collect the performance information on loading the page. The application runs for certain frames specified by test configuration (e.g. 400 frames), sends the results and statuses to the WebSocket server, compares the final frame rendered pixels to reference image, and closes the browser. The WebSocket server stores the received information to local storage.

Workaround for Hubs

Problems

Hubs expects to be run on some specific domains and ports like localhost:8080 or hubs.mozilla.com for security reasons. It doesn’t fit to webgfx-tests which runs applications on the testing server (localhost:3000) for the test.

Workaround

I decided to manually embed test code to Hubs and run Hubs on local Hubs server (localhost:8080). I made a branch including test code. When you run webgfx-tests, merge it to your branch.

Set up

1. Clone my webgfx-tests fork and use the Hubs branch.

$ cd work_dir
$ git clone https://github.com/takahirox/webgfx-tests.git
$ cd webgfx-tests
$ git checkout Hubs

Note: My branch is ahead of the master webgfx-tests repository for Hubs. I sent the PRs but Fernando has been too busy to review.

2. Create your Hubs branch for test and merge my WebGfxTest branch

$ cd work_dir
$ git clone https://github.com/mozilla/hubs.git
$ cd hubs
$ git branch Test
$ git checkout Test
$ git remote add takahirox https://github.com/takahirox/hubs.git
$ git fetch takahirox
$ git merge takahirox/WebGfxTest
$ npm ci

Merging WebGfxTest branch inserts test code to the Hubs client (src/hub.html).

TODO: Move WebGfxTest branch to master hubs repository?

3. Setup private CA

You need to set up a private CA to avoid an “unsecure site” page which blocks automatic tests.

Install mkcert https://github.com/FiloSottile/mkcert and then run the following commands.

$ mkcert -install
$ cd work_dir/webgfx-tests
$ mkcert -cert-file cert.pem -key-file key.pem localhost
$ cd work_dir/hubs
$ mkdir certs
$ cd certs
$ mkcert -cert-file cert.pem -key-file key.pem localhost

Confirm the location where the CA root files are placed.

$ mkcert -CAROOT
$ ls CAROOT
rootCA.pem rootCA-key.pem

And then import them to OS and/or web browsers

-- Windows --

Chrome

Windows search - Manage Computer Certificates - Trusted Root Certification Authorities - Certificates - right click - All Tasks - Import

Firefox

Preference - Privacy & Security - Certificates - View Certificates - Import

other browsers

T.B.D.

--Android mobile--

Chrome

Settings - Security & location - Advanced - Encryption & credentials - Credential storage

Firefox

Install Firefox Nightly, not regular Firefox, and open about:config and turn security.enterprise_roots.enabled on.

-- Chrome OS --

chrome://settings/certificates on Chrome - Authorities - Import

-- Android based VR headsets (Oculus Quest) --

Oculus browser

Open chrome://flags on Oculus browser and set #allow-insecure-localhost true

Firefox Reality

T.B.D.

-- Mac/Linux/iOS --

T.B.D.

4. Additional configuration for some platforms

Firefox

Enable WebXR (dom.vr.webxr.enabled) via about:config.

Run

Open two terminals. One is for Hubs and another one is for webgfx-tests.

Hubs

$ cd work_dir/hubs
$ npm run dev

webgfx-tests

$ cd work_dir/webgfx-tests
$ npm install
$ node src/main/index.js run hubs -b “firefox nightly,google chrome” -c ./examples/tests/webgfx-tests.config.json -o hubs.log -s
$ node src/main/index.js summary hubs.log -g browser

Summary example

Demo Video: https://twitter.com/superhoge/status/1311157506425516032

Note: Hubs test configuration is already set by my webgfx-tests Hubs branch. If you want to configure, edit work_dir/webgfx-tests/examples/tests/tests.json.

Note: If a web browser asks for mic permission, accept and remember the settings.

webgfx-tests For Android mobile and Android based VR headsets

Install ADB

Plug-in Android mobile via USB cable

$ adb reverse tcp:8080 tcp:8080
$ node src/main/index.js run hubs-mobile -a -h -b “chrome” -c ./examples/tests/webgfx-tests.config.json -o hubs_andoid.log -s
$ node src/main/index.js summary hubs_android.log -g browser

webgfx-tests in immersive mode

T.B.D.

webgfx-tests For Chrome OS

Currently it doesn’t seem possible to boot browsers from the command line on Chrome OS. Therefore, you manually need to boot the testing server, browser, and start the test.

$ node src/main/index.js start-server -c ./examples/tests/webgfx-tests.config.json -s

Then manually open browsers, access https://localhost:3000, and click “Test” of Hubs.

Once the test finishes, you can see the result at the bottom of the page.

Limitation: The results log won’t be written to an external file

Tips for making test case

Add people in a room

If you want to test in a room with dummy peers, you can use hubs/scripts/bot/run-bot.js.

$ node run-bot.js -r room_id

If you want multiple peers at the different spawn points named SpawnPoint[1-5].

$ for i in 1 2 3 4 5
do
  node run-bot.js -r room_id -s Spawn_Point_${i} &
done

If you want to terminate them.

$ jobs
$ kill %1 %2 %3 %4 %5

Place images, videos, or other user objects in a room

If you want to test with images, videos, other user objects in a room, you can pin them.

1. Manually enter a room on your browser
2. Place an object in the room
3. Select the object from the objects list
4. Pin it

The object keeps placed in the room even if you leave the room.

Test in a certain room

Edit the url property of “hubs*” tests in webgfx-tests/examples/test/test.json.

Known issues and features requests to Hubs or webgfx-tests

• After Hubs + webgfx-tests becomes more stable, I’d like to try to integrate this test with Hubs release test or CI test.
• Constant condition tests are hard if you want to test with remote peers. No guarantee that data arrives at a certain frame from remote peers. Workaround might be creating mock peers which run locally but act as if remote peers?
• Anything else?

┆Issue is synchronized with this Jira Task

[takahirox]

Can I ask for volunteers? I want you folks to try to setup the performance test tool (follow the Set up section and below in the issue comment) and give me the feedbacks especially about

1. Does the performance test tool run in your platforms (OS + browsers + devices)?
2. Are there any difficulties to setup and run?
3. Any feature/enhancement requests?