WIP to show how workloads from another repository could get installed and used in Speedometer, with minimal changes.

What is speedometer-workloads?

Speedometer-workloads is a standalone monorepo that contains example apps and utilities that can be installed and used, instead of the current workloads that are included in the speedometer benchmark itself.

Benefits of using speedometer-workloads

Decoupling workloads from the benchmark would allow more flexibility and control over what workloads to use for local testing and how they can be configured for these tests.

Having a more modern and dynamic development setup will promote speed of development and motivate experimentation of new workloads, without being forced to adhere to technical constraints of Speedometer itself.

Speedometer-workloads can adhere to its own release cycle and will establish a new versioning of these workloads, which will result in more granular control on what workloads to install for each release of Speedometer.

A nice side effect of this change is that the workloads would also be re-usable for other benchmarks, if the need arises in the future.

Workloads install

Workloads are now installed through a (private) npm package. Currently, one single package is used for all workloads, instead of installing each workload separately. The Benefit of using a single package is simplicity. Downside is that we can’t decide in Speedometer, which version of a specific workload should be installed. This should be an easy step to change though.

speedometer-workloads

This package should be installed as a regular dependency of Speedometer. For testing purposes, the current package is private, until the changes are approved and agreed on. Since the package is private, the npm package name is scoped: @tkober/speedometer-workloads. Link to the speedometer-workloads source: https://github.com/GoogleChromeLabs/speedometer-workloads

"dependencies": {
    "@tkober/speedometer-workloads": "^1.3.0"
}

workloads.config.folder.json

This file determines which workload should get copied over from the @tkober/speedometer-workloads package.

{
    "workloads": [
        { 
            "name": "charts-chartjs",
            "type":"static"
        }
    ]
}

copy-workloads

The copy-workloads script in the package.json file, creates a workloads folder in the root of Speedometer with a copy of the dist folders of each work that is listed in the workloads.config.folder.json file.

"scripts": {
    "copy-workloads": "npm explore @tkober/speedometer-workloads pnpm run move:apps:custom"
}

Benchmark Runner

_prepareSuite

Multiple promises have to be resolved for this step:

loadFrame: resolves once the iframe is done loading
suite.prepare: any additional preparation steps a workload might need
postMessageSent({ type: “app-ready”}): a message sent from the workload when it can accept a test

async _prepareSuite(suite) {
    const frame = this._page._frame;
    await Promise.all([
        postMessageSent({ type: "app-ready" }),
        loadFrame({ frame, url: `${suite.url}` }),
        suite.prepare(this._page)
    ]);
}

_runAllSuites

Compared to the previous implementation, we are now only appending an iframe and creating a Page class instance, if a suite is not disabled.

for (const suite of suites) {
    if (!suite.disabled){
        await this._appendFrame();
        this._page = new Page(this._frame);
        await this._runSuite(suite);
        this._removeFrame();
    }
}

Tests

Suites.push has been updated for each workload.

The url points to the workloads folder in the root of the repository.
Prepare function is now mostly just a stub, since the workloads send an “app-ready” message

Example

Suites.push({
    name: "TodoMVC-JavaScript-ES5",
    url: "/workloads/todomvc-es5/",
    tags: ["todomvc"],
    async prepare(page) {},
    tests: [...],
});

Notes

A high amount of changes is a result of removing workloads that are now installed through a npm package
For local testing, please reach out to me and I'll add you to the private npm package

@kara

WebKit / Speedometer

Speedometer workloads #416