The web-platform-tests tracking issue

orottier commented 6 months ago

The wpt suite was introduced to this lib in #42. It runs without crashing now and our aim is to improve the test score over time.

This tracking issue lists the high level TODOs. Single test failures are probably best reported and fixed with separate issues (either in the node bindings or the rust library).

[x] get rid of the wpt fork (how to deal with webaudio/resources/buffer-loader.js, reinstate 8 deleted files)
[ ] how to handle the crashtest tests?
[x] look into idlharness suite #86
[x] collect results of the browser implementations to determine baseline performance
[x] implement major missing functionality (#31 #28 ...)
[x] improve reported errors (see #8 #54 )
[x] check reporter, it seems to count some log lines as errors e.g. × Executing "test"
[x] focus on foundational test failures (the-audionode-interface, the-audioparam-interface, the-audiocontext-interface, the-offlineaudiocontext-interface, processing-model, the-audiobuffer-interface)
[x] focus on other test failures
[ ] re-enable 'the-audiocontext-interface/suspend-with-navigation.html'
[ ] re-enable 'the-audionode-interface/audionode-disconnect-audioparam.html'
[x] re-enable 'the-audionode-interface/audionode-disconnect.html'

b-ma commented 6 months ago

Thanks for the follow up, two notes:

get rid of the wpt fork (how to deal with webaudio/resources/buffer-loader.js, reinstate 8 deleted files)

I think this can be done by

Implementing the context.decodeAudioData(response, successCallback, errorCallback) signature, which should be rather simple to do on the JS side
Providing a mock to XMLHttpRequest to the runner using raw fs.readFileSync internally

collect results of the browser implementations to determine baseline performance

Maybe a bit old but there is this report: https://webaudio.github.io/web-audio-api/implementation-report.html

orottier commented 6 months ago

I have submitted a PR over at wpt for the buffer-loader.js: https://github.com/web-platform-tests/wpt/pull/43824 Let's see where that gets us

b-ma commented 6 months ago

results from #42

RESULTS:
- # pass: 3413 (+27)
- # fail: 1353 (-27)
- # type error issues: 235 (=)

files ignored: 8 (=)

b-ma commented 6 months ago

results with #61

  RESULTS:
  - # pass: 3605 (+192)
  - # fail: 1284 (-69)
  - # type error issues: 234 (-1)

b-ma commented 6 months ago

results with #66

RESULTS:
  - # pass: 3778 (+173)
  - # fail: 1290 (+6)
  - # type error issues: 59 (-175)

(the SNR is improving)

orottier commented 6 months ago

Results with v0.41.1 (no PR needed here due to patch release, just cargo update)


  RESULTS:
  - # pass: 3858 (+80)
  - # fail: 1216 (-74)
  - # type error issues: 62 (+3)

orottier commented 5 months ago

Results from #70

  RESULTS:
  - # pass: 3856 (-2) 
  - # fail: 1226 (+10)
  - # type error issues: 63 (+1)
  - # files ignored: 3 (-5)

orottier commented 5 months ago

Results from #76 (webaudio-rs v0.42 mainly):

  RESULTS:
  - # pass: 4304 (+448)
  - # fail: 782 (-444)
  - # type error issues: 59 (-4)
  - # files ignored: 3 (=)

b-ma commented 5 months ago

Results from 4b2dc95

RESULTS:
  - # pass: 4394 (+90)
  - # fail: 712 (-80)
  - # type error issues: 57 (-2)

orottier commented 5 months ago

Results from other browsers. Interesting to see they hit so many more assertions. Probably makes sense if there are worklet-node assertions for example. For more detail I will collect results per subdirectory (the-blabla-interface) and make another comparison.


Chrome 122.0.6261

./wpt run --log-mach=- --log-mach-level=info --headless --binary /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome chrome webaudio/

web-platform-test
~~~~~~~~~~~~~~~~~
Ran 8278 checks (8021 subtests, 257 tests)
Expected results: 8208
Unexpected results: 70
  test: 3 (2 error, 1 timeout)
  subtest: 67 (64 fail, 1 notrun, 1 precondition_failed, 1 timeout)

Firefox 123.0

./wpt run --log-mach=- --log-mach-level=info --headless --binary /Applications/Firefox\ Nightly.app/Contents/MacOS/firefox firefox webaudio/

web-platform-test
~~~~~~~~~~~~~~~~~
Ran 7662 checks (7405 subtests, 257 tests)
Expected results: 7097
Unexpected results: 565
  test: 10 (9 error, 1 timeout)
  subtest: 555 (526 fail, 24 notrun, 5 timeout)

Safari 17.3.1

./wpt run --log-mach=- --log-mach-level=info --headless --channel stable safari webaudio/

web-platform-test
~~~~~~~~~~~~~~~~~
Ran 8268 checks (8011 subtests, 257 tests)
Expected results: 8158
Unexpected results: 110
  test: 14 (5 error, 9 timeout)
  subtest: 96 (66 fail, 24 notrun, 6 timeout)

orottier commented 4 months ago

Results from #82 #85 and https://github.com/orottier/web-audio-api-rs/pull/455 https://github.com/orottier/web-audio-api-rs/pull/454 https://github.com/orottier/web-audio-api-rs/pull/452 https://github.com/orottier/web-audio-api-rs/pull/456


  RESULTS:
  - # pass: 4467 (+73)
  - # fail: 736 (+24)
  - # type error issues: 58 (+1)
  - # files ignored: 2 (-1)

orottier commented 4 months ago

I have made the initial quantitative comparison now per directory (per node)

Methodology:

our score = # pass + # type error
total = the reported number of subtests via the official wpt runner

I have filed an issue over at wpt to remove the empty dirs: https://github.com/web-platform-tests/wpt/pull/44749

b-ma commented 4 months ago

Cool, quite interesting

b-ma commented 4 months ago

With #87, #88 and web-audio-api-rs#1.0.0-rc.1

  RESULTS:
  - # pass: 4772 (+305)
  - # fail: 1052 (+316)
  - # type error issues: 54 (-4)
  - # files ignored: 2 (Ø)

orottier commented 4 months ago

With rust AudioParam PRs merged https://github.com/orottier/web-audio-api-rs/pull/459 https://github.com/orottier/web-audio-api-rs/pull/460

  RESULTS:
  - # pass: 5009 (+237)
  - # fail: 659 (-393)
  - # type error issues: 54 (=)
  - # files ignored: 2 (=)

b-ma commented 4 months ago

w/ #90

RESULTS:
  - # pass: 5278
  - # fail: 591
  - # type error issues: 46

b-ma commented 3 months ago

w/ update rc.3 and #103

  RESULTS:
  - # pass: 6240
  - # fail: 1072
  - # type error issues: 24

b-ma commented 3 months ago

w/ #104

RESULTS:
  - # pass: 6739
  - # fail: 800
  - # type error issues: 24

b-ma commented 3 months ago

w/ #112

RESULTS:
  - # pass: 6927
  - # fail: 677
  - # type error issues: 4
> wpt duration: 1:50.123 (m:ss.mmm)

b-ma commented 2 months ago

v0.20.0

  RESULTS:
  - # pass: 7005
  - # fail: 618
  - # type error issues: 4
> wpt duration: 2:15.084 (m:ss.mmm)

orottier commented 1 month ago

The results for the AudioWorkletNode on my machine (with the latest fix for zero-output nodes) are

npm run wpt -- --filter the-audioworklet-interface

  RESULTS:
  - # pass: 259
  - # fail: 41
  - # type error issues: 0
> wpt duration: 2:57.474 (m:ss.mmm)

But they are plagued with messages Cannot read properties of undefined (reading 'getElementsByTagName'). Should we port any browser method to NodeJs again?

b-ma commented 1 month ago

But they are plagued with messages Cannot read properties of undefined (reading 'getElementsByTagName'). Should we port any browser method to NodeJs again?

Yup I guess something like but I could find where this comes from yet...

b-ma commented 1 month ago

/w #132

RESULTS:
  - # pass: 7316
  - # fail: 528
  - # type error issues: 3
> wpt duration: 3:00.313 (m:ss.mmm)

ircam-ismm / node-web-audio-api

The web-platform-tests tracking issue #57