How to add SharedArrayBuffer litmus tests?

[flagxor]

Cristian Mattarei (cced), a researcher at Stanford has been working on developing litmus tests for SharedArrayBuffers based directly on the SAB memory model. https://github.com/FMJS/Litmus-Tests

The litmus tests cover all possible memory interactions of sizes 1-4. They take up 202MB compressed and about 1.1GB uncompressed. As threads are inherently non-deterministic, so are the tests (i.e. you run them and they should not observe any unexpected results).

We're wondering if including them in test262, or possibly as an addendum to test262 makes sense? Would they make sense as a script and submodule or directly checked in? (Considering their size) I'd be curious your thoughts on the right way to take advantage of this work.

Thanks -BradN

[ajklein]

@leobalter @rwaldron another case of a big chunk of tests, and what to do with them. Thoughts?

cc @binji @domenic @cristian-mattarei

[rwaldron]

I'm going to spend some time reviewing this to get a better understanding for the shared material.

[cristian-mattarei]

@rwaldron, let me know if you need any information/clarification.

[rwaldron]

@cristian-mattarei thanks! What "runner" is being used to execute these tests? test262-harness? Is there a special host definition being used?

[rwaldron]

Just a heads up to anyone else that comes along, and is using a Mac... 4_events.zip needs to be zipped, then can be unzipped:

zip -s 0 4_events.zip --out 4_events_unsplit.zip
unzip 4_events_unsplit.zip

Also, it contains 224,003 files.

[cristian-mattarei]

Yes, test262-harness, and this is the list of special $262 host definition being used:

• $262.agent.start
• $262.agent.receiveBroadcast
• $262.agent.report
• $262.agent.leaving
• $262.agent.broadcast
• $262.agent.getReport
• print

For instance, these tests can be run using SpiderMonkey with the following command: <gecko-js> -f <gecko-dir>/js/src/tests/shell.js -f <gecko-dir>/js/src/tests/test262/shell.js -f <gecko-dir>/js/src/tests/test262/built-ins/shell.js -f <gecko-dir>/js/src/tests/test262/built-ins/Atomics/shell.js -f <gecko-dir>/js/src/tests/test262/built-ins/Atomics/wait/shell.js -f <litmus_test.js>

where:

• <gecko-js> is the compiled js engine,
• <gecko-dir> is the root gecko dir, and
• <litmus_test.js> is the litmus test

[rwaldron]

@cristian-mattarei I'm having trouble understanding your answers.

Yes, test262-harness,

Unfortunately, the example you gave that followed is not a valid test262-harness command. That would look like this:

test262-harness --hostType=jsshell --hostPath=`which js` --test262Dir=$HOME/clonez/test262 1_event/*js

But that produces the following output:

FAIL 1_event/1_event_0-program.js
  Expected no error, got TypeError: $262.agent is undefined

FAIL 1_event/1_event_0-program.js
  Expected no error, got TypeError: $262.agent is undefined

FAIL 1_event/1_event_1-program.js
  Expected no error, got TypeError: $262.agent is undefined

FAIL 1_event/1_event_1-program.js
  Expected no error, got TypeError: $262.agent is undefined

FAIL 1_event/1_event_2-program.js
  Expected no error, got TypeError: $262.agent is undefined

FAIL 1_event/1_event_2-program.js
  Expected no error, got TypeError: $262.agent is undefined

FAIL 1_event/1_event_3-program.js
  Expected no error, got TypeError: $262.agent is undefined

FAIL 1_event/1_event_3-program.js
  Expected no error, got TypeError: $262.agent is undefined

FAIL 1_event/1_event_4-program.js
  Expected no error, got TypeError: $262.agent is undefined

FAIL 1_event/1_event_4-program.js
  Expected no error, got TypeError: $262.agent is undefined

FAIL 1_event/1_event_5-program.js
  Expected no error, got TypeError: $262.agent is undefined

FAIL 1_event/1_event_5-program.js
  Expected no error, got TypeError: $262.agent is undefined

FAIL 1_event/1_event_6-program.js
  Expected no error, got TypeError: $262.agent is undefined

FAIL 1_event/1_event_6-program.js
  Expected no error, got TypeError: $262.agent is undefined

FAIL 1_event/1_event_7-program.js
  Expected no error, got TypeError: $262.agent is undefined

FAIL 1_event/1_event_7-program.js
  Expected no error, got TypeError: $262.agent is undefined

FAIL 1_event/1_event_8-program.js
  Expected no error, got TypeError: $262.agent is undefined

FAIL 1_event/1_event_8-program.js
  Expected no error, got TypeError: $262.agent is undefined

Ran 18 tests
0 passed
18 failed

That brings us back to your example command, which indicates that these tests have only been run in SpiderMonkey's local built version. The host runtime definitions that are used by test262-harness presently do not have the $262.agent that these tests depend on, which would exist here: https://github.com/bterlson/eshost/blob/master/runtimes/sm.js. The one that these tests are dependent on exists only in the SpiderMonkey source tree, here: https://github.com/mozilla/gecko-dev/blob/master/js/src/tests/test262-host.js#L26-L123 (@leobalter is presently working directly on this SM-local Test262). Until the host definition is upstreamed to eshost, these tests won't run in any of the Test262 general runners.

[cristian-mattarei]

@rwaldron, now I got your point. I have updated the tests with 1_event, and now those tests work with the test262-harness, but only with JSC:

test262-harness --hostType=jsshell --hostPath=`which jsc` --test262Dir=$HOME/repos/test262 all_programs_1_sab/1_event/*.js
Ran 18 tests
18 passed
0 failed

[rwaldron]

@cristian-mattarei I think there is still a misunderstanding. The expected identifier is $262, the problem is that there is no $262.agent object defined in the eshost runtime for SpiderMonkey. This code: https://github.com/mozilla/gecko-dev/blob/master/js/src/tests/test262-host.js#L37-L182 needs to be added in this file: https://github.com/bterlson/eshost/blob/master/runtimes/sm.js The problem still remains that these tests have been written against a SpiderMonkey specific host object and not the one that provided by the actual eshost tool, which is what test262-harness depends on.

[rwaldron]

The command you've provided above has a mismatched hostType/hostPath. You'd need to provide the correct hostType that matches the hostPath, for this case which jsc is the path to jsc, so you need to use the hostType jsc. Which reveals the failures:

$ test262-harness --hostType=jsc --hostPath=`which jsc` --test262Dir=$HOME/clonez/test262 all_programs_1_sab/1_event/*.js
FAIL all_programs_1_sab/1_event/1_event_0-program.js
  Expected no error, got 0: undefined

FAIL all_programs_1_sab/1_event/1_event_0-program.js
  Expected no error, got 0: undefined

FAIL all_programs_1_sab/1_event/1_event_1-program.js
  Expected no error, got 0: undefined

FAIL all_programs_1_sab/1_event/1_event_1-program.js
  Expected no error, got 0: undefined

FAIL all_programs_1_sab/1_event/1_event_2-program.js
  Expected no error, got 0: undefined

FAIL all_programs_1_sab/1_event/1_event_2-program.js
  Expected no error, got 0: undefined

FAIL all_programs_1_sab/1_event/1_event_3-program.js
  Expected no error, got 0: undefined

FAIL all_programs_1_sab/1_event/1_event_3-program.js
  Expected no error, got 0: undefined

FAIL all_programs_1_sab/1_event/1_event_4-program.js
  Expected no error, got 0: undefined

FAIL all_programs_1_sab/1_event/1_event_4-program.js
  Expected no error, got 0: undefined

FAIL all_programs_1_sab/1_event/1_event_5-program.js
  Expected no error, got 0: undefined

FAIL all_programs_1_sab/1_event/1_event_5-program.js
  Expected no error, got 0: undefined

Ran 18 tests
6 passed
12 failed

[cristian-mattarei]

Thank you for the clarification. In addition to that, these tests are intended to be run multiple times to maximize the coverage of the possible outputs. According to that, @rwaldron, do you think test262-harness would still be the right "runner" for the litmus tests?

[rwaldron]

I'm not yet sure what the right path forward is—we still need to be able to run them locally before we can make that determination. Any other similar insights? Anything you can add is appreciated.