search

tc39 / eshost

A uniform wrapper around a multitude of ECMAScript hosts. CLI: https://github.com/bterlson/eshost-cli
Other
142 stars 36 forks source link

Add XS test engine #49

Closed patrick-soquet closed 6 years ago

patrick-soquet commented 6 years ago

Thank you!

rwaldron commented 6 years ago

Exciting!!

rwaldron commented 6 years ago

I should use ${MODDABLE}/build/bin/mac/debug/xst?

rwaldron commented 6 years ago

This is excellent!

Here's my full report, including the "review environment setup"

Build xst:

rwaldron in ~/clonez/moddable/xs/makefiles/mac on public*
$ make
make -f xst.mk
# xst debug : cc xsAll.c
# xst debug : cc xsAPI.c
# xst debug : cc xsArray.c
# xst debug : cc xsAtomics.c
# xst debug : cc xsBoolean.c
# xst debug : cc xsCode.c
# xst debug : cc xsCommon.c
# xst debug : cc xsDataView.c
# xst debug : cc xsDate.c
# xst debug : cc xsDebug.c
# xst debug : cc xsDefaults.c
# xst debug : cc xsError.c
# xst debug : cc xsFunction.c
# xst debug : cc xsGenerator.c
# xst debug : cc xsGlobal.c
# xst debug : cc xsJSON.c
# xst debug : cc xsLexical.c
# xst debug : cc xsMapSet.c
# xst debug : cc xsMarshall.c
# xst debug : cc xsMath.c
# xst debug : cc xsMemory.c
# xst debug : cc xsModule.c
# xst debug : cc xsNumber.c
# xst debug : cc xsObject.c
# xst debug : cc xsPlatforms.c
# xst debug : cc xsProfile.c
# xst debug : cc xsPromise.c
# xst debug : cc xsProperty.c
# xst debug : cc xsProxy.c
# xst debug : cc xsRegExp.c
# xst debug : cc xsRun.c
# xst debug : cc xsScope.c
# xst debug : cc xsScript.c
# xst debug : cc xsSourceMap.c
# xst debug : cc xsString.c
# xst debug : cc xsSymbol.c
# xst debug : cc xsSyntaxical.c
# xst debug : cc xsTree.c
# xst debug : cc xsType.c
# xst debug : cc xsdtoa.c
# xst debug : cc xsre.c
# xst debug : cc api.c
# xst debug : cc dumper.c
# xst debug : cc emitter.c
# xst debug : cc loader.c
# xst debug : cc parser.c
# xst debug : cc reader.c
# xst debug : cc scanner.c
# xst debug : cc writer.c
# xst debug : cc xst.c
# xst debug : cc xst
make GOAL=release -f xst.mk
# xst release : cc xsAll.c
# xst release : cc xsAPI.c
# xst release : cc xsArray.c
# xst release : cc xsAtomics.c
# xst release : cc xsBoolean.c
# xst release : cc xsCode.c
# xst release : cc xsCommon.c
# xst release : cc xsDataView.c
# xst release : cc xsDate.c
# xst release : cc xsDebug.c
# xst release : cc xsDefaults.c
# xst release : cc xsError.c
# xst release : cc xsFunction.c
# xst release : cc xsGenerator.c
# xst release : cc xsGlobal.c
# xst release : cc xsJSON.c
# xst release : cc xsLexical.c
# xst release : cc xsMapSet.c
# xst release : cc xsMarshall.c
# xst release : cc xsMath.c
# xst release : cc xsMemory.c
# xst release : cc xsModule.c
# xst release : cc xsNumber.c
# xst release : cc xsObject.c
# xst release : cc xsPlatforms.c
# xst release : cc xsProfile.c
# xst release : cc xsPromise.c
# xst release : cc xsProperty.c
# xst release : cc xsProxy.c
# xst release : cc xsRegExp.c
# xst release : cc xsRun.c
# xst release : cc xsScope.c
# xst release : cc xsScript.c
# xst release : cc xsSourceMap.c
# xst release : cc xsString.c
# xst release : cc xsSymbol.c
# xst release : cc xsSyntaxical.c
# xst release : cc xsTree.c
# xst release : cc xsType.c
# xst release : cc xsdtoa.c
# xst release : cc xsre.c
# xst release : cc api.c
# xst release : cc dumper.c
# xst release : cc emitter.c
# xst release : cc loader.c
# xst release : cc parser.c
# xst release : cc reader.c
# xst release : cc scanner.c
# xst release : cc writer.c
# xst release : cc xst.c
# xst release : cc xst
rwaldron in ~/clonez/moddable/xs/makefiles/mac on public*
$ cd $MODDABLE/build/bin/mac/debug
rwaldron in ~/clonez/moddable/build/bin/mac/debug on eshost*
$ xst
xst [-h] [-m] [-s] [-v] files...
    -h: print this help message
    -m: run files as modules
    -s: run files as scripts
    -v: print XS version
without -m or -s, files are test262 cases or directories
rwaldron in ~/clonez/moddable/build/bin/mac/debug on eshost*
$ xst -v
XS 8.3.0
### directory not found: ../harness
  1. Pull eshost branch
  2. Install local eshost to eshost-cli
  3. Add xs to eshost-cli to confirm that eshost recognizes that type of host
  4. Run a bunch of tests using test262-harness that has local eshost npm installed
rwaldron in ~/review
$ ls
.
├── [drwxr-xr-x rwaldron staff     544 Feb 27 11:21]  eshost/
└── [drwxr-xr-x rwaldron staff     480 Feb 27 11:33]  eshost-cli/

2 directories, 0 files
rwaldron in ~/review
$ cd eshost
rwaldron in ~/review/eshost on master
$ git remote add patrick-soquet git://github.com/patrick-soquet/eshost.git
rwaldron in ~/review/eshost on master
$ git fetch --all
Fetching origin
Fetching upstream
Fetching bocoup
Fetching anba
Fetching patrick-soquet
remote: Counting objects: 7, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 7 (delta 4), reused 7 (delta 4), pack-reused 0
Unpacking objects: 100% (7/7), done.
From git://github.com/patrick-soquet/eshost
 * [new branch]      master         -> patrick-soquet/master
 * [new branch]      module-support -> patrick-soquet/module-support
 * [new branch]      new-interface  -> patrick-soquet/new-interface
rwaldron in ~/review/eshost on master
$ git checkout -b patrick-soquet-master patrick-soquet/master
Branch 'patrick-soquet-master' set up to track remote branch 'master' from 'patrick-soquet' by rebasing.
Switched to a new branch 'patrick-soquet-master'
rwaldron in ~/review/eshost on patrick-soquet-master
$ git pull --rebase upstream master
From github.com:bterlson/eshost
 * branch            master     -> FETCH_HEAD
First, rewinding head to replay your work on top of it...
Applying: Add XS test engine
rwaldron in ~/review/eshost on patrick-soquet-master*
$ cd ../eshost-cli
rwaldron in ~/review/eshost-cli on master*
$ npm install ../eshost
+ eshost@3.6.0
updated 1 package in 1.318s
rwaldron in ~/review/eshost-cli on master*
$ eshost --add xs xs ${MODDABLE}/build/bin/mac/debug/xst
Using config  /Users/rwaldron/.eshost-config.json
Host 'xs' added
rwaldron in ~/review/eshost-cli on master*
$ cd ~/clonez/test262-harness/
rwaldron in ~/clonez/test262-harness on master*
$ npm install ../eshost
npm WARN test262-harness@3.4.0 license should be a valid SPDX license expression

+ eshost@3.6.0
updated 1 package in 2.577s
rwaldron in ~/clonez/test262-harness on master*
$ cd ~/clonez/test262
rwaldron in ~/clonez/test262 on master*
$ test262-harness --hostType=xs --hostPath="${MODDABLE}/build/bin/mac/debug/xst" "test/built-ins/String/*.js"
FAIL test/built-ins/String/length.js (default)
  Unicode escape (surrogate pair) Expected SameValue(«1», «2») to be true

FAIL test/built-ins/String/length.js (strict mode)
  Unicode escape (surrogate pair) Expected SameValue(«1», «2») to be true

FAIL test/built-ins/String/proto-from-ctor-realm.js (default)
  Expected no error, got TypeError: ?.createRealm: call createRealm: no function

FAIL test/built-ins/String/proto-from-ctor-realm.js (strict mode)
  Expected no error, got TypeError: ?.createRealm: call createRealm: no function

Ran 179 tests
175 passed
4 failed
rwaldron in ~/clonez/test262 on master*
$ test262-harness --hostType=xs --hostPath="${MODDABLE}/build/bin/mac/debug/xst" "test/built-ins/Array/*.js"
FAIL test/built-ins/Array/proto-from-ctor-realm.js (default)
  Expected no error, got TypeError: ?.createRealm: call createRealm: no function

FAIL test/built-ins/Array/proto-from-ctor-realm.js (strict mode)
  Expected no error, got TypeError: ?.createRealm: call createRealm: no function

Ran 92 tests
90 passed
2 failed
rwaldron in ~/clonez/test262 on master*
$ test262-harness --hostType=xs --hostPath="${MODDABLE}/build/bin/mac/debug/xst" "test/built-ins/Function/*.js"
FAIL test/built-ins/Function/call-bind-this-realm-undef.js (default)
  Expected no error, got TypeError: ?.createRealm: call createRealm: no function

FAIL test/built-ins/Function/call-bind-this-realm-undef.js (strict mode)
  Expected no error, got TypeError: ?.createRealm: call createRealm: no function

FAIL test/built-ins/Function/call-bind-this-realm-value.js (default)
  Expected no error, got TypeError: ?.createRealm: call createRealm: no function

FAIL test/built-ins/Function/call-bind-this-realm-value.js (strict mode)
  Expected no error, got TypeError: ?.createRealm: call createRealm: no function

FAIL test/built-ins/Function/proto-from-ctor-realm.js (default)
  Expected no error, got TypeError: ?.createRealm: call createRealm: no function

FAIL test/built-ins/Function/proto-from-ctor-realm.js (strict mode)
  Expected no error, got TypeError: ?.createRealm: call createRealm: no function

Ran 237 tests
231 passed
6 failed
rwaldron in ~/clonez/test262 on master*
$ test262-harness --hostType=xs --hostPath="${MODDABLE}/build/bin/mac/debug/xst" "test/built-ins/TypedArray/*.js"
Ran 8 tests
8 passed
0 failed
rwaldron in ~/clonez/test262 on master*
$ test262-harness --hostType=xs --hostPath="${MODDABLE}/build/bin/mac/debug/xst" "test/language/expressions/arrow-function/**/*.js"
Ran 525 tests
525 passed
0 failed
rwaldron in ~/clonez/test262 on master*
$ test262-harness --hostType=xs --hostPath="${MODDABLE}/build/bin/mac/debug/xst" "test/built-ins/AsyncFunction/*.js"
Ran 30 tests
30 passed
0 failed
rwaldron in ~/clonez/test262 on master*
$ test262-harness --hostType=xs --hostPath="${MODDABLE}/build/bin/mac/debug/xst" "test/built-ins/Atomics/*.js"
Ran 6 tests
6 passed
0 failed
rwaldron in ~/clonez/test262 on master*
$ test262-harness --hostType=xs --hostPath="${MODDABLE}/build/bin/mac/debug/xst" "test/built-ins/WeakMap/*.js"
FAIL test/built-ins/WeakMap/proto-from-ctor-realm.js (default)
  Expected no error, got TypeError: ?.createRealm: call createRealm: no function

FAIL test/built-ins/WeakMap/proto-from-ctor-realm.js (strict mode)
  Expected no error, got TypeError: ?.createRealm: call createRealm: no function

Ran 44 tests
42 passed
2 failed
rwaldron in ~/clonez/test262 on master*
$ test262-harness --hostType=xs --hostPath="${MODDABLE}/build/bin/mac/debug/xst" "test/language/expressions/void/*.js"
Ran 18 tests
18 passed
0 failed
rwaldron in ~/clonez/test262 on master*
$ test262-harness --hostType=xs --hostPath="${MODDABLE}/build/bin/mac/debug/xst" "test/language/expressions/await/*.js"
FAIL test/language/expressions/await/await-awaits-thenables-that-throw.js (default)
  Test did not run to completion

FAIL test/language/expressions/await/await-awaits-thenables-that-throw.js (strict mode)
  Test did not run to completion

PASS test/language/expressions/await/syntax-await-has-UnaryExpression-with-MultiplicativeEPASS test/language/expressions/await/syntax-await-has-UnaryExpression-with-MultiplicativeERan 30 tests
28 passed
2 failed

I'm not sure why createRealm is producing the errors that we're seeing, but everything else is in perfect order here.

bterlson commented 6 years ago

createRealm is not supported was my understanding. @patrick-soquet is that the case? Can we just remove the impl and just throw new Error("xs does not support createRealm")?

rwaldron commented 6 years ago

I had seen this: https://github.com/Moddable-OpenSource/moddable/blob/public/xs/tools/xst.c#L1019 and assumed it was exposed for use by https://github.com/Moddable-OpenSource/moddable/blob/public/xs/tools/xst.c#L688

mathiasbynens commented 6 years ago

Re: https://github.com/bterlson/eshost/pull/49#issuecomment-368964348 jsvu supports installing xs now.

rm -rf -- ~/.jsvu && npm install jsvu -g && jsvu
patrick-soquet commented 6 years ago

Tell me if there is something more I need to do. Thank you,

mathiasbynens commented 6 years ago

@patrick-soquet Have you seen @bterlson’s comment?

patrick-soquet commented 6 years ago

Yes I did and forgot to reply. Sorry. I fixed it: global.createRealm exists now and is throwing an exception.

rwaldron commented 6 years ago

@patrick-soquet thanks for the follow up. I'm going to land this today with the goal of releasing on Monday March 19th