Specify rules for integration tests in test262

[ajvincent]

I have been working on a ECMAScript-based Membranes project which combines several different ECMAScript features:

• Proxies
• WeakMaps
• Symbols
• (to-do) Modules
• (in the "laboratory" pages) Template strings

It is their integration into one project that we (Dr. Mark Miller and I) are interested in adding to test262: specifically, as a set of integration tests. Test262 appears to contain only unit tests right now, so it's not entirely clear where integration tests might go in the first place. So we're definitely breaking new ground here.

Similarly, because we're talking about testing a subset of my membrane project's code, I would have to extract and reduce the library being tested into a form appropriate for inclusion as an example for the tests to exercise. This, I can start as a branch in my own project repository.

This ticket is to establish the guidelines for integration tests in general, using my project as a model to begin from. Hopefully, a positive resolution will include adding a set of integration tests to exercise a minimal membrane implementation.

[rwaldron]

Since test262 is explicitly for specification conformance, perhaps creating something like the test262-parser-tests project would be appropriate.

[leobalter]

I saw the membranes project and I have some questions:

• How those tests are applied to other projects?
• The tests are using Node + jasmine. Test262 is neutral and not providing favor to any JS implementation, that can’t be achieved using Node. The same goes to choosing Jasmine or any specific test framework.
• Considering all the dependencies - including above - merging this with Test262 would also mean requiring an immense work changing all the runners and their API. The ideal is to adapt all those new tests to match Test262 style.
• Even if we adopt a new integration tests project/repo at TC39, we can’t have a Node + Jasmine (or other test framework) dependency and require other runtimes to include these dependencies to their CI. For that, we will need consensus.

Saying that, those tests look indeed specific to the project you mentioned. We need the feedback from other implementors saying those tests might be useful for other projects, related to the specs.

[ajvincent]

@leobalter I would be doing the porting away from Jasmine and away from Node. (Which probably involves rewriting the tests from scratch.) Your concerns about importing dependencies ring loud and clear, and it is entirely fair to demand removal... on a branch. :-) In particular, most of the special capabilities of the es7-membrane code would probably be going away (hiding properties, adding new ones, etc.), since those would add more complexity than we need for integration tests. I think what I'm trying to say is that a severely reduced es7-membrane implementation and its tests would be what I propose for integration testing of these features against an ECMAScript implementing engine: something that honors the mirroring that a membrane is supposed to do, and nothing more.

The build script actually concatenates the source files into docs/dist/staging (which is git-ignored) before adding Node-specific wrappers in docs/dist/node and browser-specific wrappers in docs/dist/browser.

The tests do not apply to other projects. As written, they are specifically for the es7-membrane project, to make sure that es7-membrane works correctly.

[leobalter]

The tests do not apply to other projects. As written, they are specifically for the es7-membrane project, to make sure that es7-membrane works correctly.

What would the pros on having TC39 to host these tests then?

I think what I'm trying to say is that a severely reduced es7-membrane implementation and its tests would be what I propose for integration testing of these features against an ECMAScript implementing engine: something that honors the mirroring that a membrane is supposed to do, and nothing more.

In this case I believe the best option is to find a proper CI approach to test es7-membrane with different implementations/environments. Hosting the tests here or as a TC39 repo can't offer any help and might bring some maintenance I believe TC39 can't afford right now.

For this, there's a very nice tool from @bterlson to check your code (including tests) in different runtimes: https://github.com/bterlson/eshost/

[erights]

@leobalter The detailed design of many of our abstractions is motivated by the constraints of supporting membranes well --- with both security and almost perfect transparency. Most obviously in the design of WeakMaps and Proxies, but it has effected many other parts of EcmaScript, such as which internal properties we do an do not reify into symbol-named properties.

These issues are subtle and we should not be too confident that we got it right. The individual tests we've got check conformance to those specific design choices, but does not test whether those choices compose to support membranes correctly. Alex's integration tests would help us gain that confidence. I support finding some way to include them in test262.

[littledan]

Generally, it seems like a great idea to, somewhere in the world, assemble regression tests based on common/useful libraries which can be shared between implementations and assured high quality. No particular opinion on whether that should be this repository, test262-parser-tests, web-platform-tests, or someplace else; I'll leave that to the maintainers. Certainly it wouldn't fit cleanly into the test262 directory structure. Just a couple questions related to this:

• How should we be judging which things make it into this repository? Should we focus on major libraries used in practice? In a slightly different area, JavaScript benchmarking, there's been a move away from constructed benchmarks and towards observed popular frameworks and websites. This has also been useful in integration tests which are aimed at minimizing compatibility risk of semantic changes (e.g., CIGM and the Swift commit queue, which run popular libraries).
• How would the tests guarantee that the membrane boundary is secure? I can see how you can test against particular issues that might come up, but it's hard to test that there's no way around a thing. For example, I don't think it's possible to test from inside the web platform whether cross-origin iframes are implemented in separate processes (which some implementations do and some don't), even if this is believed to reduce security risks. Can you write a test to find if an engine has an insecure implementation of WeakMaps or Proxies?

[ajvincent]

I think there's a mild misconception going on here. Let me give an example of a problem I'm currently facing.

Right now, if I call Object.freeze(membraneProxy);, the membrane code fails. That's clearly a bug in the membrane implementation. But if I discover that it just can't be fixed using ECMAScript, and the membrane isn't doing anything particularly funny, that indicates a bug in the language implementation or specification.

Either way, it is not unreasonable to ask how a proxy reacts to an Object.freeze() call. Forget for a moment that the proxy's handler comes from the membrane; the point is that if the handler is implemented somewhat sanely, then Object.freeze will work.

Another case: Proxy.create(new Date()) * 1 will return NaN, while (new Date()) * 1 will return a positive number. I know we don't have value proxies or operator overloading in ECMAScript yet, just a bunch of competing proposals. That's not the point. The point is to write a test showing this interaction between a proxy, a Date object and a multiplication operator. What's the expected result today, and (in the future, when TC39 agrees on operator overloading) what changes to a proxy handler should generate a different expected result?

The tests for a Proxy most likely cover individual behaviors on individual operations against the proxy. A Membrane implementation with lots of passing tests can lead to a flurry of tests that can be added to ECMAScript's test suites illustrating how these features of the ECMAScript language work together... or don't.

Remember, tests are a form of documentation as well. Integration tests are supposed to be based on what real software engineers do in real-world applications, at the higher levels. By just testing the lower level implementation features, sure, you guarantee that an individual piece works as you expect it to... but you don't guarantee how well that piece works with others. You don't guarantee that the piece you designed works towards the purpose you originally conceived the piece for.

[ajvincent]

Well, I've gone ahead and done a first-draft of integration tests for Membranes. Even if it gets rejected overall, it gives us a starting point.

[ajvincent]

See tc39/test262-integration .