Use tape and testling, instead of mocha

[ljharb]

@paulmillr There have been a number of PRs lately that have broken tests - can we get Travis CI set up for this repo, to avoid obvious breakages?

[paulmillr]

yeah we should

[ljharb]

As the repo owner, you've got to do the travis setup. I can handle travis.yml etc as necessary once that's done :-)

[paulmillr]

done

[paulmillr]

do we really need browser phantomjs tests? isn’t just node ok?

[ljharb]

Absolutely not! Every browser will have a different JS engine, and each version might fail in subtly different ways. To be a proper shim, it needs to work in as close to 100% of environments as possible.

[cscott]

PhantomJS is just a headless webkit. That probably adds minimal value, above the node tests (since both are v8-based). It would be better to use something which could actually test against Mozilla, IE, etc. testling seems to do that (but it's slow and a bit flakey).

[ljharb]

Renamed this to reflect that we should switch away from mocha (for a number of reasons: not working in browsers we need to shim is a big one, plus, not working in testling), and then once we're using tape, use testling-ci/browserify to do automatic browser testing.

[paulmillr]

mocha supports tape

[paulmillr]

sorry, not tape but TAP — test anything protocol. testling-ci / browserify should work great with it

[ljharb]

That's true - but it also requires ES5 features, which makes it inappropriate to use in a library that targets ES3 browsers as well.

[paulmillr]

Question: Do we actually want to support ES3 browsers? Their market share is like super low

[ljharb]

I'd say so. As a shim, the goal is to patch the oldest possible implementations as much as possible. I think we should support every single browser we can test on, as well as every version of node that's reasonable, and that definitely includes ES3 browsers.

IE 8 won't be out of common usage for a very long time - 23% (according to http://www.netmarketshare.com/report.aspx?qprid=3&qpaf=&qpcustom=Microsoft+Internet+Explorer+8.0&qpcustomb=0 ) is pretty high.

[ljharb]

fwiw I'm happy to do the work to convert the tests :-) I've already got a branch in-progress with more than half of them converted anyways. It's not much of a burden to keep the support threshold super low, so we might as well. BDD-style test libs don't add much value imo anyways :-)

[paulmillr]

I personally love BDD. It's very easy to read code. @cscott ?

[cscott]

FWIW, mocha is working, and it is working in saucelabs. I don't see any reason for changing now for the sake of it.

On the other hand, shifting to test262 is a good idea. But it seems like we already have a test262-to-mocha bridge via @smikes work.

[smikes]

Planned -- not yet delivered. Probably will get some attention this weekend :-)

[paulmillr]

tape is shit. I suggest using micro-should: https://github.com/paulmillr/micro-should

[ljharb]

That's a pretty strong statement without any backing for a package I maintain :-) I use tape on 350+ packages, it works great.

[ljharb]

I'd love to hear more on the tape repo about what you find lacking.

[ljharb]

(also, micro-should uses const syntax, which means it can't run on most the engines that actually need es6-shim, so it's a bit of a nonstarter)

[paulmillr]

strong statement without any backing

Having 300 packages use it and being time-proven is great.

However 21+ dependencies for a test runner is a non-starter. Supply chain attacks are real. Any dependency update will bring potential malware to users.

Micro should is 0-dep 150 loc, third of which is parallel running which should be in an optional separate file.

The need to support ES3 environments is genuine though, micro-should won't help here.

Tape needs to drastically reduce dep count.

[ljharb]

The number of deps doesn’t affect the supply chain risk, and the risk for a test runner is virtually zero. More deps is better, and supply chain attacks are a wildly overblown attack vector.

Either way, mocha has far more deps than tape :-) (you also may want to check how much of tape’s graph i personally maintain)

[paulmillr]

Either way, mocha has far more deps than tape

If you maintain most tape deps personally, mocha indeed seems worse.

supply chain attacks are a wildly overblown attack vector

Different perspective. I would argue the opposite: that it's not as well-known as it should be. I work in infosec. There were lots of rogue NPM packages stealing SSH / cryptocurrency / environment keys. Any package could basically steal it.

and the risk for a test runner is virtually zero.

The risk for a project which handles hundreds of millions of dollars and uses test runner which has rogue deps is huge.

[ljharb]

I work in it as well; I was on the board of the OpenSSF last year. There were not, in fact, lots of these incidents - there were tons of proofs of concept, but there were a very small number of actually exploited packages.

If you're concerned with exfiltration, the mitigation is to prevent network access, which has no relationship to the number of deps you have. Separately, this package has nothing to do with handling money (most packages don't, statistically), so I'm not sure how that concern would be relevant here?

[smikes]

Did I never deliver the promised integration? Bummer. The internet remembers too well…

In any case, what is the problem we are solving here?

[ljharb]

@smikes in particular, we're stuck on mocha 3, and mocha is on v10, because mocha 4+ dropped support for engines we support; tape latest, however, eternally supports what we do.

[paulmillr]

If you're concerned with exfiltration, the mitigation is to prevent network access

Preventing network access is not an option in network apps. Also it's barely a mitigation.

It is always possible for the malware to execute a 0day attack and do something terrible to your machine. They could install a silent trojan you won't find. They could infect binaries your machine would be producing. They could gain root access, infect kernel or rewrite firmware on your hardware.

Separately, this package has nothing to do with handling money (most packages don't, statistically), so I'm not sure how that concern would be relevant here?

You don't know which developers would use your software. They could or could not be working with money. It's best to assume they will.

It's also best to assume every package is bad and contains malware. That's why auditability and reducing attack surface is important.

[ljharb]

The test runner of this package can't possibly affect consumers, so the only danger would be to developers of this package.

[paulmillr]

Developers of a package: 1) produce binaries or packed apps for end-users, the malware could be silently injected into binaries 2) can have sensitive info on their machines

[ljharb]

es6-shim is never packed or a binary produced by developers of es6-shim. Certainly consumers of es6-shim may do so, but that has no bearing on dev deps.

[paulmillr]

I'm not talking about es6 shim specifically here. I'm talking about using tape vs something else in general. For es6 it may pose less relevance.

[smikes]

In summary:

Micro-should: 0-deps but doesn’t work in ES3 environments (const)

Tape: currently in use, actively maintained

Mocha: everyone prefers to be off mocha

Paul, do you have any appetite for making an ES3-friendly micro-should? Nano-should?

[paulmillr]

I think it's very easy to just fork and place a es3 fork into the es6shim repo. No need to complicate the original repo.

[paulmillr]

I would want a consensus of maintainers tho. Maybe tape has some other benefits

[ljharb]

I don't think optimizing for fewer dependencies helps security; my experience tells me the opposite, and given the common test runners in the ecosystem, tape is the only one that's sufficiently used and also retains backwards compatibility with older environments.

[paulmillr]

It actually matters less for es6-shim and I don't maintain the library, so i'm deferring the decision to current maintainers.

The only reason i've closed the issue is: I was walking through personally created old issues and closing everything.