search

qunitjs / js-reporters

📋 Common Reporter Interface (CRI) for JavaScript testing frameworks.
MIT License
60 stars 18 forks source link

Specify data for Todo tests #94

Closed trentmwillis closed 7 years ago

trentmwillis commented 7 years ago

This proposes adding a new status of todo for both Suites and Tests. It is similar to the skipped status in that it should be counted separately from passed tests, but is also different because it is possible that todo tests can also fail.

todo should represent tests where there is some amount of incomplete work, meaning that there is at least one failing assertion in the test. These assertions should still be reported as normal, but should not cause a todo test to fail. A todo only fails when all assertions have passed. In that case, the status should be reported as failed so that reporters can properly notify that the feature is now implemented.

Original discussion occurred in https://github.com/qunitjs/qunit/pull/1006.

cc @leobalter

jzaefferer commented 7 years ago

The spec change looks alright. It would be nice to get the adapters updated as well, which would also tell us if any other framework has a similar feature. Considering that this spec still has almost zero traction, it might be a waste of time though.

trentmwillis commented 7 years ago

As far as I can tell, no framework actually implements "todo" tests. Mocha and Jasmine have "pending" tests, but those are essentially the same as "skipped" tests (just no callback defined). Thus, I don't think updating the adapters at this point in time makes much sense (since we don't know what we're adapting).

trentmwillis commented 7 years ago

Update: we're about to land todo tests in QUnit and have work in progress to land js-reporters shortly after that. In which case, we will be using the approach documented in this PR unless any objections get raised.

jzaefferer commented 7 years ago

Could you add a note to the spec that todo is something new in QUnit, along with a link to the docs or something?

trentmwillis commented 7 years ago

@jzaefferer I've updated this with more documentation for todo. I also added the data to the TapReporter, the Data helpers, and fixed CI.

That said, I still need to update the QUnitAdapter now, but I'd prefer to do that in a follow up PR to avoid this one getting too complex (though it might already be there).