search

flowbased / fbp-spec

Data-driven testing of FBP components and graphs
MIT License
32 stars 4 forks source link

Test coverage support #129

Open bergie opened 6 years ago

bergie commented 6 years ago

It would be nice to be able to get test coverage reports from fbp-spec. Something like https://www.npmjs.com/package/nyc

bergie commented 6 years ago

Note: nyc claims suppor for applications that spawn subprocesses so fbp-spec might be possible to make work.

jonnor commented 6 years ago

Does nyc produce a report file somewhere from running the tests? Have you tried wrapping the runtime command in it to see if it just works?

jonnor commented 6 years ago

Generally this is something that needs to happen on the runtime side, and in a runtime-specific manner, as how to compute coverage depends entirely on the language/execution-environment in use.

bergie commented 6 years ago

@jonnor tried wrapping fbp-spec (via its mocha interface) to nyc, and at least that didn't work:

https://github.com/noflo/noflo-core/pull/108

Getting zero coverage for the components we test via fbp-spec (but normal coverage for the ones we test via Mocha)

jonnor commented 6 years ago

@bergie what do you mean does not work? Looks fine here

noflo-core-nyc-coverage

bergie commented 6 years ago

@jonnor yes, see zero coverage reported for Repeat and RepeatAsync, which are fbp-spec tested. Everything else is tested via plain Mocha

jonnor commented 6 years ago

@bergie ok. Does not work when running nyc infront of fbp-spec either, does not pick up any files at all. So either subprocess tracking is off, it cannot handle the indirect file loading (though noflo.ComponentLoader seems to work for other), there is a problem with .js versus .coffee files or similar

bergie commented 6 years ago

@jonnor for .coffee files you will need to require coffee-coverage

See https://github.com/benbria/coffee-coverage/blob/master/docs/HOWTO-nyc.md

jonnor commented 6 years ago

Presumably that needs to be done in the process which uses the coffee files. Which would explain why it is not working in fbp-spec case: It is the runtime that requires the files (not fbp-spec), and there is no such registration there. noflo-nodejs does not have such an option either, I think?

bergie commented 6 years ago

@jonnor correct, we should probably add a mocha-style --require option to noflo-nodejs.

But first to verify this, it might make sense to add this manually to noflo-nodejs

Update: tried patching that to noflo-nodejs locally, and still got zero coverage