Using NBInclude for testing

[cstjean]

I've started using NBInclude for regression testing and it works well so far, since @test is interpreted normally, but I'd like to make a few improvements to nbinclude. NBInclude is currently quite lean and minimal, so the first question is whether you're interested in this at all, or if I should fork it into NBTesting.jl. I'm considering:

• Add a fresh_module=false option, that runs the notebook in a new module
• print_headers=false: print all markdown cells that start with # (useful for long-running tests)
• Perhaps skipping all code cells that start with #NBSKIP. That's already possible via the regex option, but it seems like a useful default.

What do you think? I can do the first and third without modifying NBinclude, so if you OK the second, I could just write the rest as a convenience function on my side.

[stevengj]

I'm skeptical of these; I'd prefer to keep the semantics similar to include. And, as you say, you can already skip #NBSKIP cells with the regex option, but I wouldn't want to make that the default since comments aren't semantically meaningful by default.

As for status updates in long-running tests, why not just add println statements? Or maybe there should be a feature of Base.Test to display a progress meter for testsets?

[cstjean]

Fair enough. I'll experiment on my side.

[simonbyrne]

Since it is not obvious (at least, it wasn't to me) how to construct excluding regexes, it might be a good idea to add it as an example to the README. After quite a bit of googling, trial and error, I ended up using the following (if there are better ways, I would be keen to know).

r"^((?!\#NBSKIP)[\S\s])*$"

[simonbyrne]

Or r"^((?!\#NBSKIP).)*$"s seems to be a bit simpler.