How to guide for testing

[hamelsmu]

A guide that shows how we test notebooks:

• fastcore.test
• How to write unit tests in notebooks
• test flags
• How to test nbdev_test, and how to interpret the outputs esp if there is a failure
• Recommendations for style
• How this may be different than pytest etc

[tcapelle]

This would be really cool! Currently, nbdev_test does not produce any interpretable output, only lists the failing notebooks without the error message (like nbdev1 used to).

[seeM]

@tcapelle it’s supposed to print each error and corresponding cell source. I looked into the nb_helpers repo, in case that’s what you’re referring to, and the errors aren’t printed there because the RichLogger class reconfigures the root logger with level=logging.ERROR, and nbdev_test use warning which logs at level WARNING which is lower than ERROR

cc @hamelsmu @jph00 - maybe we want to change the way we print those errors?

[hamelsmu]

@seeM I thought maybe you set the logging level to error or were in the process of doing that? Or is that for something else?

[seeM]

@tcapelle could you try again with latest master and see if nbdev_test produces interpretable output for you?

[seeM]

@hamelsmu do you think this is sufficiently covered in https://nbdev.fast.ai/tutorials/best_practices.html?

[hamelsmu]

No I don't think so, I think we can talk a lot more about tests esp:

• Difference from pytest (ex: why you don't need fixtures)
• test flags
• When to incorporate tests into visible documentation and when not to

[tcapelle]

Ohh missed this! thanks guys, and I actually figured this out :P

[Jacobluke-]

Hi! I'm very interested in developing a plotting package using nbdev. However, most graphical testing plugins are for IDE development. Do you have any suggestion on graphical testing in notebooks?

[seeM]

Sounds interesting @Jacobluke-! I'm not familiar with "graphical testing", could you please share some graphical testing plugins you've enjoyed using?

[Jacobluke-]

Hi @seeM , thanks for your reply.

Basically "graphical testing" ensures that our package continue to work in a consistent way. The plugin we are using is pytest-mpl, and here is a brief example on how to using this plugin.

[hugetim]

One pitfall I've run into that may be worth addressing in such a guide: when I forget to export functions (or imports), tests will still pass because the non-exported stuff is loaded in the notebook. I guess putting some tests in a separate notebook would guard against this?

[jph00]

Message ID: @.***>I'm not sure I fully understand your question/concern. If you run nbdev_prepare or nbdev_test it should test correctly, shouldn't it?

[jph00]

Message ID: @.***>Oh sorry I understand what you're saying now. Yes that can be an issue, although if you're exporting something that you never actually use anywhere else, then hopefully it's not a big problem! ;)

[hugetim]

Right, it is often caught in a test in a separate notebook if it is used elsewhere, but I won't always have exhaustive test coverage elsewhere if I think its tested in the source notebook. To be more specific about the way this has bit me, it's usually a private helper function or global variable defined in the module scope that I forget to #|export, not something that would be directly called from elsewhere.