search

treebeardtech / nbmake

📝 Pytest plugin for testing notebooks
https://pypi.org/project/nbmake/
Apache License 2.0
179 stars 18 forks source link

Does the cell "raises-exception" tag actually work? #111

Closed mmcdermott closed 11 months ago

mmcdermott commented 11 months ago

Describe the bug I have a notebook I am testing with nbmake, and I have a cell that throws an exception, and I have added the documented metadata tag, but the test fails.

Additional context I also took a look in the code, and while I am not familiar with the codebase, I'm not seeing code that is looking for that metadata tag. Is this feature actually enabled in the current branch? Thank you for any help you can provide.

alex-treebeard commented 11 months ago

Hey @mmcdermott, thanks for reporting a suspected issue. The docs may be wrong here.

This is implemented and enabled.

If it still doesn't work please send over pip list output, otherwise we probs need to fix the docs.

mmcdermott commented 11 months ago

HI @alex-treebeard ; Wow, sorry, I totally missed that spot in the code! I tried searching for "raises-exception" but must have misspelled it or github's search is just poor. My issue is likely somehow with the surrounding components then (pytest, mainly). I'll try to compile a minimum working example; the actual issue I encountered was with the notebook modified in this commit: https://github.com/mmcdermott/EventStreamGPT/commit/422e57a69be70d2ea3422bcef3ffdb805540b3d7#diff-35a419cb3e0104ad34683e0639c82142f3f30043b7b9032f7a49ed86df0e7a93 but there are a lot of parts there so I'll try to simplify.

The version I'm using is nbmake 1.4.3, with pytest 7.4.0

mmcdermott commented 11 months ago

Ok, I figured out the issue. I think maybe it is related to a jupyter version, or just how cell metadata is edited in general? In any case, based on the test notebook you linked, I realized that the metadata I added included a "custom" tag outside of the metadata (like how it is shown in the README of this repo). But I was editing the metadata via the sidebar in Jupyter Lab, and I think that custom is not actually something that is needed. So, I tried removing it and just dumping in the metadata directly, and that worked.

In particular, I first added: 

  "custom": {
    "metadata": {
      "tags": [
        "raises-exception"
      ]
    }
  }

to the cell's metadata, and the test failed, but I just tried adding

  "metadata": {
    "tags": [
      "raises-exception"
    ]
  }

and now the test passes. So I think it is just a my knowledge issue of cell metadata, though it might help to update the README documentation to be a bit clearer. In any case, apologies for the unnecessary issue (and for missing the implementation of that function!) and thank you for the quick turnaround and the great tool!