treebeardtech / nbmake

📝 Pytest plugin for testing notebooks
https://pypi.org/project/nbmake/
Apache License 2.0
189 stars 18 forks source link

Notebook tests don't have names in pytest junit reports #127

Open hulsed opened 1 month ago

hulsed commented 1 month ago

Describe the bug Notebook tests are not being given names in junit reports. This makes it difficult to differentiate different tests in CI tools like bamboo that parse junit.xml to determine test names.

To Reproduce Steps to reproduce the behavior:

  1. From any repo, such as nbmake-examples, run nbmake from pytest and have it generate a junit report, e.g., "pytest --junit-xml=junit.xml --nbmake"
  2. Open the junit report for a notebook test and look at the "name" property for each test, which will be blank.

Expected behavior Names should be populated with something similar to "classname".

Screenshots Notebook tests (no names) junit output from nbmake-examples (notice the lack of names given for each test): image

Example of what doctests produce for names: image

How nameless tests show up in bamboo (no info on which test this was): image

Desktop (please complete the following information):

Additional context I believe a very easy fix for this would be to simply pass a name from the NotebookItem class during initialization, by changing pytest_items.py from:

  class NotebookItem(pytest.Item):
      nbmake = True

      def __init__(self, "", filename: str):
          super().__init__("Notebook: "+parent, parent)
          self.filename = filename

To:

  class NotebookItem(pytest.Item):
      nbmake = True

      def __init__(self, parent: Any, filename: str):
          super().__init__("Notebook: "+parent, parent)
          self.filename = filename
alex-treebeard commented 2 weeks ago

hey @hulsed thanks for taking the time to write up this issue - I'm scheduling a bit of time to take a look and hopefully fix.