needextract referencing external needs raises exception.

[lehaas]

Describe the bug We want to link import requirements from project A to project B. Therefore, we use needs_external_needs to import the needs.json from project A into project B and .. needextract:: to import the requirement. This causes the following exception:

Extension error (sphinx_needs.directives.needextract):
Handler <function process_needextract at 0x00000127A5885160> for event 'doctree-resolved' threw an exception (exception: 'NoneType' object has no attribute 'parent')

With .. needtable:: the build works fine and I can import the requirement.

To Reproduce Steps to reproduce the behavior:

1. Create projectA:

   mkdir projectA
   cd projectA
   sphinx-quickstart 

   I choose not to separate source and build folder during the build, i.e., Separate source and build directories (y/n) [n]: n

2. Configure projectA by adding the following lines to projectA/conf.py:

   extensions = ["sphinx_needs"]
   needs_build_json = True
   version = "1.0"

3. Create a requirement by adding the following lines in projectA/index.rst:

   .. req:: REQ_1
     :id: REQ_1

4. Build the documentation together with the needs.json:

   make html

5. Create projectB:

   cd ..
   mkdir projectB
   cd projectB
   sphinx-quickstart 

   Again, I chose not to separate source and build folder during the build, i.e., Separate source and build directories (y/n) [n]: n

6. Configure projectB by adding the following lines to projectB/conf.py:

   extensions = ["sphinx_needs"]
   needs_build_json = True
   version = "1.0"
   needs_external_needs = [
      {
          "base_url": "../../../projectA/_build/html",
          "json_path": "../projectA/_build/html/needs.json",
          "version": "1.0",
          "id_prefix": "",
      },
   ]

7. Extract REQ_1 from projectA by adding the following lines to projectA/index.rst:

   .. needextract::
     :filter: id == 'REQ_1'

8. Build the documentation

   make html

9. See the following error

   Extension error (sphinx_needs.directives.needextract):
   Handler <function process_needextract at 0x00000127A5885160> for event 'doctree-resolved' threw an exception (exceptio  -packages\sphinx_needs\css\modern\styles.csssn: 'NoneType' object has no attribute 'parent')

Expected behavior We expect the that REQ_1 is included in the build of projectB.

Setup We are using python Python 3.8.13. And the following version of sphinx/sphinx-needs:

sphinx-needs                  1.0.1
Sphinx                        5.1.1

I hope the description helps to reproduce the issue :)

The issue template was copied from labs-zola.

[danwos]

Thanks for the detailed report. It's really helpful. :+1: This one needs a deeper analysis, but I hope to find some time soon.

[PhilipPartsch]

Only from checking the code, without trying to reproduce it: In Project B is the path in base_url and json_path different. I thought they should mostly be the same, especially the "../"

[twodrops]

@PhilipPartsch A needtable shows both Project A and Project B data, only needextract does not work, Also the external links are created correctly within the needtable and the navigation across project works. So it has to be something with needextract right.

[danwos]

Sorry to say, but needextract will not work with external_needs, because external needs have no docutils/html based representation, so no content got already rendered by sphinx, which we can reuse/copy for the need-copy defined by needextract.

The only solution is to use needimport and allow it to download the needs.json file from external sources. But this would be a new feature ;)

[lehaas]

@danwos, thanks a lot for the update :) Sad to hear that it cannot be fixed..

I interpret your proposal to download needs.json from external sources as a mere download of needs.json and using it as it can be used now, in case I have the needs.json locally. For me that would result in the following drawbacks compared to the use of needextract:

• in case only parts of the requirements are imported (i.e., using :filter:), then all the requirements linked within (i.e. using :links:) will be dead links.
  • One option might be to import the entire needs.json in one place and afterwards use needsextract to link the included requirements. But that results in the same error as above, probably for the same reason.
• It is not possible to jump to the external needs project from which the requirement is imported and which might contain additional information outside of the .. req:: definition or related requirements.

What is your opinion on the points above? Can the be enabled with needimport? If you see a way, why is it not possible to extend needextract by the feature to use external_needs?

[arwedus]

Hi @danwos ,

I have the same issue but with a different extension in sphinx-needs 2.1.0a (a pre-build mostly similar to sphinx-needs 2.1) in our repository with a multi-project setup.

Project A:

.. needextract::
   :filter: (type == 'test-sw') and ("TAP" in id)
   :layout: clean

project B:

.. test-sw:: caption
   :id: TC_SW_TAP_284351
   :da_status: draft
   :verifies: REQ_SW_TAP_284351

   bla...

configuration:

Project A references the needs.json of project B via needs_external_needs.

The extension error we get is:

all_needs_generator - INFO - All needs successfully exported

Extension error (sphinx_needs.needs):
Handler <function process_creator.<locals>.process_caller at 0x7f9095a5a280> for event 'doctree-resolved' threw an exception (exception: Need TC_SW_TAP_284351 has no content node.)

This exception appears only at the end, after all sphinx write processes (we use parallel build in this example) have been joined.

Since I understood you that needextract simply can't be used to print a verbatim copy of needs that have been defined in a different project and are only available as reference targets via needs_external_needs mechanism, I'd request you to catch this exception and provide a helpful warning message, instead of triggering an exception.

The detailed exception I get (with sphinx-build -v) is:

Traceback (most recent call last):
  File "/home/.../site-packages/sphinx/events.py", line 96, in emit
    results.append(listener.handler(self.app, *args))
  File "/home/.../site-packages/sphinx_needs/needs.py", line 350, in process_caller
    check_func(app, doctree, fromdocname, current_nodes[check_node])
  File "/home/.../site-packages/sphinx_needs/directives/needextract.py", line 116, in process_needextract
    need_extract = create_need(
  File "/home/.../site-packages/sphinx_needs/debug.py", line 70, in wrapper
    return func(*args, **kwargs)
  File "/home/.../site-packages/sphinx_needs/layout.py", line 78, in create_need
    assert content_node is not None, f"Need {need_id} has no content node."
AssertionError: Need TC_SW_TAP_284351 has no content node.