Feature: needimport supports caching and configured needs.json sources

[arwedus]

We've used needextract to show a few needs out of many, many needs imported from some other tool (via needs.json and needimport) in the right place in the documentation.

What bugs me is that the links never go to the place where the needextract is rendered, but always to the location where the original need is defined. This is a problem for the actual use case needextract was made for: If I want to generate a customer documentation (in HTML!) and want to show only the relevant parts of needs to the customer, I would do something like this:

Doc1: Needimport Page
=====================

.. just importing my architecture model or whatever...

.. needimport:: //build/model/architecture_needs.json
   :hide:

And in the actual docu:

Doc2: Customer docu
===================

Component Foo documentation
---------------------------

.. needextract::
   :filter: id == "COMP_FOO"
   :layout: complete

.. needextend:: COMP_FOO
   :+implements: REQ_SW_FOO_123495823

The Foo component does X and Y. It is implemented in ...

What I get with this approach is an error message / extension error at the end of the build process:

Extension error (sphinx_needs.needs):
Handler <function process_creator.<locals>.process_caller at 0x7f1eef8d5160> for event 'doctree-resolved' threw an exception (exception: Need COMP_FOO has no content node.)

Alternatives considered

An alternative would be to "needimport" the same needs.json many times. But I fear that this would not be so performant for my use case (7000 needs). Maybe, if needimport would cache all need and not re-read the same json file twice, this might not be a big build time problem, but I also don't want to have people provide the same needs.json path in 1000 individual documents.

Also, one needs to be super careful with the filters to avoid importing the same needs twice, while at the same time not forgetting to import those needs which are linked but have no dedicated documentation page.

Expectations

In order to not break the current behavior, maybe add an option to set whether a needextract "overrules" the location of the original link.

Also, add a user friendly error message instead of the extension error above.

Needextract "overruling" locations of original definition may bring it's own bunch of non-determinism problems. Unfortunately, attempts to implement it have also not been successful so far (see #689 ).

So let's rephrase the expectations more generally:

I want a directive to include selected imported needs exactly where I need them, with links going to this page. I want to define the path to the data source (needs.json) only once. I want the data source to be read only once.

Solution idea

I have a suggestion to improve the yaaa-needimport:

• Define needimport sources in conf.py centrally, just like needs_external_needs, and reference them by key in the needimport directive.
• Support a cached needimport for centrally configured needimport sources. This way, one can use the needimport directive many times for the same source with different filters, but the needs are only imported from file once.

[danwos]

We have done already some analysis and tests in the past but Sphinx is a beast and has always overwritten or not accepted our changed links. So further work is needed here.

I guess this ticket is a duplicate of #689. So I would like to close this one

[arwedus]

@danwos: Thanks for your quick answer. I already assumed that it would not be easy to do. Also, needextract can appear multiple times for the same need... in this case, it's probably hard to do proper error handling.

I've therefore adapted the issue to consider a new solution approach, improving the featur set of needimport.

[danwos]

Here is a technical proposal for the problem. It shall fix the following problems:

• With the use of multiple needimports, the same JSON file gets loaded each time
• The author must know location of a needs.json file.

Idea

Provide a "importable_objects" database/dict, which holds all importable need-objects and is loaded/filled only once, at the beginning of a sphinx-build.

This is technically related to the external_needs mechanism, but with one important difference: Entries in the "importable_objects" db are not part of the Sphinx-Needs data by default. A need_import is needed to bring this data into the docs.

Spec

Provide a new conf.py option, called needs_importable_data, which is a dict:

needs_importable_data = {
   "my_data": {
       "url": "http://example.com/docs/needs.json" # or use "path"
       "path: "my/docs/needs.json"  # relative to conf.py
       "version": "1.0"  
       "filter": "status == 'open'"
    }
}

filter and version are used as in needimport and are optional:https://sphinx-needs.readthedocs.io/en/latest/directives/needimport.html#needimport

The key is used as an identifier, which can be used like:

   .. needimport:: my_data

The data is separated by the key of needs_importable_data, so a needimport and its filter is only executed on one entry and not on the common data of multiple, importable sources. For this use case, needimport shall be used multiple times.

:warning: The data of needs_importable_data is not available to filters of neetable, needflow and co.

[arwedus]

Hi @danwos , I like the proposal. Please make "path" accept both absolute and relative paths.

Just one open point:

⚠️ The data of needs_importable_data is not available to filters of needtable, needflow and co.

Does this restriction also exist for needimport? Do you mean that the data is not available without a needimport that actually imports it? Or in general not?

[danwos]

Correct, the data is not available as long as it got not imported via needimport.

[chrisjsewell]

Provide a new conf.py option, called needs_importable_data

I don't feel (yet) another configuration option is needed, just add an (optional) key to need_external_needs items, to specify whether the needs should be automatically imported

[arwedus]

@chrisjsewell : The way of importing is conceptually different.

• needs_external_needs: Works like sphinx.ext.intersphinx, lets you link to needs defined in another sphinx project. You need another sphinx project.
• needs_importable_data - which I'd rather call needs_import_sources, @danwos - is a cached source for needimport. And needimport adds a need into your sphinx project at the place of the directive.

This means, needimport and the associated new option (name tbd), is closer to our custom extensions that provide directives to create (import) needs loaded from some files, than it is to needs_external_needs.

[chrisjsewell]

The way of importing is conceptually different.

I'm afraid I disagree; there should be one clear way for users to define external need data sources, it does not need to be any more confusing

Both needs_external_needs and needimport ending up calling exactly the same function to import data into the project. The only difference here, is at what stage that data is loaded, and how you handle source-mapping. There is already a wider area of improvement to deal with source-mapping of need items (and the potential dual nature of their external source location and local import location in the project)

You need another sphinx project.

no this is not the case; you simply need a source of needs data items, which can be anything that adheres to the needs data schema (this is also actually the case for sphinx.ext.intersphinx; I know of multiple non-sphinx tools that generate objects.inv which intersphinx can pull in; in both cases it's just conceptually data with a defined schema that can come from anywhere and does not have to be generated by sphinx)

[arwedus]

@chrisjsewell : I have given you my perspective as a user of sphinx-needs.

With needimport, I'm placing needs in my project, i.e. they appear as original "document nodes". with needs_external_needs, I'm only referencing them and their "document node" appears in another project.

I support the wish to keep sphinx-needs configuration options to the minimum number required, but in this case at least I as a user would be confused.

@danwos : In the end it's your call as a product owner. Now you've heard two perspectives, my user experience as a needs-import/needs_external_needs power user, and @chrisjsewell's perspective with the developer PoV.

[chrisjsewell]

I have given you my perspective as a user of sphinx-needs.

yeh no worries, its good to discuss 👍 and I see your viewpoint as well