Decide whether to show included spec fields

bendichter commented 6 years ago

make apidoc gives the following error:

Bens-MacBook-Pro:docs bendichter$ make apidoc
PYTHONPATH=/Users/bendichter/dev/nwb-extensions/simulation_output/docs/nwb_docutils:/Users/bendichter/dev/nwb-extensions/simulation_output/docs/source:/usr/local/hdf5: python /Users/bendichter/dev/nwb-extensions/simulation_output/docs/nwb_docutils/generate_format_docs.py
/Users/bendichter/dev/nwb-extensions/simulation_output/docs/nwb_docutils/generate_format_docs.py:60: UserWarning: DISABLING RENDERING OF SPEC GRAPHS DUE TO IMPORT ERROR
  warnings.warn('DISABLING RENDERING OF SPEC GRAPHS DUE TO IMPORT ERROR')
Generating output directory: /Users/bendichter/dev/nwb-extensions/simulation_output/docs/source/_format_auto_docs
Traceback (most recent call last):
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/spec/namespace.py", line 379, in load_namespaces
    inc_ns = self.get_namespace(s['namespace'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 348, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/spec/namespace.py", line 240, in get_namespace
    raise KeyError("'%s' not a namespace" % name)
KeyError: "'core' not a namespace"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/bendichter/dev/nwb-extensions/simulation_output/docs/nwb_docutils/generate_format_docs.py", line 1356, in <module>
    main()
  File "/Users/bendichter/dev/nwb-extensions/simulation_output/docs/nwb_docutils/generate_format_docs.py", line 1263, in main
    resolve=spec_resolve_type_inc)
  File "/Users/bendichter/dev/nwb-extensions/simulation_output/docs/nwb_docutils/generate_format_docs.py", line 88, in load_nwb_namespace
    namespace.load_namespaces(namespace_file, resolve=resolve)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 348, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/spec/namespace.py", line 381, in load_namespaces
    raise ValueError("Could not load namespace '%s'" % s['namespace'])
ValueError: Could not load namespace 'core'
make: *** [apidoc] Error 1

jcfr commented 6 years ago

Thanks for reporting the issue.

What are the steps leading to this error ?

As a side note, you can now pip install nwb-docutils, this will make available these command line tools

bendichter commented 6 years ago

nwb_init_sphinx_extension_doc --project simulation_output --author "Ben Dicher and Kael Dai" --version "0.1.0" --release alpha --spec_dir ~/dev/nwb-extensions/simulation_output/spec/ --namespace_filename ~/dev/nwb-extensions/simulation_output/spec/simulation_output.namespace.yaml --default_namespace simulation_output --external_description ~/dev/nwb-extensions/simulation_output/docs/description.rst --external_release_notes ~/dev/nwb-extensions/simulation_output/docs/release_notes.rst --output ~/dev/nwb-extensions/simulation_output/docs

error:

cp: /Users/bendichter/anaconda3/lib/python3.6/site-packages/nwb_docutils/../../Legal.txt: No such file or directory
Copy of utils dir failed: ['cp', '/Users/bendichter/anaconda3/lib/python3.6/site-packages/nwb_docutils/../../Legal.txt', '/Users/bendichter/dev/nwb-extensions/simulation_output/docs/utils']

make apidoc

error:

PYTHONPATH=/Users/bendichter/dev/nwb-extensions/simulation_output/docs/utils:/Users/bendichter/dev/nwb-extensions/simulation_output/docs/source:/usr/local/hdf5: python /Users/bendichter/dev/nwb-extensions/simulation_output/docs/utils/generate_format_docs.py
python: can't open file '/Users/bendichter/dev/nwb-extensions/simulation_output/docs/utils/generate_format_docs.py': [Errno 2] No such file or directory
make: *** [apidoc] Error 2

That's just a path error. It should be nwb-extensions/simulation_output/docs/nwb_docutils/generate_format_docs.py

But when I fix that:

apidoc

error:

Bens-MacBook-Pro:docs bendichter$ make apidoc
PYTHONPATH=/Users/bendichter/dev/nwb-extensions/simulation_output/docs/nwb_docutils:/Users/bendichter/dev/nwb-extensions/simulation_output/docs/source:/usr/local/hdf5: python /Users/bendichter/dev/nwb-extensions/simulation_output/docs/nwb_docutils/generate_format_docs.py
/Users/bendichter/dev/nwb-extensions/simulation_output/docs/nwb_docutils/generate_format_docs.py:60: UserWarning: DISABLING RENDERING OF SPEC GRAPHS DUE TO IMPORT ERROR
  warnings.warn('DISABLING RENDERING OF SPEC GRAPHS DUE TO IMPORT ERROR')
Generating output directory: /Users/bendichter/dev/nwb-extensions/simulation_output/docs/source/_format_auto_docs
Traceback (most recent call last):
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/spec/namespace.py", line 379, in load_namespaces
    inc_ns = self.get_namespace(s['namespace'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 348, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/spec/namespace.py", line 240, in get_namespace
    raise KeyError("'%s' not a namespace" % name)
KeyError: "'core' not a namespace"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/bendichter/dev/nwb-extensions/simulation_output/docs/nwb_docutils/generate_format_docs.py", line 1356, in <module>
    main()
  File "/Users/bendichter/dev/nwb-extensions/simulation_output/docs/nwb_docutils/generate_format_docs.py", line 1263, in main
    resolve=spec_resolve_type_inc)
  File "/Users/bendichter/dev/nwb-extensions/simulation_output/docs/nwb_docutils/generate_format_docs.py", line 88, in load_nwb_namespace
    namespace.load_namespaces(namespace_file, resolve=resolve)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 348, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/spec/namespace.py", line 381, in load_namespaces
    raise ValueError("Could not load namespace '%s'" % s['namespace'])
ValueError: Could not load namespace 'core'
make: *** [apidoc] Error 1

oruebel commented 6 years ago

@bendichter the init_sphinx_extension_doc script needs to be updated to not copy the sources that are now in nwb-docutils but rather to use nwb-docutils directly.

jcfr commented 6 years ago

I just push some fixes to nwb-docutils that account for the fact script are not anymore located in the nwb-schema/docs/utils.

There is no intermediate errors anymore. Here are the steps so far:

Install from source:

pip install git+https://github.com/NeurodataWithoutBorders/nwb-docutils

Then:

mkdir -p /tmp/scratch && cd $_

git clone git@github.com:NeurodataWithoutBorders/nwb-extensions.git
cd  nwb-extensions/extensions/simulation_output

nwb_init_sphinx_extension_doc \
      --project test \
      --author "Dr. Master Expert" \
      --version "1.2.3"  \
      --release alpha    \
      --output my_extension_docs   \
      --spec_dir .  \
      --namespace_filename simulation_output.namespace.yaml   \
      --default_namespace simulation_output

cd my_extension_docs
make apidoc

The following error is reported:

Generating output directory: /tmp/nwb-extensions/extensions/simulation_output/my_extension_docs/source/_format_auto_docs
Traceback (most recent call last):
  File "/home/jcfr/Projects/pynwb/src/pynwb/form/spec/namespace.py", line 379, in load_namespaces
    inc_ns = self.get_namespace(s['namespace'])
  File "/home/jcfr/Projects/pynwb/src/pynwb/form/utils.py", line 348, in func_call
    return func(self, **parsed['args'])
  File "/home/jcfr/Projects/pynwb/src/pynwb/form/spec/namespace.py", line 240, in get_namespace
    raise KeyError("'%s' not a namespace" % name)
KeyError: "'core' not a namespace"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jcfr/.virtualenvs/pynwb-py35/bin/nwb_generate_format_docs", line 11, in <module>
    load_entry_point('nwb-docutils', 'console_scripts', 'nwb_generate_format_docs')()
  File "/tmp/nwb-extensions/extensions/simulation_output/my_extension_docs/nwb_docutils/generate_format_docs.py", line 1263, in main
    resolve=spec_resolve_type_inc)
  File "/tmp/nwb-extensions/extensions/simulation_output/my_extension_docs/nwb_docutils/generate_format_docs.py", line 88, in load_nwb_namespace
    namespace.load_namespaces(namespace_file, resolve=resolve)
  File "/home/jcfr/Projects/pynwb/src/pynwb/form/utils.py", line 348, in func_call
    return func(self, **parsed['args'])
  File "/home/jcfr/Projects/pynwb/src/pynwb/form/spec/namespace.py", line 381, in load_namespaces
    raise ValueError("Could not load namespace '%s'" % s['namespace'])
ValueError: Could not load namespace 'core'
Makefile:170: recipe for target 'apidoc' failed
make: *** [apidoc] Error 1

jcfr commented 6 years ago

@oruebel A new version of nwb-docutils was just published on pypi, the new version doesn't copy the source of utils into the extension.

jcfr commented 6 years ago

In the process we understood the following:

(1) Addressing the error removing the reference to `core` namespace

Keeping the reference to the core schema leads to the error posted above. Removing the line with - namespace: core from allows to generate documentation:

(2) Keeping the reference to `core` and generating the documentation

This can be done by apply the patch associated with pull request #5

questions and summary

Which approach do we want to promote for extension documentation ?

If we decide to go with (1), we would need to:

(a) fix the documentation generator script to ignore the core namespace or
(b) fix the namespace builder export function

I think going with 1.a would be more sensible. We should also probably leverage inter-sphinx capabilities to ensure references to core schema documentation are solved.

See http://www.sphinx-doc.org/en/stable/ext/intersphinx.html#module-sphinx.ext.intersphinx

What about dependent extensions ?

To ensure extension A that depends on extension B can effectively resolve references to the documentation of extension B, it would need to know how to find the corresponding documentation.

Few approaches are possible:

assume extensions are named consistently and get documentation from readthedoc

or

include the documentation link in the generated yaml file

oruebel commented 6 years ago

In both cases (1) and (2) the document generator needs to be fixed because the docs won't link back properly to the docs if you remove the 'core' from the schema.

At least for now, I think we can simplify this problem by focusing on what is needed for the 'core' docs from the nwb-schema repo and the extension docs from the new nwb-extensions repo.

If we want to render only the parts that are new to the extension then a possible solution would be to:

Updated SchemaHelper.sort_type_hierarchy_to_sections to filter out the types we don't want to show
Add a corresponding flag to conf_doc_autoge (i.e., the configuration file that is being used to configure generate_format_docs)
Update all calls in generate_format_docs where links between sections of the document are being created. This should be all places where the "get_reference" method is being called. This should be replaced by a new function that in addition to internal reference within a Sphinx doc can then also create links to types that are documented in other external docs.

Note, there is also an additional variant to this, in that we can set spec_resolve_type_inc setting for extensions to True. In this case the docs will be always fully resolved (instead of just showing what is new). E.g., if you create an extension of TimeSeries this would show all fields inherited from TimeSeries plus all the changes from the extensions, i.e., it would show the full, final spec not just the parts that have changed.

bendichter commented 6 years ago

I like the idea of setting spec_resolve_type_inc=True. In fact, I like that in general, as opposed to requiring the user to follow links back through the documentation.

I have 2 example extensions up, each with documentation rendered, including the entire core. Right now it's difficult to know which classes belong to the extension, so I think limiting what is shown to new extension classes would be very useful.

hdmf-dev / hdmf-docutils