tudat-multidoc
This repository contains the docstring source for the tudatpy
repository. This repository contains a description of the Tudatpy
API, which is rendered here.
tudat-multidoc
In the tudat-multidoc
directory, there are two subdirectories:
tudat-multidoc/docstrings
, where the documentation is actually written in YAML files (this constitues the "source"
of the API system). Instructions on how to write the docstrings and on the structure of such directory
are provided in the dedicated docstrings/README.md
file.tudat-multidoc/multidoc
, where the software to build the API is located. The content of this repository
should not be modified. The branches used to write the documentation are the following (listed for each repo):
tudat-bundle
: maintudat
: developtudatpy
: developtudat-multidoc
: maintudat-multidoc/multidoc
: developIf you are writing the API for a specific module/project, you are recommended to create a new branch and then open a pull request.
tudatpy
Building the API documentation requires a compilation of tudatpy. A normal local compilation of tudatpy is done using the tudat-bundle repository. To build the documentation, start by following steps 1, 2 and 3 of the tudat-bundle setup
In step 3, make sure that tudat-bundle and its submodules are on the following branches:
tudat-bundle
: maintudat
: developtudatpy
: developtudat-multidoc
: maintudat-multidoc/multidoc
: developAs with the regular tudat-bundle, building tudatpy with API documentation requires a conda environment.
The associated environment.yaml
file can be found in this repository here.
tudat-multidoc
conda environment:conda env create -f environment.yaml
tudat-multidoc
environment:conda activate tudat-multidoc
The default procedure for building tudatpy with API documentation uses the terminal. The build procedure is
defined (on Mac/Linux/Windows WSL), using the build.sh
file of tudat-bundle
(see here).
This file can be left unchanged for the procedure here. However, for improved compilation speed, the files final line can be changed from:
cmake --build .
to:
cmake --build . -jX
replacing X with a number defining the number of parallel threads to use during the compilation (for instance -j8 for 8-thread compilation).
Running the following command:
python cli a
will then commence the compilation of tudatpy as along with the generation of the API documentation. In the sections that follow, the details of the underlying procedure are discussed
A set of command-line interface tools have been made to automate the building of tudat(py) and its API documentation.
The cli
command must be executed
tudat-multidoc
environment activated and.multidoc.cfg
file (e.g., in the tudat-bundle
directory).This file provides all of the configurable arguments for the cli
tool. The variables are fairly straightforward.
PLEASE NOTE: Ensure (1) current working directory is
tudat-bundle
, (2) acli
directory is present in your version of thetudat-bundle
and (3) a.multidoc.cfg
file exists in root oftudat-bundle
.
Below gives four different commands that you will be using in your workflow.
# the scope of this tool
python cli d # document [1]
python cli b # build [2]
python cli s # sphinx [3]
python cli a # all [*]
The subcommands, their scopes and their required order of execution are summarised by the following:
[document/d]
Generating documented versions of project sources.[build/b]
Building the tudat-bundle
and its contained project sources.[sphinx/s]
Building the sphinx
API documentation for the builds of the tudat-bundle
subprojects.[all/a]
Executes 1,2 and 3 in that order.After running python cli d
, the following directories will appear in your tudat-bundle
directory:
tudat-bundle/.tudatpy-documented
It is listed under tudat-bundle/.gitignore
, so it will
not be tracked. This repository is essentially a copy of the original tudatpy repository,
with the addition of the docstrings (see below). It constitutes the source from which the
subsequent build of tudatpy will be done.
PLEASE NOTE: The
.tudatpy-documented
directory will be overwritten every time thepython cli d
command is executed. Therefore, if you want to make changes to the project source code, do NOT do it in this folder, but in the original directories, then re-run the documenting command.
This is what happens in this step: the docstrings located in the
tudat-multidoc/docstrings
directory are parsed, validated, formatted (currently, in NumPy style) and
placed in a C++ function. This is located in:
.tudatpy-documented/include/tudatpy/docstrings.h
for tudatpyThis function will be used during the next step (the build step) to link each docstring to the related C++/Python object.
After running the python cli b
, the projects are built from the "documented" directory listed above.
The output of the build will be located in the cmake-build-release
folder (or similar, depending on your system).
For Python, every binary object generated in this phase will have its own __doc__
attribute generated
from the raw docstrings. It is possible to check this by importing the kernel (located at
cmake-build-release/.tudatpy-documented/tudatpy/kernel.so
) into Python. From there, it is sufficient to import
a certain function or class object to check the content of its __doc__
attribute.
In this step, the Sphinx source code is also generated and it is located in:
cmake-build-release/.tudatpy-documented/docs
for tudatpyThese are .rst
files that will be used by Sphinx in the next step as source files.
These source files exploit the Sphinx autoclass
and autofunction
commands to extract the docstring
from each Python object by accessing their __doc__
attribute, similarly as it was explained above.
After running the python cli s
, the sphinx html files are generated. These will be located in two directories:
tudat-bundle/.docs-output/tudatpy
for tudatpyTo check the output of the html files, it is recommended to open the index.html
file with your preferred browser
(or, if you are using CLion, with CLion's built-in html viewer) and navigate through the various html pages from there.
The API docs reside at https://py.api.tudat.space, where the stable
version (see menu at lower left of this page)
contains the docs of the tudatpy master branch, and the latest
version of the tudatpy develop branch. The docs are rebuilt everytime a new tudatpy conda
package is succesfully built (see here). The progress and log of the online docs
build can be found here. After merging a pull request into main
of tudat-multidoc
, this will
automatically be processed into the new version of the online API docs when a new build is triggered.
Check if the environment.yaml
file inside tudat-bundle
contains the missing dependency (you may have the environment installed from an older file). Not present in there? Please post the issue here, or add the dependency yourself.
Please see the section below on Some Notes/FAQ. If your issue is not mentioned there, or the templates are not as they should be yet, please post an issue on the multidoc
repository here.
As a developer contributing to the yaml
source files in tudat-multidoc/docstrings
, you are likely to encounter issues during execution of CLI document
.
Typically, the issues are one of the following two kinds:
In the filtering process, the parser separates the cpp and python specific information of the yaml
source.
It does so by parsing only # [cpp]
and # [py]
tagged lines, respectively, along with the agnostic (untagged) lines of the source.
This means that the yaml
source has to match the syntax expected by the parser, when only considering these subsets of lines.
It is difficult to keep track of the syntax of the filtered yaml
, therefore errors like the following will arise frequently:
5897-INFO-Parsing yaml file: ./tudat-multidoc/docstrings/numerical_simulation/environment_setup/ephemeris.yaml with kwargs: {'py': True}
5897-ERROR-Broken .yaml file ./tudat-multidoc/docstrings/numerical_simulation/environment_setup/ephemeris.yaml dumped as BROKEN-{'py': True}-ephemeris.yaml.
[...]
in "<unicode string>", line 46, column 9:
extended_summary: "Instances of ...
^
expected <block end>, but found '<scalar>'
in "<unicode string>", line 58, column 21:
short_summary: "Class for defining settings from ...
^
The first line
5897-INFO-Parsing yaml file: ./tudat-multidoc/docstrings/numerical_simulation/environment_setup/ephemeris.yaml with kwargs: {'py': True}
indicates the yaml
source file (ephemeris.yaml
) and filtering settings (kwargs: {'py': True}
) during which the error occured.
The second line is key to troubleshooting this issue:
5897-ERROR-Broken .yaml file ./tudat-multidoc/docstrings/numerical_simulation/environment_setup/ephemeris.yaml dumped as BROKEN-{'py': True}-ephemeris.yaml.
It states that an auxiliary file BROKEN-{'py': True}-ephemeris.yaml
has been dumped in ./
(i.e. on the same level as tudat-multidoc
).
This file shows what the parser "sees" under the current filtering settings (kwargs: {'py': True}
).
The syntax issue can be found in the line that is indicated by the traceback in
yaml.parser.ParserError: while parsing a block mapping
in "<unicode string>", line 46, column 9:
extended_summary: "Instances of ...
^
where the line count does not refer to the yaml
source, but only includes the lines that pass the filter, therefore matching the
line count of the auxiliary file.
Following this trace in the auxiliary file, the syntax issue should become apparent.
Note that in many cases, the second error is due to the same issue as the first error and will be disappear after resolution of the first one.
In most cases the underlying issue that a missing or erroneous # [py]
/ # [cpp]
tag.
Consequently, the affected line or block of text passes the filter, mistakenly appearing in the filtered yaml
.
The erroneous line or block then often does not relate well to the rest of the yaml
, raising a parser error of some kind.
IMPORTANT: Since the auxiliary file is used to inspect the error, one may accidentally perform the corrections on the auxiliary file, leaving the source unchanged and not resolving the issue. In order to avoid frustration, ensure to implement in the corresponding section(s) of the source file.
In tudat-multidoc/multidoc/multidoc/parsing/models.py
the pydantic python module is used in order to define data models
for formatting the information that is eventually assembled in the API reference. Every key in the yaml
source, such as classes
functions
, methods
, properties
, etc refers to such a data model and/or an attribute thereof.
In order to be formatted correctly, the information grouped under each key has to be compatible with the data model
the key refers to, else pydantic raises validation errors
.
An example of such an error is given below:
58600-INFO-Parsing yaml file: ./tudat-multidoc/docstrings/simulation/environment_setup/ephemeris.yaml with kwargs: {'py': True}
[...]
File "pydantic/main.py", line 406, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 2 validation errors for Module
classes -> 0 -> methods -> 1 -> returns -> description
field required (type=value_error.missing)
functions -> 2 -> returns -> type
field required (type=value_error.missing)
The first line
58600-INFO-Parsing yaml file: ./tudat-multidoc/docstrings/simulation/environment_setup/ephemeris.yaml with kwargs: {'py': True}
indicates the file in which the pydantic issue occurred, i.e. ephemeris.yaml
.
The next lines
File "pydantic/main.py", line 406, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 2 validation errors for Module
state the kind of error (validation error
) as well as the amount of errors encountered.
Next, the problematic instances of data classes are stated
classes -> 0 -> methods -> 1 -> returns -> description
field required (type=value_error.missing)
This reads as "in first instance (index 0) of a class
data model, second (index 1) instance of a method
data model,
the return
entry is missing a data field (named description
), which is required by the return
data model."
The count of instances at the highest level (i.e. in this case classes -> 0
) refers to the counts of class
data structures within the aforementioned file ephemeris.yaml
.
Analogously the other validation error
functions -> 2 -> returns -> type
field required (type=value_error.missing)
can be read as "in third instance (index 2) of a function
data model, the return
entry is missing a data field
(named type
), which is required by the return
data model."
NOTE: In a continuous effort to make the pydantic data models more lenient, the data fields
description
andtype
have been declared to be optional fields in thereturn
data model. Therefore, the specific errors above may not arise in this exact form anymore, but serve a demonstrative purpose.
Pydantic validation errors can easily be resolved by providing the missing data field to the data structure in the yaml
source.
In selected cases it may be a better solution to change the troubling pydantic data model to be more lenient, i.e. making the missing data field
an optional member of the data model.
Will my build be overwritten?
Don't worry, builds aren't discarded, only updated if changes are found in the source when executing b/a
.
Why are some elements not documented as expected in the output HTML?
This is expected given our current state of development. The possibilities (I'm aware of) are:
multidoc
used for overloads) are not dealt with yet in the generated docstrings.h
file yet (see here).multidoc
was designed with the intention of namespace-module equivalency in mind. This means that if the namespace in cpp does not match the module structure in the pybind exposure, multidoc
cannot automagically generate the API docs. This extends to the directory structure of the cpp
source. The following exemplar statements demonstrate this requirement:
tudatpy.interface.spice
should be exposed inside a namespace in the tudatpy
source as tudatpy::interface::spice
.
get_docstring(element,variant)
is redefined in each namespace separately, to ensure no ambiguity when calling get_docstring()
(e.g. module_1.create()
/module_2.create()
). This design was adhered to to ensure the undocumented version of tudatpy
can be compiled with a default get_docstring()
which returns an arbitrary No doc found
.tudat::interface::spice
should be a namespace that encompasses all source files in a directory found as tudat/interface/spice/*
.
docstrings/interface/spice.yaml
are collected as a dictionary to replace all //! get_docstring(element, variant)
tags in the tudat header files. These files are iterated through according to the defined API structure, to avoid clashes. The namespace address/location is then used to generate the sphinx .rst
seen here.Constants
are not yet implemented in the expansion of the module templates (e.g. see py-sphinx-module.jinja2) which use the term Attribute
here instead incorrectly.Why does doxygen take so long to build?
Unfortunately, I've yet to design a configuration of the target source directories in the Doxyfile.in
which list only those stated in the API docstrings.