The IncludeWorkflow operator is an easy way to create new Bonsai operators and reuse existing code. They are used extensively in some packages like Bonvision and Bonsai.ML. However these operators are essentially blind to docfx and are not automatically documented in the Reference section.
Background
The standard dotnet docfx command actually runs two commands in its pipeline:
dotnet docfx metadata
This command generates ManagedReference files in .yml format from .NET source code. In addition, it creates a .manifest file which is used in docfx build, as well as namespace.yml files for the namespace API list and toc.yml for the Table of Contents. All of these are placed in the dest folder defined in docfx.json (api/)
dotnet docfx build
This command generates .html files from the .yml files in api/ and places them in the output folder defined in docfx.json (_site).
Proposed Solution
One possible solution would be a patch to perform the role of the dotnet docfx metadata in creating .yml files but for .bonsai files. In addition, it would modify .manifest, toc.yml and namespace.yml to include these operators.
Such a patch would come in the docfx pipeline like this:
An advantage of this solution is that because it is injecting into the existing pipeline for C# operators, it can make use of any future improvements we make (such as the proposed ManagedReference based API template https://github.com/bonsai-rx/docfx-tools/pull/9).
Proposed Implementation
1) Parse .bonsai files (essentially XML files) and extract the operator description, ExternalizedMapping properties and their description (if available), and exclude PropertyMapped properties which are hidden.
2) If there are embedded C# operators that are part of the package repository, parse the .cs files in the src folder for more information.
3) If there are other embedded IncludeWorkflow operators, parse them.
4) If the properties come from operators that are not part of the package repository, but are instead part of the package dependencies, parse the XML doc file that is included with the dependency in the .bonsai local environment.
Potential problems
For embedded operators that come from a dependency, there is limited information in the XML doc file (and sometimes, no accompanying XML doc file). That might complicate the search for property types and inputs/outputs.
Alternative Solutions Considered
Dummy CS placeholders
One early hack that I explored, was to create dummy .cs files with just the bare minimum information (namespace, class declaration) to be recognized by the docfx metadata command, and then use the docfxoverwrite function during docfx build to include supplementary markdown articles that would appear on the API page. That works, but the overwrite articles still need to be individually created, and would be in a different format than the other operators.
Docfx build plugin
Another option I explored was Docfx's support for document processors and custom plugins, which run during the docfx build process. I made some progress on a plugin to generate .yml files from .bonsai files but I abandoned this effort for several reasons:
The plugin runs during the docfx build process, which I am not sure is the right way place to generate .yml ManagedReference files. I found myself having to run docfx build twice (once to generate .yml files from .bonsai files, and then .html files from those new .yml files).
I did not find any classes that would have helped to generate ManagedReference files and ended up having to implement the logic from scratch.
With no C# background it became exceedingly hard and complicated for me (even with ChatGPT :P)
Using existing IncludeWorkflow associated classes and methods in the Bonsai core library
I'm sure there are easier ways of doing the patch using existing classes and methods included in the Bonsai Core library, but I am not sure where to start and how to go about doing it :P
Motivation
The IncludeWorkflow operator is an easy way to create new Bonsai operators and reuse existing code. They are used extensively in some packages like Bonvision and Bonsai.ML. However these operators are essentially blind to
docfxand are not automatically documented in theReferencesection.Background
The standard
dotnet docfxcommand actually runs two commands in its pipeline:This command generates ManagedReference files in .yml format from .NET source code. In addition, it creates a
.manifestfile which is used indocfx build, as well asnamespace.ymlfiles for the namespace API list andtoc.ymlfor the Table of Contents. All of these are placed in thedestfolder defined indocfx.json(api/)This command generates
.htmlfiles from the .yml files inapi/and places them in theoutputfolder defined indocfx.json(_site).Proposed Solution
One possible solution would be a patch to perform the role of the
dotnet docfx metadatain creating .yml files but for .bonsai files. In addition, it would modify.manifest,toc.ymlandnamespace.ymlto include these operators.Such a patch would come in the
docfxpipeline like this:An advantage of this solution is that because it is injecting into the existing pipeline for C# operators, it can make use of any future improvements we make (such as the proposed
ManagedReferencebased API template https://github.com/bonsai-rx/docfx-tools/pull/9).Proposed Implementation
1) Parse .bonsai files (essentially XML files) and extract the operator
description,ExternalizedMappingproperties and their description (if available), and excludePropertyMappedproperties which are hidden.2) If there are embedded C# operators that are part of the package repository, parse the
.csfiles in the src folder for more information.3) If there are other embedded
IncludeWorkflowoperators, parse them.4) If the properties come from operators that are not part of the package repository, but are instead part of the package dependencies, parse the XML doc file that is included with the dependency in the .bonsai local environment.
Potential problems
For embedded operators that come from a dependency, there is limited information in the XML doc file (and sometimes, no accompanying XML doc file). That might complicate the search for property types and inputs/outputs.
Alternative Solutions Considered
Dummy CS placeholders One early hack that I explored, was to create dummy
.csfiles with just the bare minimum information (namespace, class declaration) to be recognized by thedocfx metadatacommand, and then use thedocfxoverwrite function duringdocfx buildto include supplementary markdown articles that would appear on the API page. That works, but the overwrite articles still need to be individually created, and would be in a different format than the other operators.Docfx build plugin Another option I explored was Docfx's support for document processors and custom plugins, which run during the
docfx buildprocess. I made some progress on a plugin to generate .yml files from .bonsai files but I abandoned this effort for several reasons:docfx buildprocess, which I am not sure is the right way place to generate .ymlManagedReferencefiles. I found myself having to rundocfx buildtwice (once to generate .yml files from .bonsai files, and then .html files from those new .yml files).ManagedReferencefiles and ended up having to implement the logic from scratch.Using existing IncludeWorkflow associated classes and methods in the Bonsai core library I'm sure there are easier ways of doing the patch using existing classes and methods included in the Bonsai Core library, but I am not sure where to start and how to go about doing it :P