RFC - Documentation

[jrepp]

Hi All!

I'd like to float the idea of improving the documentation of the Daisy library.

There are several different approaches to doing documentation, there exist some nice fully featured code documentation tools out there. For a simple library however it might make sense to pursue a very basic approach.

One example is something I did for a very basic library I released on GitHub: https://github.com/jrepp/ipv6-parse/blob/master/makedoc.py

This allows me to just write Markdown in my header and export it to the header. It's DRY and allows me to just go MD straight onto GitHub and I think the results look pretty nice.

Here's a matrix of options: https://en.wikipedia.org/wiki/Comparison_of_documentation_generators

What's funny/sad about this is that if you look at all those options they all have one of the following limitations:

• No native Markdown/RST support
• Requires learning a new specialized language/markup
• Requires some other large dependency (install all the things)

I found a simple script tailored to the project where you just type python makedox.py > MYDOCS.md to be more than sufficient. It looks great on GitHub and is simple to augment with handwritten RST or MD of your own.

I've also used https://readthedocs.org/ in the past but only for python, it's probably possible to use it for C as well. This would provide CI and automated hosting for documentation.

Thoughts?

[PaulBatchelor]

@jrepp do you have an example of your documentation generator in action?

Inline code documentation generators like doxygen have never really appealed to me. They never looked good. They always seem to clutter up the code. I've never found the generated documentation useful either. It could be something one gets used to. Maybe it makes sense for more large-scale APIs.

It'd be great to specify what kind of documentation we are talking about. Some would say there are four types of documentation: tutorials, how-to guides, explanation, and technical reference.

For technical reference, there have been previous discussions for ways to document daisysp specifically in #4. The idea was to do something similar to Soundpipe documentation (my library, so I am very very biased here). It's loosely based on the way the Csound manual is structure, where documentation is written as metadata.

For libdaisy documentation, I'd probably opt for external documentation for now, in the form of markdown documents. Tutorials, how-to guides, and explanations would be good to think about. It might be too early to think about technical reference for libdaisy, since the API is still being developed right now.

My favorite C API documentation of all time is SQLite. The structure an organization allow a newcomer to quickly get started. If you follow enough links on their tutorial, you'll eventually hit technical reference bedrock. BUT the bedrock is often written as more of an explanation or article, full of useful technical tidbits and justifications for why things are the way they are. And there are always, always links to more articles to read. It's really fun, actually! It may not be practical/realistic to do what SQLite does. Their wiki documentation system is directly integrated into their own source control system Fossil, which also has their ticketing system. And the core structure of their Fossil repo is, you guessed it, SQLite. Talk about yak shaving, right?

I can't not go into a discussion about documentation without mentioning literate programming, which is something I've tried out on a few more recent projects of mine. I've used Knuth's CWEB and org-mode with tangling done via an org tangler of mine called Worgle. I don't necessarily recommend literate programming for Daisy documentation, but it could be useful for certain parts.

[stephenhensley]

Personally, I'm a sucker for markdown. But I'm still working out how we should document each library (I believe that both of them have a different set of requirements.

We should keep this discussion open for sure, especially for the libdaisy side of things where technical reference will be a bit more important due to the different levels of hardware interaction etc.

For the DSP library, I'm heavily leaning toward a metadata driven approach similar to soundpipe. As @PaulBatchelor mentioned this is based around csound's manual, which I think we both have had very positive experiences with. This would help integrate each module's purpose, description, usage, and an example into a single page of documentation.

For libdaisy, I could be convinced to use some sort of inline documentation/parser (i like the little python script @jrepp shared, I'd love to see the output it produces). However, I do agree that I have seen many reference documents generated in this fashion and have found very few of them as helpful as they should have been.

Regardless, we should start getting some documentation underway, especially as more people begin working on the project. But I think there is plenty left to discuss in terms of the details.

[jrepp]

@jrepp do you have an example of your documentation generator in action?

Here's a link to the project, the README.md comes right out of the header ipv6.h: https://github.com/jrepp/ipv6-parse/

The idea is that the code is self-documenting, and you don't have to repeat stuff in separate documents or use some new meta-data language. You write markdown in your comments, and it exports and annotates them.

It'd be great to specify what kind of documentation we are talking about.

I'd love to make some browsable documentation from the code. We could auto-generate the wiki pages with functions, structures, and descriptions. I agree that spending time on documentation in general (across all categories) will undoubtedly be worthwhile. I noticed that making it easy to export comments from code into markdown makes me more inclined to write higher quality comments. For one, I know that I only need to do it once, and it will look great on the GitHub web interface. Secondarily, I guess I'm pretty comfortable with markdown. I think it matches how I would use ASCII to mark up some comments to make them more readable.

My favorite C API documentation of all time is SQLite.

SQLite is an incredible piece of software. I respect that they are dogfooding their stuff and they have some great best-practices. There is a lot of engineering effort they have put into correctness and documentation.

The work they do comes from their goals of creating an embedded, high-quality general-purpose database library. So they go for full code-coverage in their tests and an impressive amount of rigor around documentation and correctness.

I'll check out the Csound documentation more to see how they achieve their results. It sounds like it's a goal of making a general-purpose DSP library that users can easily add new types and functions (similar to Csound?). The user-generated types could then be auto-documented and included in a general-purpose documentation repository.

Plenty more to discuss here - if you guys are open to it, I can explore a small proof-of-concept. I think little tasks like this could be a way for me to make the most of the time I have to contribute.

[stephenhensley]

Wow, the ipv6-parse example is really nice. It does have the negative impact of increasing the size of the header files by quite a bit, but the output is nice and clean. Plus it gives the developer one place to make changes.

This combined with something like pandoc could make some beautiful PDFs.

I think this would be a nice tool for documenting the libdaisy library.

When I'm using software that has generated documentation like this, I mostly reference the header files themselves, and it keeps me from having to have a separate window open with the docs. So that could be seen as an added benefit. (Certain things, including STs HAL library actually are a bit more enlightening in the context of the header rather than in a separate document -- as long as you know where to look).

On the contrary, here is an example of at least semi-generated documentation: here is the mess that is the ST H7 HAL (which is still being used at the lower levels of libdaisy)

Even with the "How to use this driver" sections, this is difficult documentation to use, and sift through. I often have to also have the Reference Manual open, along with the header/source files that I'm using to work out what I need to do.

---

SQLite is indeed a marvel, and the documentation is very nice as well.

That type of documentation would be very well suited for something like libdaisy where there are multiple levels of use from higher level stuff like dsy_audio, to the device drivers for specific audio codecs, flash chips, etc.

---

DaisySP is definitely more of general purpose DSP library meant to have several modules that all have the same interface (init, process, etc.), and is meant to be both easy to use, and efficient.

This is where something like the csound manual and even @PaulBatchelor's soundpipe documentation really shine.

If we do go the lua/metadata route for this, I'd probably write a script to automate the initial creation of each module's metadata, to reduce the amount of time-overhead needed to write separate files with documentation. Seems like the format is pretty standard, and as long as files/functions are named similarly a lot of the boilerplate could be easily generated.

Right now I only have the DSP library building for this specific ARM platform, but I have some prototyping software for testing/developing DSP on a PC, and it would be nice to eventually be able to use/develop DaisySP on different platforms including Cortex-A ARM, and standard PC OSes.

That said, many of the paradigms used in DaisySP are less beneficial outside of a bare-metal hardware environment (static memory allocation, pure C, etc.). There are a lot of C++ DSP libraries out there, but not many (at least that I've seen) of the flavor being created here.

[andrewikenberry]

I like the idea of using markdown on libdaisy, and lua/similar for DaisySP.

[andrewikenberry]

For now we're using makedoc.py > markdown > pdf for both libs technical reference documentation.

We'll open a more specific issue for any changes down the line (but for now we just want to have some documentation, rather than none.)