Idea: automatically test examples

numero-744 commented 2 years ago

Pre-scriptum: sorry for the long text

It is possible to add Rust code into documentation (equivalent of scaladoc). Not only it supports syntax highlighting, it also enable testing the examples, to make sure it is up to date with the documented API.

An idiomatic way to write an example in Rust is to call a function, then assert its result, so in the example there is the assertion into the which the user/reader sees the returned value. For things not returning a value (we have plenty of them to describe hardware), no assertions is done (there are already unit tests elsewhere in Rust, just like with Scalatest), and it still checks that it builds and does not panic (~ throw an exception).

For SpinalHDL, it could be possible to check if examples build and if it returns an error or not (for instance design errors). For full examples with hardware + test, check if the test is successful.

I think the main question is: how have a source translated to RTD and scala test files?

First, to check with current API I think it is possible to clone the SpinalHDL template for tests and add / replace files (GitHub actions or other mechanism would enable caching dependencies).

Then, how to have testable code? It does not seem possible to automatically translate all current examples and voilà the tests run and pass. I see several ways, all of them are opt-in and would not break current documentation (so the documentation would need refactoring of each test to make it testable). These are listed below (to be updated during discussion):

Tests in scala files

Sphinx can include files as code blocks, and these files can be used in a scala environment but it does not seem easy to deal with:

Sphinx can take them as-is
How to integrate them into tests?
You may not want to show all the lines to not have heavy full examples everywhere in the docs. When showing code which goes into a component, adding a "component" start and end all the time may be noisy.
Put all examples in the same directory and have paths like ../../examples/section/subsection/example.rst? Or put examples with the code and make them harder to find to execute tests?

Partial tests in scala files

Having Scala files with only shown part of a test and some metadata to help including it in a test. Still needs to be injected in some code.

Parsing `rst` files

Please don't do that.

Writing a Sphinx extension

I think it is the most flexible solution so I'm gonna write a little bit about what I think we could get.

This extension, written in Python, would bring tags like .. spinalComponent, followed some metadata, then by code (like current situation). It would generate an example like:

test("some test name from some metadata?") {
  class ComponentNameFromMetadata extends Component {
    the code of the example
  }
  SpinalConfig.generateVerilog(new ComponentNameFromMetadata)
  // I would like a "check" function to not generate a file ^^'
}

There would be several tags, for several contexts (Component, full test, maybe others?).

It would also make it possible to have more features:

Hiding some lines (for instance declaring signals, but the example is often more clear if they appear on it) by prefixing them with a # (like Rustdoc)
Checking that some code does not build (scalatest has a function for that) or has design errors (check that generation throws an error), and in this case print the example a different way.
A "copy" button to copy example code wrapped into an App to play around with it, or a button to show this App.

wifasoi commented 2 years ago

maybe we can use https://linuxtv.org/downloads/sphinx-1.2.3/markup/code.html it gives you the ability to hide code lines. Id so, then we can test the code snipset with the normal scala toolchain. For make the code interactive.. we can use jupiter notebooks (need to check if there is some good plugin for sphinx) or external service we can leverage for the time being (maybe there is a code explorer for scala somewhere)

wifasoi commented 2 years ago

I think is best if i add some code guard.. like//[START EXAMPLE] and //[STOP EXAMPLE] so sphinx can filter out the harness necessary to run the example:

.. literalinclude:: example.scala
   :language: scala
   :start-after:  [START EXAMPLE]
   :end-after: [STOP EXAMPLE]

This is alredy implemente in the extraction script in my branch (with all the extracted script) here: https://github.com/wifasoi/SpinalDoc-RTD/tree/separated_examples

numero-744 commented 2 years ago

To test examples from Scaladoc I have found https://github.com/ThoughtWorksInc/sbt-example but for now I fail to set it up.

I will™ open an issue on their repo with the error I get.

SpinalHDL / SpinalDoc-RTD