WIP: Docs updates

[saulshanabrook]

• [x] Add instructions for buildings docs
• [x] Verify that existing snippets are still accurate
• [x] Try out c domain to see if it gets us anything nicer than what we have now (maybe better linking between functions)
• [x] run doctests in travis

I looked a bit into trying an autodoc solution for the C code. Looks like breathe or hawkmoth are both still being actively developed. The advantage of autodoc is that it would keep the docs up to date with the source code and we could re-use the source code comments to create the documentation. However, I worried that the tools to do this for C code would introduce too many extra build steps (doxygen, etc) and be less flexible than the current approach. For example, I don't think we could keep the current approach of documenting different parts of the _ndt struct, if we used an autodoc approach. There is a also pre-release project Sphinx AutoAPI that currently supports .NET, go, python, and javascript, which might be another useful tool if we wanted to try auto-generating the C docs (reference issue to document Python C extensions with it https://github.com/rtfd/sphinx-autoapi/issues/100).

It would also be nice to test the documentation examples to make sure they are still valid, but I couldn't find a straightforward way to do this for C code.

[skrah]

Just a quick note about generated docs that could save you some work: Generally I'm not a fan of auto-generated docs.

If I see them anywhere, usually I go straight to the source code. :)

[saulshanabrook]

@skrah sounds good, I won't pursue that anymore then. Do you have strong feelings about using the c domain versus code snippets?

[skrah]

@saulshanabrook Do you mean removing .. code-block:: c from the code snippets and use the C domain globally?

If you manage a setup such that doc/ndtypes/* uses the Python domain globally and doc/libndtypes/* uses the C domain globally, that would be great -- provided that the setup is not too complex.

I didn't find such a Sphinx option.

[saulshanabrook]

@skrah that would be helpful too, but no I mean using things like the c:function directive instead of code blocks. I have tried this out on the init page locally, here is a screenshot of the difference:

Once nice thing is that it will now automatically link the type arguments to their documentation. For example clicking on the ndt_constraint_t f argument of ndt_typedef will take you to the documentation for that type.

[saulshanabrook]

I added support for testing the Python code snippets. All work as documented (besides a few formatting differences that I fixed), besides one:

$ make doctest
sphinx-build -b doctest -d build/doctrees . build/html
Running Sphinx v1.7.2
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
building [doctest]: targets for 19 source files that are out of date
updating environment: 0 added, 0 changed, 0 removed
looking for now-outdated files... none found
running tests...

Document: ndtypes/quickstart
----------------------------
1 items passed all tests:
  10 tests in default
10 tests in 1 items.
10 passed and 0 failed.
Test passed.

Document: ndtypes/pattern-matching
----------------------------------
**********************************************************************
File "ndtypes/pattern-matching.rst", line 159, in default
Failed example:
    p.match(c)
Expected:
    True
Got:
    False
**********************************************************************
1 items had failures:
   1 of  91 in default
91 tests in 1 items.
90 passed and 1 failed.
***Test Failed*** 1 failures.

Document: ndtypes/types
-----------------------
1 items passed all tests:
  46 tests in default
46 tests in 1 items.
46 passed and 0 failed.
Test passed.

Document: ndtypes/buffer-protocol
---------------------------------
1 items passed all tests:
   2 tests in default
2 tests in 1 items.
2 passed and 0 failed.
Test passed.

Doctest summary
===============
  149 tests
    1 failure in tests
    0 failures in setup code
    0 failures in cleanup code
build finished with problems.
make: *** [doctest] Error 1

The example that failed is:

   >>> p = ndt("(Any) -> Scalar")
   >>> c = ndt("(?{a: 10 * uint8}) -> uint8")
   >>> p.match(c)
   True

[skrah]

Is the c:function change big? If not, let's just do it -- I'll see if the colors are ok for me once we push it to readthedocs.

[skrah]

Let's just delete the failing example. -- It's an open question whether Any should match optional types or only ?Any.

[saulshanabrook]

Is the c:function change big? If not, let's just do it -- I'll see if the colors are ok for me once we push it to readthedocs.

It is a lot of line changes, but I went ahead and did it yesterday. I also set the theme in the config to be the rtd theme, so it looks the same locally as it does on there.

I am just trying to get the doc tests to run properly on travis now.

[saulshanabrook]

In getting the doctests to run in travis, I realized that I was running them locally against my installed version of ndtypes instead of the one in the repo. I changed that and now I am getting a bunch of errors. I will go through these and update them to work with the existing code.

$ python setup.py doctest
Running Sphinx v1.7.2
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
building [doctest]: targets for 19 source files that are out of date
updating environment: 0 added, 0 changed, 0 removed
looking for now-outdated files... none found
running tests...

Document: ndtypes/pattern-matching
----------------------------------
**********************************************************************
File "ndtypes/pattern-matching.rst", line 78, in default
Failed example:
    p.match(c)
Expected:
    True
Got:
    False
**********************************************************************
File "ndtypes/pattern-matching.rst", line 103, in default
Failed example:
    p.match(c)
Expected:
    True
Got:
    False
**********************************************************************
File "ndtypes/pattern-matching.rst", line 154, in default
Failed example:
    p.match(c)
Expected:
    True
Got:
    False
**********************************************************************
File "ndtypes/pattern-matching.rst", line 314, in default
Failed example:
    p.match(c)
Expected:
    True
Got:
    False
**********************************************************************
File "ndtypes/pattern-matching.rst", line 341, in default
Failed example:
    p.match(c)
Expected:
    True
Got:
    False
**********************************************************************
File "ndtypes/pattern-matching.rst", line 346, in default
Failed example:
    p.match(c)
Expected:
    True
Got:
    False
**********************************************************************
1 items had failures:
   6 of  88 in default
88 tests in 1 items.
82 passed and 6 failed.
***Test Failed*** 6 failures.

Document: ndtypes/types
-----------------------
**********************************************************************
File "ndtypes/types.rst", line 407, in default
Failed example:
    ndt("(distance: float32, velocity: float32) -> float32")
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/doctest.py", line 1330, in __run
        compileflags, 1), test.globs)
      File "<doctest default[0]>", line 1, in <module>
        ndt("(distance: float32, velocity: float32) -> float32")
    ValueError: missing typedef for key 'distance'
**********************************************************************
File "ndtypes/types.rst", line 419, in default
Failed example:
    ndt("(sum: float64, ...) -> float64")
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/doctest.py", line 1330, in __run
        compileflags, 1), test.globs)
      File "<doctest default[0]>", line 1, in <module>
        ndt("(sum: float64, ...) -> float64")
    ValueError: missing typedef for key 'sum'
**********************************************************************
File "ndtypes/types.rst", line 431, in default
Failed example:
    ndt("(uint32, uint32, product: float64) -> float64")
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/doctest.py", line 1330, in __run
        compileflags, 1), test.globs)
      File "<doctest default[0]>", line 1, in <module>
        ndt("(uint32, uint32, product: float64) -> float64")
    ValueError: missing typedef for key 'product'
**********************************************************************
File "ndtypes/types.rst", line 445, in default
Failed example:
    ndt("(uint64, ..., scale: uint8) -> uint64")
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/doctest.py", line 1330, in __run
        compileflags, 1), test.globs)
      File "<doctest default[0]>", line 1, in <module>
        ndt("(uint64, ..., scale: uint8) -> uint64")
    ValueError: 1:13: syntax error, unexpected COMMA, expecting RPAREN

**********************************************************************
File "ndtypes/types.rst", line 452, in default
Failed example:
    ndt("(uint64, scale: uint8, ...) -> uint64")
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/doctest.py", line 1330, in __run
        compileflags, 1), test.globs)
      File "<doctest default[0]>", line 1, in <module>
        ndt("(uint64, scale: uint8, ...) -> uint64")
    ValueError: missing typedef for key 'scale'
**********************************************************************
File "ndtypes/types.rst", line 459, in default
Failed example:
    ndt("(..., color: uint32, ...) -> uint64")
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/doctest.py", line 1330, in __run
        compileflags, 1), test.globs)
      File "<doctest default[0]>", line 1, in <module>
        ndt("(..., color: uint32, ...) -> uint64")
    ValueError: 1:5: syntax error, unexpected COMMA, expecting RPAREN

**********************************************************************
1 items had failures:
   6 of  46 in default
46 tests in 1 items.
40 passed and 6 failed.
***Test Failed*** 6 failures.

Document: ndtypes/quickstart
----------------------------
1 items passed all tests:
  10 tests in default
10 tests in 1 items.
10 passed and 0 failed.
Test passed.

Document: ndtypes/buffer-protocol
---------------------------------
1 items passed all tests:
   2 tests in default
2 tests in 1 items.
2 passed and 0 failed.
Test passed.

Doctest summary
===============
  146 tests
   12 failures in tests
    0 failures in setup code
    0 failures in cleanup code
build finished with problems.

[skrah]

This is what I expected: ndt("(..., color: uint32, ...) -> uint64") is the old function syntax that permitted keyword arguments and variadic arguments.

All such examples can be removed. I guess the PR gets quite big now -- Would it be possible to open separate PRs for the doctest and c::function features?

[saulshanabrook]

Sure, I can split this PR into two, one for ./python doc changes and the other for ./libndtypes changes.

[saulshanabrook]

Broke this into #42 and #43