Closed TeroFrondelius closed 9 years ago
Documentation is essential. There needs to be something for people to read, even if it's rudimentary and incomplete. This falls squarely into the "drudgery" category referred to earlier, and is often the first area where a new open source project falls down. Coming up with a mission statement and feature list, choosing a license, summarizing development status—these are all relatively small tasks, which can be definitively completed and usually need not be revisited once done. Documentation, on the other hand, is never really finished, which may be one reason people sometimes delay starting it at all.
Here is the Julia style: https://github.com/JuliaLang/julia/tree/master/doc
Julia Documentation README
Julia's documentation is written in reStructuredText, a good reference for which is the Documenting Python chapter of the Python Developer's Guide.
Prerequisites for building the documentation
The documentation is built using Sphinx and LaTeX. On ubuntu, you'll need the following packages installed:
latex-cjk-all texlive texlive-lang-cjk texlive-latex-extra
On OS X, you can install install MacTex using the GUI installer
Building the documentation
Build the documentation by running
$ make helpdb.jl $ make html $ make latexpdf
File layout
conf.py Sphinx configuration helpdb.jl REPL help database stdlib/ Julia standard library documentation UNDOCUMENTED.rst Undocumented functions (to be filled in and copied to the correct location in stdlib/)
Sphinx extensions and theme
The extensions to Sphinx and the theme are in the https://github.com/JuliaLang/JuliaDoc repository, and can also be used to style package documentation.
@ovainola did you test the @doc? For me it's looks like that in the future the documentation would be clear:http://julia.readthedocs.org/en/latest/manual/documentation/#other-notes
@TeroFrondelius I tested the @doc macro. There's a bit more boiler to write if you compare with the notation in the link. Though it doesn't hurt if we do documentation with @doc atm, since we only have to delete couple of words to match it with the new notation.
Here is our documentation hosting / building system: https://readthedocs.org/ /CC @ovainola @ahojukka5
And here is the official package for https://readthedocs.org//: https://github.com/JuliaLang/JuliaDoc
Latest documentation page generated in readTheDocs: http://juliafem.readthedocs.org/en/latest/
Ok we're almost done with this issue. We also need something to convert notebooks to rst files in order to progress. After little googling I found a workaround that might work
https://github.com/rtfd/readthedocs.org/issues/579
[quote] arsenovic commented on Dec 20, 2013 Here is the Makefile directive i use to run and convert ipython notebooks to reST (followed by some minor sedding).
notebooks:
for file in ./source/tutorials/.ipynb; do\
runipy -q --pylab -o $$file;\
done
cd ./source/tutorials;ipython nbconvert --to rst *.ipynb
sed -i 's/`/
/g' ./source/tutorials/.rst
I suspect that a lot of projects will start using ipython notebook-based docs, because of their ease.
a conversation about this can be found here http://python.6.x6.nabble.com/IPython-User-usage-of-notebooks-in-sphinx-td5041731.html [\quote]
@ahojukka5 check this: http://juliafem.readthedocs.org/en/latest/getfem_examples/README/
I would claim it's almost working system already. (Obviously some mistake in configuration, because it's called REAME. )
Well it's making reference back to GitHub, not rendering the notebooks to html. At least for functional testing the notebooks should be run and checked that they compile.
I was thinking something like this, where notebooks are properly rendered to html:
http://www.damian.oquanta.info/posts/zen-themes-updated.html
Almost forgot, is it possible to get readthedocs url -> juliafem.org/docs?
@ahojukka5 see #7
Here is good news for our own domain http://read-the docs.readthedocs.org/en/latest/alternate_domains.html
@ovainola why we don't see our documentation from https://github.com/JuliaFEM/JuliaFEM/blob/master/src/JuliaFEM.jl in our newly generated http://juliafem.readthedocs.org/en/latest/
@TeroFrondelius since readTheDocs atm generates documentation from the /docs forder. This feature still needs some configuration.
This is an alternative https://github.com/MichaelHatherly/Lexicon.jl. It contains doc test as well as some sort of scoring system.
Looks cool. Magic in the documentation seems to be in the https://github.com/MichaelHatherly/Lexicon.jl/blob/master/docs/build.jl file, which runs all the tests and generates .md files into docs/api/ folder.
I made some googling and found packages https://github.com/MichaelHatherly/Lexicon.jl & https://github.com/MichaelHatherly/Docile.jl and they seems to be closely linked: Docile provides the @doc macro and Lexicon build and generates the standalone documentation. These packages have more commits & issues (about 10 open vs 60 closed) compared to JuliaDoc ( 2 open and 6 closed) so I'd say we should give them a try.
Ok, I tried Lexicon.jl and it's quite easy to use. But there's a catch: It's quite heavily linked to Doctile and in the source file it uses @document macro. This forces all the strings above functions to act as docstrings so it's basically automatic @doc for all the string. Here's an example how docstrings have to be written with Lexicon: https://github.com/QuantEcon/QuantEcon.jl/blob/master/src/lqcontrol.jl.
It also comes with doctests which is nice.
I forked the repository where I've been testing and here's a output of Lexicon: http://juliafemjl.readthedocs.org/en/latest/
note: docstring are written with numpy style and lexicon create markdown which has different syntax.
@ovainola will the question mark "?" work in REPL with Lexicon.jl?
Update: works
julia> using JuliaFEM
help?> JuliaFEM.elasticity_solver.calc_local_matrices!
calc_local_matrices! (generic function with 1 method)
julia> using Lexicon
help?> JuliaFEM.elasticity_solver.calc_local_matrices!
calc_local_matrices! (generic function with 1 method)
[method]
JuliaFEM.elasticity_solver.calc_local_matrices!(X, u, R, Kt, N, dNdξ, λ_, μ_, ipoints, iweights)
Calculate local tangent stiffness matrix and residual force vector R = T - F
Details:
source: (17,"/home/olli/.julia/v0.4/JuliaFEM/src/elasticity_solver.jl")
and if you have already @doc in front of docstring, Lexicon won't create any documentation..
I'd say we give it a try with Lexicon & markdown syntax. If all in favor, I'll make a change in contributing.md
Hi all, hope you don't mind my input here. When new packages are registered in METADATA
I generally try to have a look to see what their plans are for docstrings.
It contains doc test as well as some sort of scoring system.
The doctest
function is pretty basic currently, and just runs each code block in each docstring and records whether they "passed", "failed", or were "skipped". "Skipping" happens when the language isn't set to julia
for a code block.
they seems to be closely linked
Yes, Docile provides the backend collection of docstrings while Lexicon handles the presentation.
It's quite heavily linked to Doctile and in the source file it uses
@document
macro.
Recently the need to use @document
was removed. Docile now documents any imported package that contains bare docstrings without needing to be imported into that package.
will the question mark "?" work in REPL with Lexicon.jl?
If you'd like that to work for your users immediately then I'd recommend just adding import Lexicon
to your toplevel module. It'll add a slight increase to your package load time, but given you're writing a FEM package it probably won't account for the majority of the total time.
and if you have already
@doc
in front of docstring, Lexicon won't create any documentation..
Docile will document a package that contains only @doc
or only bare docstrings, not both. Given that https://github.com/JuliaLang/julia/pull/11836 was merged a few days ago I'd recommend just using bare docstrings, which should work in both 0.4 and 0.3 (with Docile's help).
If you run into any problems feel free to ping me with @MichaelHatherly.
@MichaelHatherly thanks for your comments. Do you know any good example repository, where Lexicon is used together with readthedocs? @ovainola has been studying more this topic, but I will try to briefly describe the wanted process:
Basically, Lexicon just extracts the docstrings from your package and dumps them into markdown files.
What you do from there is completely up to you. I'm not sure whether you'll be able to get the readthedocs server to do (3) or (4) since that would require having julia
present on their end. For (5) - (7): possibly, I've never tried doing that.
What I've been doing is just using mkdocs, which readthedocs supports, to do the presentation of the generated markdown files. I know Sims.jl has a build.jl
script that does some more complex things. There's also some other packages listed on in Docile Readme some of which may help.
In your REQUIRE
file I see you're only supporting Julia 0.4 and up, in which case there's quite a bit of work we're doing on updating the Julia manual and standard library docs right now, which should also be suitable for generating package documentation. So you might what to keep an eye on developments there.
I haven't followed this conversation very closely but here's my contribution for subject.
In another project I did automatic API documentation generation using Sphinx and Buildbot. In principle it was doing make sphinx-apidoc && make html in makefile which was automatically executed when buildbot polled source and noticed changes. It was one side communication, i.e. buildbot didn't have to pull anything to repository but api documentation was generated on the fly before building documentation. I think we are planning to do something similar with this project. It seems that using readthedocs differs a bit form this approach?
The readthedocs docs explain it a lot better than I'd be able to:
https://read-the-docs.readthedocs.org/en/latest/builds.html
and
https://read-the-docs.readthedocs.org/en/latest/webhooks.html
are probably the most relevant links for what you're interested in doing.
Ok I almost got working automatic documentation generation (including notebooks!) at juliafem.kapsi.fi. With @ovainola we found several issues related to simple decision whether to use markdown syntax or restructured text format. rst + ipython nbconvert + sphinx is one possibility. nbconvert markdown support is somehow broken, notebooks are not rendered properly. mkdocs uses markdown syntax. Github supports both markdown and rst. So if we get automatic api generation working with rst then I don't see any problems. Using markdown the biggest problems is how to render notebooks.
@MichaelHatherly Does Lexicon care what kind of syntax you're using inside docstrings? Atm we're using Markdown syntax but If we'll change to reStructuredText does it have some kind of effect? Of course we'll have to change the file suffix to .rst...
Lexicon uses the Markdown.jl package for docstring parsing, so you'll need to use markdown syntax. There is support for changing the format of docstrings but until there are packages that can parse other formats you'll need to stick with markdown. As far as I know there's no rst parser in the METADATA yet.
Ok that seems to be issue. Sphinx is not able to parse markdown.
1) convert markdown to rst before build using pandoc or similar? 2) find/develop rst parser for julia?
If we end up to write some very primitive parser by ourself (it's doesn't need to be very advanced if we don't use anything special in our docstrings), how do we use it with Lexicon?
Sphinx is not able to parse markdown.
I believe there is some support for it quite recently: https://github.com/sphinx-doc/sphinx/pull/1747
find/develop rst parser for julia?
That would be really cool to have. The most important part of Markdown.jl is that it renders markdown docstrings rather nicely in the REPL.
how do we use it with Lexicon?
You'd need to add a .docile
config file to you're src/
directory with something like this in it:
import Docile.Interface: Docs, data, parsedocs
import ReST # ?
parsedocs(docs::Docs{:rst}) = ReST.parse(data(docs))
Dict(
:format => :rst,
)
where ReST.parse
converts a raw docstring to some kind of intermediate representation that has the necessary writemime
methods defined for it. Placing the Dict
last in the file is important.
@MichaelHatherly Could you point me the relevant part of code where this is done using Markdown? As far as I understood this ReST.parse
takes docstring as input argument and returns some "object" which have method writemime
which again returns something when called. I could give a try for this approach because if we use numpy docstyle I think there's not so much to parse, it's already almost rst. But for that in need more information about this writemime
.
Lexicon hooks into Docile's parsing here. That's the equivalent to the line
parsedocs(docs::Docs{:rst}) = ReST.parse(data(docs))
that you'd put into a .docile
file.
As far as I understood this
ReST.parse
takes docstring as input argument and returns some "object"
Markdown.parse
is defined here and takes an AbstractString
and converts to a Markdown.MD
object defined here. There's several layers of calls that you'll need to walk through to get to the actual parsing mechanisms.
which have method
writemime
which again returns something when called
All the rendering methods are here. writemime
methods are defined to write to an IO
object which could be STDOUT
, a file, or perhaps an IOBuffer
. They don't actually return anything significant.
(Note that I've linked to the copy of Markdown.jl found in Lexicon. The package has been moved into Julia Base
as of 0.4
.)
If we want to use numpy docstyle in our code it sounds a good idea to use numpydoc package with sphinx to get things working. Maybe @pv could give us some ideas how to combine julia docstrings with numpydoc..?
You can probably reuse at least parser and formatter. You may need to modify a few parts IIRC there are autodoc style assumptions on working on Python importable objects. sphinx.ext.napoleon also has a separate implementation of Numpy style parser, don't know if it's tied to Python autodoc or not.
I tried the following with numpydoc and sphinx.ext.napoleon (the latter also supports numpy documentation style)
Foobar
------
.. py:function:: src.codes.make_sum(a, b)
Sum two numbers.
Parameters
----------
a : float
First number.
b : float
Second number.
Returns
-------
float
sum of numbers `a` and `b`.
Using numpydoc this rendered like it should be while using sphinx.ext.napoleon it gives SEVERE: Unexpected section title.
errors (napoleon functionality is linked to autodoc what we cannot use with julia).
So what we need to do to get this working is that by using Lexicon save/update we create function header info .. py:function:: src.codes.make_sum(a, b)
(actually jl:function here..?) and write the rest of the docstring as it is to file. Doesn't sound too cumbersome.
Then we add numpydoc to conf.py and if everything goes like in movies we should get our api documentation automatically generated.
Almost working
Docile: caching 5 modules from 'JuliaFEM'.
INFO: writing documentation to api/JuliaFEM.rst
INFO: writing documentation to api/JuliaFEM.abaqus_reader.rst
INFO: writing documentation to api/JuliaFEM.elasticity_solver.rst
INFO: writing documentation to api/JuliaFEM.interfaces.rst
INFO: writing documentation to api/JuliaFEM.xdmf.rst
ERROR: LoadError: MethodError: `save` has no method matching save(::ASCIIString, ::Base.Multimedia.MIME{sy mbol("text/rst")}, ::Lexicon.Index, ::Lexicon.Config)
Closest candidates are:
save(::AbstractString, ::Base.Multimedia.MIME{symbol("text/rst")}, ::Lexicon.Index, !Matched::Lexicon.rs tConfig)
save(::AbstractString, !Matched::Base.Multimedia.MIME{symbol("text/html")}, ::Lexicon.Index, ::Lexicon.C onfig)
save(::AbstractString, !Matched::Base.Multimedia.MIME{symbol("text/md")}, ::Lexicon.Index, ::Lexicon.Con fig)
...
in save at /home/users/thdr/.julia/v0.4/Lexicon/src/render.jl:237
in anonymous at /home/users/thdr/sites/juliafem.org/buildbot-slave/runtests/build/docs/build_api.jl:49
in cd at ./file.jl:22
in include at ./boot.jl:254
in include_from_node1 at loading.jl:133
in process_options at ./client.jl:306
in _start at ./client.jl:406
while loading /home/users/thdr/sites/juliafem.org/buildbot-slave/runtests/build/docs/build_api.jl, in expr ession starting on line 43
make: *** [apidoc] Error 1
should work now. Problem was the save function with index.rst, which I had commented out in my local repo. I made a commit where I commented the line. Problem isn't fixed yet, but build should pass now
Possible interestin package https://github.com/jakebolewski/JuliaParser.jl
Finally, automatic documentation works! Docstrings are read using Lexicon & generated into html with sphinx. Url to documentation: http://www.juliafem.org/api/JuliaFEM.elasticity_solver.html
This was our last issue in "getting started".
We need to decide, how to make the manual. I found one potential option: https://github.com/JuliaLang/JuliaDoc
If I understood correctly this haven't been fully decided in the Julia: https://github.com/JuliaLang/julia/issues/3988
If you can find more Julia style approach, please bring it to the discussion.