Documentation: generate lightweight documentation for as_packages/kep package

[epogrebnyak]

I want some quick and lightweight auto-generated documentation for https://github.com/epogrebnyak/rosstat-kep-data/tree/as_packages/kep

We have some documentation generated with pydoc and pycco - run by https://github.com/epogrebnyak/rosstat-kep-data/blob/master/src/make_docs.bat/ as desribed in issue #25. The intent is still to

generate lightweight documentation that can help enhance comments and structure code better.

the original make_docs.bat soultion is not longer good for kep.package, becasue it has many subfolders and I could not make these progams parse subfolders effectively.

Perhaps SPHYNX can do the job, but so far it is too complex for me.

[epogrebnyak]

As see that changing line https://github.com/epogrebnyak/rosstat-kep-data/blob/master/src/make_docs.bat/#L30 is easiest for pycco method, no clue how to make pydoc work well, suggestions about pydoc, pycco, SPHINX or other package wlecome.

[epogrebnyak]

Guidelines to generate Sphinx documentation (for src folder, we need kep though), not tested, but promising.

Steps:

• start sphinx-apidoc
• make some changes in conf.py
• generate documentation with docs/make html

У Sphinx есть утилита, которая может в две команды генерировать документацию: http://sphinx-doc.org/invocation.html#invocation-of-sphinx-apidoc , http://stackoverflow.com/a/8086048/1214547 . На нашем коде её можно вызывать из папки src/ примерно так:

sphinx-apidoc -F -H kep -A epogrebnyak -V 0.1 -o ../docs .

После первого вызова она сгенерирует в папке docs несколько файлов, среди них conf.py. Там надо будет раскомментировать строку, добавляющую в sys.path один элемент, и заменить '.' на '..' (потому что Sphinx импортирует файлы, прежде чем генерировать по ним документацию, и это позволит ему увидеть src). После этого можно будет вызывать из папки docs команду make html, и это сгенерирует документацию. Итого, после первого запуска можно всё делать примерно таким bat-файлом, лежащим в src/:

sphinx-apidoc -F -H kep -A epogrebnyak -V 0.1 -o ../docs .
cd ../docs
make html
cd ../src

[gabrielelanaro]

Hi, sorry for late response.

I think the best documentation tool is sphinx, that's what I use for my projects, mostly because it is more well-maintained and estabilished.

I'm not too familiar with sphinx-apidoc, what I do is manually write .rst files like this one:

https://raw.githubusercontent.com/chemlab/chemlab/master/docs/api/chemlab.core.rst

Where there are sphinx directives needed to generate the documents. Notice that to use those directives the modules must be importable when you do "make html" on the sphinx project. (In the example file I sent you, for example, chemlab.core.Atom must be importable)

Also, you need to activate the autodoc extension in sphinx like in this line:

https://github.com/chemlab/chemlab/blob/master/docs/conf.py#L81

As far as I know sphinx-apidoc will generate those rst files for you.

[epogrebnyak]

@vrastonero - Denis, can you generate sphynx documentation in docs folder based on comments above?

[vrastonero]

@epogrebnyak - Yes, I'll try. By the way, should I perform this action in as_packages branch as described in title (it seems deprecated now) or in master?

[epogrebnyak]

Master branch better now or a new branch

пн, 14 дек. 2015, 14:42, vrastonero notifications@github.com:

@epogrebnyak https://github.com/epogrebnyak - Yes, I'll try. By the way, should I perform this action in as_packages branch as described in title (it seems deprecated now) or in master?

— Reply to this email directly or view it on GitHub https://github.com/epogrebnyak/rosstat-kep-data/issues/49#issuecomment-164416902 .

[vrastonero]

@epogrebnyak - Well, I've done it. There are some issues I have to point at.

1. I couldn't run sphinx-apidoc from /kep because of module name clash: kep.io overwrites standard i/o streams. So I put bat-file level above.
2. Docs created with errors caused by invalid paths in io.settings.py and selector.var_names.py, so I fixed paths.
3. I forced to add ../docs/_build directory with html files to repository.

[alexanderlukanin13]

@epogrebnyak Sphinx is added in https://github.com/epogrebnyak/rosstat-kep-data/pull/64 Should we close this issue and open concrete issues about documenting particular modules?

[epogrebnyak]

Let's keep it open, I have not tested the documentation scripts. Is generated documentation itself in repo now?

воскресенье, 20 декабря 2015 г. пользователь Alexander Lukanin написал:

@epogrebnyak https://github.com/epogrebnyak Sphinx is added in #64 https://github.com/epogrebnyak/rosstat-kep-data/pull/64 Should we close this issue and open concrete issues about documenting particular modules?

— Reply to this email directly or view it on GitHub https://github.com/epogrebnyak/rosstat-kep-data/issues/49#issuecomment-166075685 .

[alexanderlukanin13]

@epogrebnyak Necessary Sphinx files are in repo. Files generated by sphinx-apidoc are not in repo:

kep.*.rst
modules.rst

I don't recommend adding them to source control because they are completely auto-generated.

HTML output files (or any other output files in docs/build) are added to .gitignore.

Please see docs/README.md for details.

[epogrebnyak]

1d9d5a7ee6c631de7d7366af77e5e5a2db818371 switches off failing documentation test

[epogrebnyak]

@Pastafarianist: can you test if documentation is generated ok and it is meaningful? My intent/scenario/need is to browse through HTML as rerfenece to see if the code is well-documented/structured, but I do not want spend time learning setting up spynx.

[epogrebnyak]

For now there seems to be nothing in /docs/build/html/index.html (generated afer running make html)

[dniku]

I've fixed the instructions for building the documentation and the configuration script. Please see https://github.com/epogrebnyak/rosstat-kep-data/blob/master/docs/README.md. They work on my computer with the current state of the repository. The last step (starting a web server) is optional, you can just open index.html in a browser.

Also, Sphinx has produced quite a lot of warnings, we may want to look into them.

.../rosstat-kep-data/docs/source/kep.database.rst:19: WARNING: missing attribute mentioned in :members: or __all__: module kep.database, attribute common
.../rosstat-kep-data/docs/source/kep.database.rst:19: WARNING: missing attribute mentioned in :members: or __all__: module kep.database, attribute csv2db
.../rosstat-kep-data/docs/source/kep.database.rst:19: WARNING: missing attribute mentioned in :members: or __all__: module kep.database, attribute database
.../rosstat-kep-data/docs/source/kep.database.rst:19: WARNING: missing attribute mentioned in :members: or __all__: module kep.database, attribute hardcoded
.../rosstat-kep-data/docs/source/kep.database.rst:19: WARNING: missing attribute mentioned in :members: or __all__: module kep.database, attribute label_csv
.../rosstat-kep-data/docs/source/kep.database.rst:19: WARNING: missing attribute mentioned in :members: or __all__: module kep.database, attribute load_spec
.../rosstat-kep-data/docs/source/kep.database.rst:19: WARNING: missing attribute mentioned in :members: or __all__: module kep.database, attribute plots
.../rosstat-kep-data/docs/source/kep.database.rst:19: WARNING: missing attribute mentioned in :members: or __all__: module kep.database, attribute query
.../rosstat-kep-data/docs/source/kep.database.rst:19: WARNING: missing attribute mentioned in :members: or __all__: module kep.database, attribute stream
.../rosstat-kep-data/docs/source/kep.database.rst:19: WARNING: missing attribute mentioned in :members: or __all__: module kep.database, attribute temp
.../rosstat-kep-data/docs/source/kep.database.rst:19: WARNING: missing attribute mentioned in :members: or __all__: module kep.database, attribute var_names
.../rosstat-kep-data/docs/source/kep.database.rst:19: WARNING: missing attribute mentioned in :members: or __all__: module kep.database, attribute word
.../rosstat-kep-data/kep/database/__init__.py:docstring of kep.database:8: ERROR: Unexpected indentation.
.../rosstat-kep-data/kep/database/__init__.py:docstring of kep.database:9: WARNING: Block quote ends without a blank line; unexpected unindent.
.../rosstat-kep-data/kep/database/__init__.py:docstring of kep.database:11: ERROR: Unexpected indentation.
.../rosstat-kep-data/kep/database/__init__.py:docstring of kep.database:12: WARNING: Block quote ends without a blank line; unexpected unindent.
.../rosstat-kep-data/kep/database/__init__.py:docstring of kep.database:21: ERROR: Unexpected indentation.
.../rosstat-kep-data/kep/file_io/specification.py:docstring of kep.file_io.specification.load_spec:6: WARNING: Definition list ends without a blank line; unexpected unindent.
.../rosstat-kep-data/kep/file_io/specification.py:docstring of kep.file_io.specification.load_spec:6: SEVERE: Unexpected section title or transition.

-----
.../rosstat-kep-data/kep/query/plots.py:docstring of kep.query.plots.write_plots:1: WARNING: Inline emphasis start-string without end-string.
.../rosstat-kep-data/docs/source/kep.query.rst:43: WARNING: missing attribute mentioned in :members: or __all__: module kep.query, attribute common
.../rosstat-kep-data/docs/source/kep.query.rst:43: WARNING: missing attribute mentioned in :members: or __all__: module kep.query, attribute csv2db
.../rosstat-kep-data/docs/source/kep.query.rst:43: WARNING: missing attribute mentioned in :members: or __all__: module kep.query, attribute database
.../rosstat-kep-data/docs/source/kep.query.rst:43: WARNING: missing attribute mentioned in :members: or __all__: module kep.query, attribute hardcoded
.../rosstat-kep-data/docs/source/kep.query.rst:43: WARNING: missing attribute mentioned in :members: or __all__: module kep.query, attribute label_csv
.../rosstat-kep-data/docs/source/kep.query.rst:43: WARNING: missing attribute mentioned in :members: or __all__: module kep.query, attribute load_spec
.../rosstat-kep-data/docs/source/kep.query.rst:43: WARNING: missing attribute mentioned in :members: or __all__: module kep.query, attribute query
.../rosstat-kep-data/docs/source/kep.query.rst:43: WARNING: missing attribute mentioned in :members: or __all__: module kep.query, attribute stream
.../rosstat-kep-data/docs/source/kep.query.rst:43: WARNING: missing attribute mentioned in :members: or __all__: module kep.query, attribute temp
.../rosstat-kep-data/docs/source/kep.query.rst:43: WARNING: missing attribute mentioned in :members: or __all__: module kep.query, attribute word
.../rosstat-kep-data/kep/query/__init__.py:docstring of kep.query:8: ERROR: Unexpected indentation.
.../rosstat-kep-data/kep/query/__init__.py:docstring of kep.query:9: WARNING: Block quote ends without a blank line; unexpected unindent.
.../rosstat-kep-data/kep/query/__init__.py:docstring of kep.query:11: ERROR: Unexpected indentation.
.../rosstat-kep-data/kep/query/__init__.py:docstring of kep.query:12: WARNING: Block quote ends without a blank line; unexpected unindent.
.../rosstat-kep-data/kep/query/__init__.py:docstring of kep.query:21: ERROR: Unexpected indentation.

For our particular use case, module index from Sphinx-generated documentation is probably the most interesting. Please have a look at it and close the issue if it looks okay to you.

[epogrebnyak]

1. Can you think customizing index.html - it only has two working links, the rest of test is generic and a bit useless
2. What is the logic behind keeping html files in .gitignore?

[dniku]

The logic of keeping HTML in .gitignore is to reduce the size of the repository. These files are generated automatically, and they can produce pretty large commits if we keep them in the repo. Personally, I regard the repository as the storage for files that need to be versioned, nothing else.

It also becomes hard to track changes I make to the code if there are a lot of files which are generated automatically: git status starts showing a lot of garbage.

However, we already break this logic pretty badly by storing images, PDFs and SQLite databases, so we might as well add the docs, although I'd prefer not only not to add them, but also to transfer the other binary/autogenerated files somewhere outside of the current repo.

[dniku]

I have updated the index page:

[epogrebnyak]

@Pastafarianist: SPHINX is broken again - I deleted some configs. Can you restore in mater please?

[dniku]

@epogrebnyak by pure accident, I have just noticed your comment. Is this still a problem?