Update pyproject.toml; update docs folder

[bonicim]

• Puts sphinx dependencies in pyproject.toml so that users don't have to manually install sphinx to generate docs
• Updates README with instructions on how to use sphinx documentation
• Adds more dev-dependencies that are promised in README in https://github.com/VOLTTRON/copier-poetry-volttron-agent/pull/41/files

How tested:

I built a test agent using this branch. I tested out Makefile under docs. I also verified that the dev tools (e.g. safety, yapf, pre-commit, mypy) were working and properly installed.

~/test-36/docs via test-36
➜ make html
Running Sphinx v4.5.0
making output directory... done
[autosummary] generating autosummary for: index.rst
loading intersphinx inventory from https://docs.python.org/3/objects.inv...
loading intersphinx inventory from https://www.sphinx-doc.org/en/master/objects.inv...
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: [new config] 1 added, 0 changed, 0 removed
reading sources... [100%] index
/Users/boni407/test-36/docs/source/index.rst:16: WARNING: toctree contains reference to nonexisting document 'usage'
/Users/boni407/test-36/docs/source/index.rst:16: WARNING: toctree contains reference to nonexisting document 'agent'
/Users/boni407/test-36/docs/source/index.rst:16: WARNING: toctree contains reference to nonexisting document 'quick-start'
/Users/boni407/test-36/docs/source/index.rst:16: WARNING: toctree contains reference to nonexisting document 'fail-url-test-on-purpose'
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] index
/Users/boni407/test-36/docs/source/index.rst:6: WARNING: unknown document: usage
/Users/boni407/test-36/docs/source/index.rst:6: WARNING: undefined label: installation
generating indices... genindex done
writing additional pages... search done
copying static files... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded, 6 warnings.

The HTML pages are in build/html.

====================== slowest reading durations =======================
0.027 index

~/test-36 via test-36 via 🐍 system 3.9.10
➜ which safety
/Users/boni407/test-36/.venv/bin/safety

~/test-36 via test-36 via 🐍 system 3.9.10
➜ which mypy
/Users/boni407/test-36/.venv/bin/mypy

~/test-36 via test-36 via 🐍 system 3.9.10
➜ which yapf
/Users/boni407/test-36/.venv/bin/yapf

~/test-36 via test-36 via 🐍 system 3.9.10
➜ which pre-commit
/Users/boni407/test-36/.venv/bin/pre-commit

~/test-36 via test-36 via 🐍 system 3.9.10
➜ which sphinx-build
/Users/boni407/test-36/.venv/bin/sphinx-build

[kefeimo]

@bonicim I feel this approach is a bit advanced for template users.

• (Assuming myself is a non-advanced user), following the common practice like Pandas and Numpy, I am going to look for requests.txt, instead of looking into pyproject.toml.
• I argue that, people normally know what to do with requirements.txt but I am not sure about pyproject.toml.
• I also argue that sphinx is not the key feature of the custom-agent-project, i.e., the content of the documentation is what matters, instead of the rendering result using sphinx. So, maybe we don't need to make it so automatic.

[schandrika]

@kefeimo just some thoughts on your comments. Let me know if this addresses your concerns

• regarding pyproject.toml - There are numerous packaging solutions, we are simply giving a template and the user is free to use whatever they prefer. As long we document a way to package and push code, I would consider it a logical end point for our template.
• sphinx is simply what we use for the volttron core and agents documentation. I feel using sphinx is just good for consistency sake. Again I feel this is only a template and as long as it is clearly documented and none of the pieces here raises exception I would call it a good logical stopping point. We can surely add/improve once we have more users using this

[bonicim]

@bonicim I feel this approach is a bit advanced for template users.

• (Assuming myself is a non-advanced user), following the common practice like Pandas and Numpy, I am going to look for requests.txt, instead of looking into pyproject.toml.
• I argue that, people normally know what to do with requirements.txt but I am not sure about pyproject.toml.
• I also argue that sphinx is not the key feature of the custom-agent-project, i.e., the content of the documentation is what matters, instead of the rendering result using sphinx. So, maybe we don't need to make it so automatic.

During the feedback on Wednesday's meeting, I thought we were going to automate the installation of sphinx so user's don't have to install themselves, with the goal of allowing users to render the documentation using sphinx so that they can see it on a broswer. Maybe I'm wrong, but what should be the user experience with documentation building?

As for requirements.txt, we don't provide that since we are using pyproject.toml to store all the dependencies.

[schandrika]

@bonicim My understanding is the same. We decided to use sphinx to make html docs for agents.

[kefeimo]

~@bonicim @schandrika, my concern is mainly from the design principles of decoupling, and isolation.~ ~ I don't think it is a good idea to ship sphinx dependencies with the user package requirement. (If I understand correctly, once ruuning poetry install, users will have all dependencies requested by your pyproject.toml file. ~ ~ Use Pandas and Numpy as examples. Neither of them will install sphinx dependencies when running pip install numpy (or pip install pandas). i.e., there is no doc dependencies in their pyproject.toml (of numpy, of pandas). Numpy isolates documentation using doc_requirements.txt, Pandas use environment.yml;~ ~* From a user experience point of view, I don't think it is a lot of gain by directly running make html. (I personally use a separate virtual environment to build html from sphinx. And I would specify requirements.txt under doc to render reproducible documentation in ReadtheDocs. ref: Reproducible Builds - Read the Docs: Documentation Simplifie) On the contrary, it is very troublesome to resolve dependency conflict, e.g., the user has a conflict with sphinx version in pyproject.toml.~

@bonicim @schandrika, thanks for the explanation. Ref: Dependency Management With Python Poetry ref: Package Python Projects the Proper Way with Poetry

[bonicim]

~@bonicim @schandrika, my concern is mainly from the design principles of decoupling, and isolation.~ ~ I don't think it is a good idea to ship sphinx dependencies with the user package requirement. (If I understand correctly, once ruuning poetry install, users will have all dependencies requested by your pyproject.toml file. ~ ~ Use Pandas and Numpy as examples. Neither of them will install sphinx dependencies when running pip install numpy (or pip install pandas). i.e., there is no doc dependencies in their pyproject.toml (of numpy, of pandas). Numpy isolates documentation using doc_requirements.txt, Pandas use environment.yml;~ ~* From a user experience point of view, I don't think it is a lot of gain by directly running make html. (I personally use a separate virtual environment to build html from sphinx. And I would specify requirements.txt under doc to render reproducible documentation in ReadtheDocs. ref: Reproducible Builds - Read the Docs: Documentation Simplifie) On the contrary, it is very troublesome to resolve dependency conflict, e.g., the user has a conflict with sphinx version in pyproject.toml.~

@bonicim @schandrika, thanks for the explanation. Ref: Dependency Management With Python Poetry ref: Package Python Projects the Proper Way with Poetry

@kefeimo The sphinx dependency is declared under the section, tool.poetry.dev-dependencies. That section contains "packages that contributing developers should download to iterate on this project. Dev dependencies are not required to run the app, and won't be downloaded when the app is built by default.". Users do have the option to install only the required dependencies using poetry install --no-dev. I can probably add that blurb in the README.