Integrate plone.restapi docs with Plone 6 Documentation

[stevepiercy]

The Plone Documentation Team is actively working on improving documentation for Plone 6.

See https://github.com/plone/documentation/issues/1178

The Plone Documentation Team has already integrated Volto docs, and now we're coming for plone.restapi. 😁

The Volto development team wanted to keep their docs together with code, build a standalone set (at least temporarily), and integrate them into the main Plone 6 Documentation.

I understand that the docs are currently hosted separately at RTD:

https://plonerestapi.readthedocs.io/en/latest/

Does the plone.restapi team wish to do the same as the Volto team, do nothing, or do something else?

See also:

• https://community.plone.org/t/plone-6-documentation-ready-for-contributions/14741
• https://community.plone.org/t/plone-6-documentation-adds-volto-frontend-docs-automatic-deployments/14786

[tisto]

@stevepiercy we would definitely prefer to keep the docs close to the code. We follow a "documentation first" approach on new features and I don't accept PRs for features that do not come with proper docs. This only works if code and docs are in one place.

Our docs use https://github.com/collective/sphinxcontrib-httpexample so we won't be able to move to MK anytime soon I guess. Since we are using Sphinx that shouldn't be an issue though.

I personally won't have any time this year to work on this I am afraid. If anyone wants to pick this up I am happy to do what I can to help though.

[erral]

We were able to use sphinxcontrib-httpexample with Markdown files in a Sphinx based documentation here:

https://github.com/eea/clms-api-docs

The syntax is a bit different if you compare it with RST, but it works:

https://github.com/eea/clms-api-docs/blob/master/source/download.md?plain=1#L25

https://eea.github.io/clms-api-docs/download.html#find-the-items-to-be-downloaded

My 2 cents.

[stevepiercy]

@tisto got it. We will keep docs with code.

We used rst-2-myst as a conversion tool for Plone Trainings.

We would create a new long-lived feature branch to test things out and make sure that we retain all the current desired features, similar to what we are doing with Volto docs.

The Docs Team would do all the work, but we would like to have reviews from core contributors to the project. We use Netlify to preview pull requests, or we could use ReadTheDocs, since you already have that set up (and it is easier to do so). Can you provide a list of core contributors who have sufficient familiarity to review and approve pull requests?

@erral thanks for the snippet. That's very helpful.

[stevepiercy]

So this was a surprise. I found I did not need to convert plone.restapi docs from reStructuredText to MyST to import them into the main Plone 6 Docs. Check this out:

With that information, do the people who actually write documentation for plone.restapi prefer to write documentation in reStructuredText or MyST? I don't have an opinion, except that continuing to write in reST means that I would not have to convert files, and that git history would be retained. AFAIK, when converting files, git history cannot track between a change in filename from *.rst to *.md. It's possible writing in MyST is easier as it is Markdown with adapted reST syntax tossed in, and so you might get more contributors.

[tisto]

@stevepiercy w00t! That's awesome. I wouldn't mind switching to MK. I actually prefer to write MK over ReST these days. Though, most pages in the restapi docs contain http examples that rely on ReST. Therefore I don't see how we could move anytime soon. We would need to rewrite the http examples ReST extension to MK first.

[stevepiercy]

MyST syntax is a superset of Markdown and modified reST. rst-2-myst converts almost everything. For example:

..  http:example:: curl httpie python-requests
    :request: ../../src/plone/restapi/tests/http-examples/principals.req

becomes

```{eval-rst}
..  http:example:: curl httpie python-requests
    :request: ../../src/plone/restapi/tests/http-examples/principals.req

I don't know for sure whether it will convert this yet. I will test tonight.

[stevepiercy]

I should clarify the issue of git history with conversion to MyST. The history of the *.rst files is preserved but stops at the point of conversion to *.md because the file gets renamed. You just cannot do git blame file.md to see beyond the conversion datetime, and instead need to do git blame file.rst.

@stevepiercy w00t! That's awesome. I wouldn't mind switching to MK. I actually prefer to write MK over ReST these days.

@tisto who else who writes docs these days should weigh in? It would be good to get consensus before doing a conversion.

[stevepiercy]

@tisto in order to build plone.restapi in the main docs with http.example, is it correct that these two directories must be present with this hierarchy?

• plone.restapi/src
• plone.restapi/docs/source

For example:

plone.restapi/docs/source/blocks.rst

.. http:example:: curl httpie python-requests
   :request: ../../src/plone/restapi/tests/http-examples/actions_get.req

All that works fine for standalone docs, but when I import it into the main docs, then Sphinx cannot find the path to ../../src/plone/restapi/tests/http-examples/actions_get.req. I tried several ways to import the docs without editing that path, but failed.

If we moved that http-examples directory out of src and into docs/source, would that break anything?

[stevepiercy]

Ultimately the problem is that plone/documentation has symlinks in docs to submodules:

.
├── Makefile
├── docs
│   ├── _static
│   ├── _templates
│   ├── addMetaData.py
│   ├── backend
│   ├── classic-ui
│   ├── conf.py
│   ├── contributing
│   ├── glossary.md
│   ├── index.md
│   ├── plone.restapi -> ../submodules/plone.restapi/docs/source
│   ├── robots.txt
│   ├── spelling_wordlist.txt
│   └── volto -> ../submodules/volto/docs/source
└── submodules
    ├── plone.restapi
    └── volto

Thus when building the whole documentation, a reference in a file such as ./plone.restapi/actions.rst is unable to locate ../../src/plone/restapi/tests/http-examples/actions_get.req. When it goes up two directories relative to itself, there is no src directory to go into.

I came up with one option where I normalized src directories. Here's what I did:

1. In plone/plone.restapi/docs/source/, find ../../src/plone/restapi/tests/http-examples, and replace with ../plone.restapi.http-examples/
2. Make symlinks.

# From the root of `plone/plone.restapi`
ln -s ../src/plone/restapi/tests/http-examples ./docs/plone.restapi.http-examples

# From the root of `plone/documentation`
ln -s ../submodules/plone.restapi/docs/source ./docs/plone.restapi.docs
ln -s ../submodules/plone.restapi/src/plone/restapi/tests/http-examples ./docs/plone.restapi.http-examples

I added the creation of the symlinks to the Makefile in each repo, and it seems to work. Here's the restult of tree -L 2 . to give y'all a better picture.

.
├── Makefile
├── docs
│   ├── _static
│   ├── _templates
│   ├── addMetaData.py
│   ├── backend
│   ├── classic-ui
│   ├── conf.py
│   ├── contributing
│   ├── glossary.md
│   ├── index.md
│   ├── plone.restapi.docs -> ../submodules/plone.restapi/docs/source
│   ├── plone.restapi.http-examples -> ../submodules/plone.restapi/src/plone/restapi/tests/http-examples
│   ├── robots.txt
│   ├── spelling_wordlist.txt
│   └── volto -> ../submodules/volto/docs/source
└── submodules
    ├── plone.restapi/
    └── volto

The big question is, Are y'all OK with editing the path and adding a symlink to your Makefile?

I don't have any other solution to this problem, but I'm open to any suggestions.

[tisto]

@stevepiercy not sure I can fully follow. Though, just go ahead and create a PR or a new Makefile command. I guess we could just add another makefile command for the plone6 docs and this wouldn't interfere with anything else, right?

[stevepiercy]

@tisto I'll push commits to the two repos. Could I have access to RTD to enable the branch plone6docs for publication as hidden and stand alone? Either that, or have someone who has sufficient access to do so. I'm stevepiercy on RTD. I want to make sure that the stand alone docs will still build with this change.

[tisto]

@stevepiercy I added you as maintainer to RTD.

[stevepiercy]

w00t! I don't believe it. https://plonerestapi.readthedocs.io/en/plone6docs/actions.html#listing-available-actions Whee!

[stevepiercy]

Oh, wait, that uses the old paths. Testing again.

[stevepiercy]

Bah. I cannot find how to do it on RTD, as it completely ignores Makefile. I don't think they support making symlinks like Netlify does.

[stevepiercy]

I made a request to grant Netlify access to plone/plone.restapi. Would someone with GitHub Plone Organization access please approve the request?

[tisto]

@stevepiercy I approved your request.

[stevepiercy]

Bah, I cannot get GitHub Actions to honor symlinks either. https://github.com/plone/plone.restapi/runs/5357509841?check_suite_focus=true#step:10:146

[tisto]

@stevepiercy can't you just run the tests to create the example files in the right location or copy them over?

[stevepiercy]

I will try that.

[stevepiercy]

Thanks for the idea @tisto. I got it working locally. It still fails on GitHub Actions. I have no clue what I am missing. Is it not possible to do a simple cp -R?

https://github.com/plone/plone.restapi/runs/5358667529?check_suite_focus=true#step:10:132

[stevepiercy]

I completely changed my approach to this problem. Just symlink the root of the project, adjust a few paths here and there, and it works.

[stevepiercy]

For maintainers, returning to the question of whether to convert all the files from reST to MyST: Is that something you want me to do now, later, or not at all? I'm happy to go along with the consensus. Please let me know. Thank you!

[stevepiercy]

I'm going to close this issue as complete, and open a new issue to discuss whether to convert the documentation from reST to MyST.