new moveit_core python documentation

[PeterMitrano]

This is sort of a meta-issue for tracking documentation of the new python bindings. I'm worried things might not be working, but maybe it's just taking time for things to sync.

First of all, I see that http://docs.ros.org/en/noetic/api/moveit_core/html/ now has python and c++ which suggests my docs changes in #2547 are happening, however the python docs are not showing up.

I'm also seeing a failed build here

https://travis-ci.com/github/ros-planning/moveit_tutorials/jobs/497517586

which is weird because for the C++ external links it's complaining don't exist, do exist, for example:

http://docs.ros.org/noetic/api/moveit_core/html/cpp/classplanning__scene_1_1PlanningScene.html

[PeterMitrano]

I'm looking on the build farm to try to find why it's not happening, but I can't find any docs jobs for moveit_core. If anyone has suggestions where to look or what might be wrong I'd appreciate it

[rhaschke]

I'm also seeing a failed build here https://travis-ci.com/github/ros-planning/moveit_tutorials/jobs/497517586

This was due to the cpp subfolder introduced with your cpp + python doc config. It was fixed in https://github.com/ros-planning/moveit_tutorials/pull/628

I'm looking on the build farm to try to find why it's not happening, but I can't find any docs jobs for moveit_core.

The doc job is here: https://build.ros.org/view/Ndoc/job/Ndoc__moveit__ubuntu_focal_amd64/ Could you compare the output there with your local one?

[PeterMitrano]

Thanks! I see an error:

06:25:49 Configuration error:
06:25:49 There is a programmable error in your configuration file:
06:25:49 
06:25:49 Traceback (most recent call last):
06:25:49   File "/usr/lib/python3/dist-packages/sphinx/config.py", line 368, in eval_config_file
06:25:49     execfile_(filename, namespace)
06:25:49   File "/usr/lib/python3/dist-packages/sphinx/util/pycompat.py", line 150, in execfile_
06:25:49     exec_(code, _globals)
06:25:49   File "/tmp/ws/src/moveit/moveit_core/doc/conf.py", line 17, in <module>
06:25:49     import sphinx_rtd_theme
06:25:49 ModuleNotFoundError: No module named 'sphinx_rtd_theme'

Seems like that's not supported in whatever build environment this job is running in. Do you know how to change that? I.e. something like pip install sphinx-rtd-theme ? The default docs theme is really difficult to read and navigate, so I'd like to at least try to make the RTD theme work.

[rhaschke]

I guess you need to pull in the dependencies explicitly via <doc_depend>python3-sphinx-rtd-theme</doc_depend> in package.xml.

[PeterMitrano]

oh ok, I'll make a PR and try that. I noticed that locally using doc_depend doesn't work with rosdep (https://github.com/ros-infrastructure/rosdep/issues/498), so if that's what the build farm uses then we'll have to make it a regular depend

[rhaschke]

Hm. I hope that the build farm uses a different mechanism then...

[rhaschke]

Let's shift the discussion from https://github.com/ros-planning/moveit/pull/2645#issuecomment-834912867 to this open issue here. The remaining issue is that we need a full build to generate our pybind11 bindings from which we then can auto-generate the python docs via rosdoc_lite / sphinx.

The solution suggested by @PeterMitrano in https://github.com/ros-planning/moveit/pull/2645#issuecomment-835455764 is to commit the auto-generated python docs into the source repository. This also could be automated by our CI process. However, I don't like the idea to include auto-generated stuff into the git repository.

Alternatively, we could run rosdoc_lite as part of our GHA workflow and rsync the generated docs ourselves to http://docs.ros.org. @tfoote, would that be possible, i.e. would you be willing to grant us the corresponding permission?

[gavanderhoorn]

What about the reverse? rosdoc_lite in CI, push results to some known place, have the buildfarm import the artefacts (perhaps as part of a custom doc job) and put them on docs.ros.org.

[rhaschke]

Would work as well, but requires an intermediate "known place" to push/pull from.

[PeterMitrano]

Perhaps a separate GitHub repo would be an ok intermediate place to have that?

On Thu, May 20, 2021, 05:14 Robert Haschke @.***> wrote:

Would work as well, but requires an intermediate "known place" to push/pull from.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ros-planning/moveit/issues/2606#issuecomment-844896664, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA6TGEQU5YTVKEAOL6STYXTTOTHGLANCNFSM43AMHIZA .

[PeterMitrano]

Is anyone familiar with the build/CI infra willing to look into this at some point? This issue is blocking the tutorials (https://github.com/ros-planning/moveit_tutorials/pull/624) and more generally the expansion of the python bindings.

[PeterMitrano]

so i started by trying to get rosdoc_lite installed on the CI server but I can't figure it out. Here's my attempt

https://github.com/UM-ARM-Lab/moveit/tree/build-docs-in-ci

I see in the logs that it installed it in the install_target_dependencies that it installed it, but I get rosdoc_lite, command not found. Perhaps it needs to be installed during the install_downstream_dependencies step but idk how to make that happen.

[rhaschke]

I just noticed that this issue is still open. As we didn't get feedback from @tfoote regarding https://github.com/ros-planning/moveit/issues/2606#issuecomment-844889302, I suggest the following options:

1. Build autosummary docs (.rst files) as part of CI (GHA) and auto-push those files to a branch noetic-doc, which is then used by rosdoc_lite / the doc build farm to generate the final docs. I guess we will need two sets of config files for these two steps. @PeterMitrano: do you think this could work? I couldn't figure out the first step to just generate the .rst files.
2. Hosting API docs on https://moveit.ros.org instead of https://docs.ros.org and generating them as part of the CI job of moveit_tutorials that builds the former web page. Actually, I prefer this latter option, because it has several benefits:
   • autosummary will work out-of-the-box: All MoveIt packages will be available for the CI build.
   • We could have a common landing page for all MoveIt packages (docs.ros.org is split by package).
   • The new pages could be linked from https://docs.ros.org for backward compatibility.

@davetcoleman, @v4hn: what do you think of the idea hosting API docs on https://moveit.ros.org as well?

[v4hn]

Don't expect a reply from Dave here. Send him a mail if you need his statement. Though it could work with the infrastructure, would discourage 1. as well. It sounds like too many workarounds just to fit in and automatically pushing branches is ugly overhead.

Hosting the API of packages in the moveit/moveit2 repos on moveit.ros.org makes sense if it gives us the flexibility to generate the docs any way we like. Concerning building them from moveit_tutorials, can we trigger these gh actions upon changes in the moveit repository? If we would need to trigger it manually from there I would prefer to setup an action on the main moveit branches.

We already have something like a "landing page" on the website but it would obviously be nicer to have it integrated with the doc style instead of maintaining a separate site nobody ever looks at.

Do you think you or @PeterMitrano could find the time to set this up?

[PeterMitrano]

I also like the second option, but I don't get why we'd piggyback on moveit_tutorials?

Regarding this page https://moveit.ros.org/documentation/source-code-api/ I don't think people use this, fwiw.

I can spend a bit of time on this, I'm more motivated if I know you two are available for help/review. I did a bit of poking into the GHAs for moveit when I was trying to generate docs, and got stuck with rosdoc_lite not being available, so I feel like we stlil have that problem?

where should I look to start? Here? https://github.com/ros-planning/moveit_tutorials/runs/3886517242?check_suite_focus=true seems like that's generating docs. Maybe we can set up an analogous action in the moveit repo that builds docs and deploys to moveit.ros.org ?

[rhaschke]

Both of you, Peter and Michael, pointed out an important issue I didn't yet consider: If we use moveit_tutorials CI to generate the API docs, we won't get the latest updates from the MoveIt core repo. However we could setup a cron job...

Peter, I guess you think of generating docs from MoveIt's core GHA, but injecting them into moveit.ros.org, right? That's a viable option to solve the first issue. However, I'm not sure about the structure of the website. Who is responsible for the site at Picknik, @nbbrooks or @skolk? I was surprised to not find any impressum on the site...

I would like to have a short chat with someone over at Picknik to sort this out. If we have a way to go, I'm happy to setup the initial config.

[gavanderhoorn]

Both of you, Peter and Michael, pointed out an important issue I didn't yet consider: If we use moveit_tutorials CI to generate the API docs, we won't get the latest updates from the MoveIt core repo.

can't you use something like peter-evans/repository-dispatch to trigger a workflow run from moveit?

I was surprised to not find any impressum on the site...

that's a really German thing to say ;)

[rhaschke]

Sure, that's an option as well. Thanks for pointing it out! However, I like the idea of building the API doc independently from moveit.ros.org, but injecting them into the site.

[nbbrooks]

Tagging @tylerjw and @vatanaksoytezer to relay how ROS 2 solves this and see if it is applicable for ROS 1.

[tylerjw]

I can't comment on how we should go about this for python bindings as that seems like a technical chicken and egg problem that will have to be solved somehow in CI. As to how official API documentation is being generated and the current state of it there is this issue: https://github.com/osrf/infrastructure/issues/10

In the near future, we hope to have API documentation built into the docs.ros.org website again for recent versions of MoveIt.

[rhaschke]

@nbbrooks and @tylerjw, this is neither related to ROS2 nor is it a chicken-and-egg problem. I have developed a procedure to build the API docs as part of our CI. You just have to decide where to deploy them, see https://github.com/ros-planning/moveit/pull/2914.

[PeterMitrano]

Where do we stand on this? is there anything I should do to keep it moving? @rhaschke @tylerjw

[v4hn]

So it looks like moveit.ros.org directly maps to github pages, so the whole domain content is handled there. From some very preliminary resource browsing it looks like all the website has to be deployed in one job.

I just looked at the moveit.ros.org repo a bit and it seems the build-and-deploy workflow is automatically setup for the main branch in the "pages" config. This could be changed to an explicit workflow that involves the deploy action.

So @rhaschke, it looks like nobody is really responsible for the website which is why there was no response here. There's just the (at the moment trivial) github pages config and a registered subdomain that refers to a github server.

I believe the best way forward would be to implement a custom deploy action that generates the documentation from the moveit repository and trigger the workflow in moveit.ros.org from this repo here whenever the relevant files change. In theory I guess there could be some caching that does not require the docs to be rebuilt every time someone changes the website.

Thanks for mentioning this effort in the maintainer meeting @PeterMitrano .

[vatanaksoytezer]

We used https://github.com/ros-planning/moveit2_tutorials/commit/f680cbe8533f6049986b09165806c8ae019fe31d for generating API for MoveIt2, does it look reasonable to do something similar for MoveIt?