rosdoc2 --package-path `<work_space>/.../rclpy/rclpy` generates many warnings and some pages are empty.

fujitatomoya commented 3 months ago

Description

I was trying to build the doc for https://github.com/ros2/rclpy with rosdoc2. build process completes but there are many warnings came up and some API pages are just empty.

How to reproduce

root@tomoyafujita:~/ros2_ws/colcon_ws# rosdoc2 build --package-path ./src/ros2/rclpy/rclpy
[rosdoc2] [INFO] Note: the user provided sourcedir for Sphinx '/root/ros2_ws/colcon_ws/src/ros2/rclpy/rclpy/docs/source/' will be used.
...

Warning Messages

[autosummary] generating autosummary for: about.rst, api.rst, api/actions.rst, api/clock.rst, api/context.rst, api/execution_and_callbacks.rst, api/init_shutdown.rst, api/logging.rst, api/node.rst, api/parameters.rst, ..., rclpy.type_hash.rst, rclpy.type_support.rst, rclpy.utilities.rst, rclpy.validate_full_topic_name.rst, rclpy.validate_namespace.rst, rclpy.validate_node_name.rst, rclpy.validate_parameter_name.rst, rclpy.validate_topic_name.rst, rclpy.wait_for_message.rst, rclpy.waitable.rst
WARNING: Failed to import rclpy.signals.
Possible hints:
* ModuleNotFoundError: No module named 'rclpy._rclpy_pybind11'
The C extension '/root/ros2_ws/colcon_ws/build/rpyutils/_rclpy_pybind11.cpython-310-x86_64-linux-gnu.so' isn't present on the system. Please refer to 'https://docs.ros.org/en/rolling/How-To-Guides/Installation-Troubleshooting.html#import-failing-without-library-present-on-the-system' for possible solutions
* KeyError: 'rclpy'
WARNING: Failed to import rclpy.validate_parameter_name.
Possible hints:
* ModuleNotFoundError: No module named 'rclpy._rclpy_pybind11'
The C extension '/root/ros2_ws/colcon_ws/build/rpyutils/_rclpy_pybind11.cpython-310-x86_64-linux-gnu.so' isn't present on the system. Please refer to 'https://docs.ros.org/en/rolling/How-To-Guides/Installation-Troubleshooting.html#import-failing-without-library-present-on-the-system' for possible solutions
* KeyError: 'rclpy'
WARNING: Failed to import rclpy.event_handler.
Possible hints:
* ModuleNotFoundError: No module named 'rclpy._rclpy_pybind11'
The C extension '/root/ros2_ws/colcon_ws/build/rpyutils/_rclpy_pybind11.cpython-310-x86_64-linux-gnu.so' isn't present on the system. Please refer to 'https://docs.ros.org/en/rolling/How-To-Guides/Installation-Troubleshooting.html#import-failing-without-library-present-on-the-system' for possible solutions
* KeyError: 'rclpy'
WARNING: Failed to import rclpy.qos_overriding_options.
Possible hints:
* ModuleNotFoundError: No module named 'rclpy._rclpy_pybind11'
The C extension '/root/ros2_ws/colcon_ws/build/rpyutils/_rclpy_pybind11.cpython-310-x86_64-linux-gnu.so' isn't present on the system. Please refer to 'https://docs.ros.org/en/rolling/How-To-Guides/Installation-Troubleshooting.html#import-failing-without-library-present-on-the-system' for possible solutions
* KeyError: 'rclpy'
WARNING: Failed to import rclpy.impl.rcutils_logger.
Possible hints:
* ModuleNotFoundError: No module named 'rclpy._rclpy_pybind11'
The C extension '/root/ros2_ws/colcon_ws/build/rpyutils/_rclpy_pybind11.cpython-310-x86_64-linux-gnu.so' isn't present on the system. Please refer to 'https://docs.ros.org/en/rolling/How-To-Guides/Installation-Troubleshooting.html#import-failing-without-library-present-on-the-system' for possible solutions
* AttributeError: module 'rclpy.impl' has no attribute 'rcutils_logger'

Empty Page

originally i was trying to build https://github.com/ros2/rclpy/blob/eaee1d2b75d39072ea532517d39c6fa5e53452b1/rclpy/rclpy/topic_or_service_is_hidden.py#L19-L30 doc section, but it came up with.

Yadunund commented 3 months ago

Hi @fujitatomoya,

Thanks for reporting this problem.

I was not able to reproduce this locally for rolling or iron branchs of rclpy. Here's a screenshot from the local build which succeeds albeit with many warnings.

The buildfarm doc job also does not seem to have any problem as the documentation for rclpy.topic_or_service_is_hidden module can be seen here.

Could you share a gist with your entire terminal output?

fujitatomoya commented 3 months ago

@Yadunund thanks for replying, here is the terminal log output https://gist.github.com/fujitatomoya/b0d778a71ba205476c01291387101f9b

what i did is the following, i am not really sure if this is expected procedure.

https://github.com/ros-infrastructure/rosdoc2?tab=readme-ov-file#installation
build ROS 2 source with https://docs.ros.org/en/rolling/Installation/Alternatives/Ubuntu-Development-Setup.html#build-ros-2
source install/local_setup.bash in that workspace.
rosdoc2 build --package-path ./src/ros2/rclpy

Yadunund commented 3 months ago

Thanks for sharing the gist and steps to reproduce. The The doc job has similar warnings when it executes , see log, except there are no errors No module named 'rclpy._rclpy_pybind11' errors.

One thing that is different in your steps vs the docjob is that the latter does not source any built ROS 2 workspace before running rosdoc2 build. Could you try doing the same, ie skip step 2 & 3 above? Technically it should still work if a ROS workspace is sourced but maybe there is a bug somewhere.

fujitatomoya commented 3 months ago

ie skip step 2 & 3 above? Technically it should still work if a ROS workspace is sourced but maybe there is a bug somewhere.

the followings are exact commands that i used. hmmm, it does not work either... the error messages are almost same with previous procedure https://github.com/ros-infrastructure/rosdoc2/issues/66#issuecomment-1995299198

### start docker container
docker run -it --rm ros:rolling

### update and install packages
apt update && apt upgrade -y
apt install -y python3-pip

### install required packages for documentation
cd
git clone https://github.com/ros2/ros2_documentation.git
cd ros2_documentation/
pip install -r requirements.txt -c constraints.txt
cd ..

### install rosdoc2
git clone https://github.com/ros-infrastructure/rosdoc2.git
cd rosdoc2/
pip install --user --upgrade .
mv /root/.local/bin/rosdoc2 /usr/local/bin/
cd ..

### try to build rclpy doc via rosdoc2
git clone https://github.com/ros2/rclpy.git
rosdoc2 build --package-path ./rclpy/rclpy/
[rosdoc2] [INFO] Note: the user provided sourcedir for Sphinx '/root/rclpy/rclpy/docs/source/' will be used.
[rosdoc2] [INFO] Running sphinx-apidoc: 'sphinx-apidoc -o ../../../rclpy/rclpy/docs/source -e /root/rclpy/rclpy/rclpy' in 'docs_build/rclpy/rclpy'
[rosdoc2] [INFO] Running Sphinx-build: 'sphinx-build -c . ../../../rclpy/rclpy/docs/source /root/docs_build/rclpy/rclpy/sphinx_output' in 'docs_build/rclpy/rclpy'
Running Sphinx v7.2.6
[rosdoc2] enabling autodoc
[rosdoc2] enabling intersphinx
[rosdoc2] overriding theme to be 'sphinx_rtd_theme'
[rosdoc2] extending intersphinx mapping
[rosdoc2] adding markdown parser
WARNING: Invalid configuration value found: 'language = None'. Update your configuration to a valid language code. Falling back to 'en' (English).
[autosummary] generating autosummary for: about.rst, api.rst, api/actions.rst, api/clock.rst, api/context.rst, api/execution_and_callbacks.rst, api/init_shutdown.rst, api/logging.rst, api/node.rst, api/parameters.rst, ..., rclpy.type_hash.rst, rclpy.type_support.rst, rclpy.utilities.rst, rclpy.validate_full_topic_name.rst, rclpy.validate_namespace.rst, rclpy.validate_node_name.rst, rclpy.validate_parameter_name.rst, rclpy.validate_topic_name.rst, rclpy.wait_for_message.rst, rclpy.waitable.rst
WARNING: Failed to import rclpy.validate_namespace.
Possible hints:
* ModuleNotFoundError: No module named 'rclpy._rclpy_pybind11'
The C extension '/opt/ros/rolling/lib/python3.10/site-packages/_rclpy_pybind11.cpython-310-x86_64-linux-gnu.so' isn't present on the system. Please refer to 'https://docs.ros.org/en/rolling/How-To-Guides/Installation-Troubleshooting.html#import-failing-without-library-present-on-the-system' for possible solutions
* KeyError: 'rclpy'
WARNING: Failed to import rclpy.qos.
Possible hints:
* ModuleNotFoundError: No module named 'rclpy._rclpy_pybind11'
The C extension '/opt/ros/rolling/lib/python3.10/site-packages/_rclpy_pybind11.cpython-310-x86_64-linux-gnu.so' isn't present on the system. Please refer to 'https://docs.ros.org/en/rolling/How-To-Guides/Installation-Troubleshooting.html#import-failing-without-library-present-on-the-system' for possible solutions
* KeyError: 'rclpy'
WARNING: Failed to import rclpy.event_handler.
Possible hints:
* ModuleNotFoundError: No module named 'rclpy._rclpy_pybind11'
The C extension '/opt/ros/rolling/lib/python3.10/site-packages/_rclpy_pybind11.cpython-310-x86_64-linux-gnu.so' isn't present on the system. Please refer to 'https://docs.ros.org/en/rolling/How-To-Guides/Installation-Troubleshooting.html#import-failing-without-library-present-on-the-system' for possible solutions
...

rkent commented 3 months ago

I can duplicate the pybind11 errors using the description above. I can also eliminate them following the same instructions, with two small changes:

...
docker run -it --rm ubuntu:jammy
...
apt install -y python3-pip git
...

With those changes, I get the same sorts of errors that are typical of rosclpy (like @Yadunund) but no pybind11 errors. So there is something setup in the ros:rolling docker that is causing this. Note the buildfarm is mostly using ubuntu:jammy for running rosdoc2.

rkent commented 3 months ago

To add to this: docker image ros:rolling is derived from ros:rolling-ros-core-jammy, which also shows the pybind11 errors. That in turn is built from this Dockerfile which derives from ubuntu:jammy. Manually duplicating that install, I can reproduce the pybind11 errors using the recipe in that Dockerfile. However, if I leave off the last step (running source "/opt/ros/$ROS_DISTRO/setup.bash" -- in the ENTRYPOINT) then the pybind11 errors go away. So this problem is associated with trying to run rosdoc2 after sourcing ROS2.

fujitatomoya commented 3 months ago

@rkent thanks for the info, i successfully compiled the docs with your suggested procedure.

probably what we can do here is to add documents,

add https://github.com/ros2/ros2_documentation?tab=readme-ov-file#prerequisites to prerequisite and apt install -y python3-pip git.
DO NOT source ROS 2 colcon workspace.
(nice to have) Dockerfile, so that user can avoid having this problem.

Yadunund commented 3 months ago

@rkent thanks for sharing your valuable insights. Glad we found the cause of the problem and a temporary work around. I've opened https://github.com/ros-infrastructure/rosdoc2/issues/68 so that we may resolve the issue properly.

But I wouldn't generalize that ROS 2 should not be sourced. rclpy seems to be the exception given it's complex setup. For example if you try generating documentation for other packages like launch_ros with ROS 2 sourced and it should work fine

fujitatomoya commented 3 months ago

i will go ahead to close this in favor of https://github.com/ros-infrastructure/rosdoc2/pull/67 and https://github.com/ros-infrastructure/rosdoc2/issues/68 as tracking issue.

ros-infrastructure / rosdoc2