ros2 topic hz perform worse than c++ code, also worse than result in another docker container

Bug report

Required Info:

Operating System:
- Ubuntu18.04
Installation type:
- source
Version or commit hash:
- humble
DDS implementation:
- FastDDS

Steps to reproduce issue

run two docker container using the following docker-compose.yml

container1:
image: image_name
network_mode: "host"
cpuset: 2-9
cpu_shares: 2048
shm_size: 4G
container_name: container1
runtime: nvidia
privileged: true
environment:
  - NVIDIA_VISIBLE_DEVICES=all
logging:
  options:
    max-size: "${DOCKER_LOGS_MAX_SIZE:-100M}"

container2:
image: image_name
network_mode: "host"
privileged: true
cpuset: 12-17
cpu_shares: 2048
shm_size: 2G
container_name: container2
environment:
  - NVIDIA_VISIBLE_DEVICES=all
runtime: nvidia
logging:
  options:
    max-size: "${DOCKER_LOGS_MAX_SIZE:-100M}"

publish sensor_msgs::msg::PointCloud2 type message with frequency 10hz in container1
run subscriber and process data c++ code in container2, record the frequency subscription callback called
use ros2 topic hz tool follow the frequency of topic in both containers
Expected behavior
In both containers, ros2 topic hz shows the same frequency which is around 10hz
Log of c++ code in container2 shows that pointcloud2 messages arrive each 100ms
Actual behavior
In container1 terminal, ros2 topic hz shows frequency slightly under 10hz
In container2 terminal, ros2 topic hz shows frequency highly under 10hz, verity from 7hz to 9hz
In log of container2, c++ code receive message each 100ms stably

Additional information

Since code in both container are quite complicate, I can't paste the code here. As discussed in the blow question, ros2 topic hz result is lower than c++ code because the qos used is different. But why the hz results in different containers turn out to be different?

https://answers.ros.org/question/350753/ros2-topic-hz-provides-wrong-rate-for-larger-msgs/ Also, since ros2 topic hz result is not reliable, is therea a better way to track the transportaion stability between many nodes?

Ubuntu 18.04

Humble supports Ubuntu 22.04 as tier 1 platform. I am not sure if Humble can built with Ubuntu 18.04. (btw, Ubuntu 20.04 is community support.) see more details for https://docs.ros.org/en/humble/Releases/Release-Humble-Hawksbill.html#supported-platforms.

In container1 terminal, ros2 topic hz shows frequency slightly under 10hz

i believe that in this case, shared memory transport (not zero copy via LoanedMessage) can be enabled as localhost communication. this could be one of the benefits for the performance.

In container2 terminal, ros2 topic hz shows frequency highly under 10hz, verity from 7hz to 9hz

inter-process communication via network interface (in the case of docker, via bridge) would be one of the reason for this.

In log of container2, c++ code receive message each 100ms stably

there could be several differences, like language advantage for performance, rclpy pybind performance overhead.

above is my assumption, i did not test and analyze the performance difference. hopefully this helps.

ros2 / ros2cli