Long-term testing [14400]

[pnn16004]

Hi,

I tried using Fast DDS Monitor to monitor ROS2 for 24 hours, but when I came back after 13 hours the application had frozen and all the graphs were empty. I don't think it's a problem with the PC as all the other windows were fine.

Has anyone tested Fast DDS Monitor for longer periods and know if it works or not? Are there any measures I can do to make it more stable, like disabling some of the visual parts?

I saw on issue #147 about using Fast DDS Statistics Backend, but I'd rather use Fast DDS Monitor if possible since it's much simpler and doesn't need much setup.

[jparisu]

Hi @pnn16004

Indeed, this is a known limitation of Fast DDS Monitor.

First: a problem here is that the tool Fast DDS Statistics Backend, which is the actual backend of the graphical application Fast DDS Monitor, stores in RAM all the data that has been received. Thus, at some point the amount of data monitored is too big to be stored.

Second: using dynamic charts in Fast DDS Monitor highly consumes CPU and memory as it asks periodically for new data to the backend, and it stores every new point separately from the backend.

We know these limitations, and have some ideas on how to solve them. But so far, there is not any road map milestone, as the Fast DDS Monitor is not thought to be used in long executions. It is thought to be used in specific time points of the execution, where a human can easily check and understand some issues in the network.

Possible solutions

I suggest you to try these solutions in order to achieve your scenario to run:

• Use only those statistics topics that you want to monitor. Using the env var, you can set which statistics you want to send. Limiting these, your program will store less data and could run further.
• Do not use dynamic charts for long executions. You could see all the old data stored by historical charts.
• If you want to store the data so you can process it afterwards, use the Fast DDS Statistics Backend instead of the Monitor. You can also build a custom application over the Backend API so it manages the data as you want, and you could even store this data in memory and clean the backend from time to time.

[pnn16004]

I will try these two steps and see if it can run for longer, otherwise I will use the Fast DDS Statistics Backend if I have time.

Thank you!

[pnn16004]

When I use the historical chart it doesn't look the same at all as the dynamical one, and all the statistics have the same value. Here is how it looks:

Do you know why this is? Do I need to do any extra steps in the setup to use historical charts?

[jparisu]

Hi @pnn16004

For the image you sent, I don't think you are missing any configuration. And no, you do not need extra setup to use historical charts.

It is only required to have in mind that configuring a historical chart requires different arguments than a dynamic one. Most important one is setting correctly the Number of bins. This value set the number of points that will appear in the chart. Using 0 is not recommended, as it shows the raw data, that could be difficult to read from a user point of view. Having this in mind, just try to play with different values of Number of bins and with the view buttons in the top right of the chart. For more information, please refer to the documentation: https://fast-dds-monitor.readthedocs.io/en/latest/rst/user_manual/chart_panel/historic_series.html

NOTE: You could use dynamic data without problem, just do not leave it running for a long time. Removing a dynamic chart will leave no trace of data or memory, and will not affect the future performance.

[pnn16004]

Hi,

Setting the number of bins solved it! I'm guessing the more points the more accurate it is. I put 50 and it looks pretty good, while 100 seems more accurate but is harder to understand.

I tried leaving Fast DDS Monitor open without any charts open to store historical data, but it seems to crash after a while anyways. I assume it's overloading on old data.

Fast DDS Statistics Backend is probably the solution to all my problems.

Thank you for your help!

[jparisu]

Hi @pnn16004 ,

Sorry to hear that. I hope Fast DDS Statistics Backend fulfilled your requirements. However, do not forget that it is the Backend the one that stores the data in memory while executing. Its strength is that it retrieves methods to dump all the data in disk and to remove the data and start over.