Proxy calls popen() in hot path

intel-staging / libptpmgmt_iaclocklib

The Clock Manager is a library for monitoring network time synchronization on the local platform. Disclaimer: This project is under development. All source code and features on the main branch is for the purpose of testing or evaluation and not production ready. We will upstream the code and archive this GitHub repo thereafter.

Other

4 stars 5 forks source link

Proxy calls popen() in hot path #92

Open rupran opened 2 weeks ago

rupran commented 2 weeks ago

Commit 268de83c2e86 ("jclklib: Fix proxy not able to reconnect to ptp4l") implemented a reconnect possibility for the proxy in case ptp4l was stopped and restarted. However, this introduces a call to popen in the main event loop inside ptp4l_event_loop .

This can lead to unpredictable delays and a lot of overhead[1] in the proxy, and should be handled much more efficiently.

[1]: On my system, the jclklib_proxy process uses ~20 percent with a logSyncInterval of 0 (1 second interval between messages) in the ptp4l configuration file and ~65 percent with a logSyncInterval of -5 (31.25ms message interval) of a core on the current main branch (that is, with the popen call in place).

Replacing the contents of is_ptp4l_running with return true; reduces this load to a negligible number.

yoongsiang2 commented 1 week ago

@rupran Thank you for your feedback. We appreciate your comments and will definitely take them into consideration. We will work on finding a more efficient way to check liveness of ptp4l. Will share to you the solution asap.

JunAnnLaiIntel commented 4 days ago

Hi @rupran, thanks for reporting the issue. I am trying to reproduce it, however I get around 2% cpu usage, as shown in snapshot below. This value is still around the same even the contents of is_ptp4l_running is replacing with return true. I am using ADL-N board with Ubuntu 22.04.

Do you mind to share your environment and the way you check the cpu usage?

rupran commented 18 hours ago

Our environment is based on Debian 12 with PREEMPT_RT and running on an Intel Xeon W-1115MLE CPU.

While the values under CPU% in top and htop show a low number, pinning the proxy to core 3 by modifying the run_proxy.sh script like the following:

--- a/jclklib/proxy/run_proxy.sh
+++ b/jclklib/proxy/run_proxy.sh
@@ -11,4 +11,4 @@
 SCRIPT_PATH="$(dirname "$0")"
 TEST_PATH="${SCRIPT_PATH}/../.."

-LD_LIBRARY_PATH=$TEST_PATH $SCRIPT_PATH/jclklib_proxy
+LD_LIBRARY_PATH=$TEST_PATH taskset 0x8 $SCRIPT_PATH/jclklib_proxy

shows that the core is running at higher load:

Running perf top -C 3 shows that the time is spent in libc and the kernel, most likely in preparation for spawning the new process (further down the stack, ld-linux-x86-64.so also takes up cycles in symbol resolution during loading).

yoongsiang2 commented 13 hours ago

Our environment is based on Debian 12 with PREEMPT_RT and running on an Intel Xeon W-1115MLE CPU.

While the values under CPU% in top and htop show a low number, pinning the proxy to core 3 by modifying the run_proxy.sh script like the following:
--- a/jclklib/proxy/run_proxy.sh
+++ b/jclklib/proxy/run_proxy.sh
@@ -11,4 +11,4 @@
 SCRIPT_PATH="$(dirname "$0")"
 TEST_PATH="${SCRIPT_PATH}/../.."

-LD_LIBRARY_PATH=$TEST_PATH $SCRIPT_PATH/jclklib_proxy
+LD_LIBRARY_PATH=$TEST_PATH taskset 0x8 $SCRIPT_PATH/jclklib_proxy
shows that the core is running at higher load:

Running perf top -C 3 shows that the time is spent in libc and the kernel, most likely in preparation for spawning the new process (further down the stack, ld-linux-x86-64.so also takes up cycles in symbol resolution during loading).

thank for the detail. We will try on our machine.