Open stepanblyschak opened 1 year ago
@prsunny need improvement in fast-reboot
To clarify, this is a degradation comparing to older images, for example (time is different as taken from another platform):
202012:
Mar 9 15:57:40.792921 sonic INFO systemd[1]: Started switch state service.
Mar 9 15:57:48.824060 sonic NOTICE swss#orchagent: :- main: --- Starting Orchestration Agent ---
8 sec
202211:
Mar 9 16:07:14.097283 sonic INFO systemd[1]: Started switch state service.
Mar 9 16:07:26.429175 r-bulldog-02 NOTICE swss#orchagent: :- main: --- Starting Orchestration Agent ---
12 sec
@prsunny @vaibhavhd @yxieca I did some POC to measure improvements and I observed that while I am able to get orchagent start much quicker it does not have an affect on control plane restoration time. We must have both orchagent running and syncd running in order to start re-configuration, however, currently syncd is a limiting factor on a platform I am doing this test. Syncd starts even later than orchagent.
To conclude, I am holding on with orchagent improvements until I find an optimization path for syncd. Once we have syncd start sooner than orchagent we can proceed with orchagent optimizations.
Description
Orchagent daemon starts >20 sec after SWSS container started:
Steps to reproduce the issue:
Describe the results you received:
Systemd SWSS start script takes 2 sec:
Supervisor sleeps for 1 sec before spawning any processes:
Supervisor dependent-startup plugin starts processes:
orchagent is configured to depend on portsyncd which depends on rsyslogd. Since supervisord has startsecs=1 by default for every process, there will be at least 2 seconds delay. Actually, rsyslogd is not guaranteed to be available in 1 sec, so this synchronization is weak. However, we can observe 3+3 sec delay before starting orchagent which imply possible supervisord_dependent_startup overhead. Rest of the time - around 10 sec we spend in /usr/bin/orchagent.sh before orchagent is started. Also, the delay is not consistent ranging from 15-22 sec in testing. This creates a lot of noise when trying to analyze other possibilities for optimizations.
Describe the results you expected:
SWSS orchagent is critical daemon. It is necessary to start it as soon as possible at boot to avoid any delays. Expected orchagent start imidiatelly after container is started.
Output of
show version
:Output of
show techsupport
:Additional information you deem important (e.g. issue happens only occasionally):
sysdump_test_push_gate_reboot_policer.tar.gz