Recently warm-reboot with control plane assistant setup started failing on orchagent restart check:
Nov 2 14:38:07.123764 r-tigris-25 NOTICE swss#orchagent_restart_check: :- main: RESTARTCHECK failed, orchagent is not ready for warm restart with status NOT_READY
Nov 2 14:38:07.123764 r-tigris-25 NOTICE swss#orchagent_restart_check: :- main: requested orchagent to do warm restart state check, retry count: 5
Nov 2 14:38:07.123837 r-tigris-25 NOTICE swss#orchagent: :- doTask: RESTARTCHECK notification for orchagent
Nov 2 14:38:07.123837 r-tigris-25 NOTICE swss#orchagent: :- doTask: orchagent|NoFreeze:false|SkipPendingTaskCheck:false
Nov 2 14:38:07.123879 r-tigris-25 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Nov 2 14:38:07.123879 r-tigris-25 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Nov 2 14:38:07.123912 r-tigris-25 NOTICE swss#orchagent: :- warmRestartCheck: WarmRestart check found pending tasks:
Nov 2 14:38:07.123924 r-tigris-25 NOTICE swss#orchagent: :- warmRestartCheck: VXLAN_REMOTE_VNI_TABLE:Vlan1000:192.168.8.1|SET|vni:1000
Nov 2 14:38:07.123924 r-tigris-25 NOTICE swss#orchagent: :- warmRestartCheck: Restart check result: NOT_READY
Nov 2 14:38:07.123996 r-tigris-25 NOTICE swss#orchagent_restart_check: :- main: RESTARTCHECK failed, orchagent is not ready for warm restart with status NOT_READY
Nov 2 14:38:07.130843 r-tigris-25 NOTICE admin: fastfast-reboot failure (10) cleanup ...
Nov 2 14:38:07.136823 r-tigris-25 NOTICE admin: Tearing down control plane assistant: 10.213.84.41 ...
The issue started to happen after SAI implementation update. In our latest SAI P2P tunnels are now supported (SAI_TUNNEL_PEER_MODE_P2P). Therefore the logic inside orchdaemon chooses a p2p implementation of evpn remote vni orch:
Description
Recently warm-reboot with control plane assistant setup started failing on orchagent restart check:
The issue started to happen after SAI implementation update. In our latest SAI P2P tunnels are now supported (
SAI_TUNNEL_PEER_MODE_P2P
). Therefore the logic inside orchdaemon chooses a p2p implementation of evpn remote vni orch:https://github.com/sonic-net/sonic-swss/blob/master/orchagent/orchdaemon.cpp#L432:
And thus, this error starts to happen only with
EvpnRemoteVnip2pOrch
.With old SAI and with
EvpnRemoteVnip2mpOrch
, this error does not appear, but a warning is printed instead:Steps to reproduce the issue:
SAI_TUNNEL_PEER_MODE_P2P
)sudo warm-reboot -c X.X.X.X
.Describe the results you received:
Warm-reboot failure on orchagent restart check.
Describe the results you expected:
Warm-reboot passes.
Output of
show version
:Output of
show techsupport
:sonic_dump_r-tigris-25_20221102_154508.tar.gz
Additional information you deem important (e.g. issue happens only occasionally):