search

sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
735 stars 1.41k forks source link

Warmboot fails with OA not ready for warm restart - pending SET oper on VXLAN table #12361

Open vaibhavhd opened 2 years ago

vaibhavhd commented 2 years ago

Description

When warmboot is issue with CPA, RESTARTCHECK fails with OA continuously being busy on a pending task VXLAN_REMOTE_VNI_TABLE:Vlan1000:192.168.8.1|SET|vni:1000

This issue has surfaced after a recent PR to fix another issue with interface creation during CPA: https://github.com/sonic-net/sonic-utilities/pull/2398

Oct  7 18:43:52.863105 str-7260cx3-acs-1 ERR swss#orchagent: :- handleSaiGetStatus: Encountered failure in get operation, SAI API: SAI_API_MIRROR, status: SAI_STATUS_INVALID_PARAMETER
Oct  7 18:43:52.892196 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:52.892219 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1

Oct  7 18:43:55.045293 str-7260cx3-acs-1 NOTICE swss#orchagent: :- doTask: RESTARTCHECK notification for orchagent 
Oct  7 18:43:55.045427 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck: WarmRestart check found pending tasks: 
Oct  7 18:43:55.045441 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck:     VXLAN_REMOTE_VNI_TABLE:Vlan1000:192.168.8.1|SET|vni:1000
Oct  7 18:43:55.045566 str-7260cx3-acs-1 NOTICE swss#orchagent_restart_check: :- main: RESTARTCHECK failed, orchagent is not ready for warm restart with status NOT_READY

Steps to reproduce the issue:

  1. Install latest master or 202205 image on a DUT
  2. Issue warmboot with -c option (with CPA)
  3. warmboot will fail with RESTARTCHECK error

Describe the results you received:

Detailed logs:

Oct  7 18:43:46.150801 str-7260cx3-acs-1 NOTICE admin: Saving counters folder before warmboot...
Oct  7 18:43:46.904575 str-7260cx3-acs-1 INFO syncd#syncd: [none] SAI_API_FDB:_brcm_sai_fdb_event_cb:212 fdbEvent: add (1) for mac 72-06-00-01-05-85 vid:0x3e8, port:0x6b lagid:0x0 flags:0x10440 flags2:0x0 lag:false station flags 0x0
Oct  7 18:43:46.904575 str-7260cx3-acs-1 INFO syncd#syncd: [none] SAI_API_NEIGHBOR:brcm_sai_xgs_common_neighbor_mac_db_update:104 Mac entry not found - skip nbr processing. add 1, dir 4
Oct  7 18:43:47.055934 str-7260cx3-acs-1 NOTICE swss#orchagent: :- addNeighbor: Created neighbor ip 192.168.1.249, 72:06:00:01:05:03 on Vlan1000
Oct  7 18:43:47.057228 str-7260cx3-acs-1 NOTICE swss#orchagent: :- addNextHop: Created next hop 192.168.1.249 on Vlan1000
Oct  7 18:43:47.143650 str-7260cx3-acs-1 NOTICE swss#orchagent: :- addNeighbor: Created neighbor ip 192.168.2.75, 72:06:00:01:05:85 on Vlan1000
Oct  7 18:43:47.144652 str-7260cx3-acs-1 NOTICE swss#orchagent: :- addNextHop: Created next hop 192.168.2.75 on Vlan1000
Oct  7 18:43:47.172837 str-7260cx3-acs-1 ERR systemd[1]: Failed to start Telemetry container.
Oct  7 18:43:47.173026 str-7260cx3-acs-1 ERR systemd[1]: Failed to start Telemetry container.
Oct  7 18:43:47.269332 str-7260cx3-acs-1 ERR systemd[1]: Failed to start Telemetry container.
Oct  7 18:43:47.363618 str-7260cx3-acs-1 ERR systemd[1]: Failed to start Telemetry container.
Oct  7 18:43:47.463093 str-7260cx3-acs-1 ERR systemd[1]: Failed to start Telemetry container.
Oct  7 18:43:47.545223 str-7260cx3-acs-1 ERR systemd[1]: Failed to start Telemetry container.
Oct  7 18:43:47.638356 str-7260cx3-acs-1 ERR systemd[1]: Failed to start Telemetry container.
Oct  7 18:43:47.739209 str-7260cx3-acs-1 ERR systemd[1]: Failed to start Telemetry container.
Oct  7 18:43:49.417956 str-7260cx3-acs-1 NOTICE swss#orchagent: :- addNeighbor: Created neighbor ip 192.168.2.91, 72:06:00:01:06:01 on Vlan1000
Oct  7 18:43:49.419900 str-7260cx3-acs-1 NOTICE swss#orchagent: :- addNextHop: Created next hop 192.168.2.91 on Vlan1000
Oct  7 18:43:49.590846 str-7260cx3-acs-1 DEBUG check_db_integrity.py: Database integrity checks passed.
Oct  7 18:43:50.050771 str-7260cx3-acs-1 INFO admin: Checking that ASIC configuration has not changed
Oct  7 18:43:50.775064 str-7260cx3-acs-1 INFO admin: ASIC config unchanged, current and destination SONiC version are the same
Oct  7 18:43:51.606226 str-7260cx3-acs-1 NOTICE admin: Setting up control plane assistant: 10.64.247.17 ...
Oct  7 18:43:51.742819 str-7260cx3-acs-1 NOTICE swss#orchagent: :- addNeighbor: Created neighbor ip 192.168.2.13, 72:06:00:01:05:23 on Vlan1000
Oct  7 18:43:51.747218 str-7260cx3-acs-1 NOTICE swss#orchagent: :- addNextHop: Created next hop 192.168.2.13 on Vlan1000
Oct  7 18:43:51.853751 str-7260cx3-acs-1 NOTICE swss#vxlanmgrd: :- doVxlanTunnelCreateTask: Create vxlan tunnel neigh_adv
Oct  7 18:43:51.854593 str-7260cx3-acs-1 NOTICE swss#orchagent: :- addOperation: Vxlan tunnel 'neigh_adv' was added
Oct  7 18:43:51.871876 str-7260cx3-acs-1 INFO systemd-udevd[12173]: Using default interface naming scheme 'v247'.
Oct  7 18:43:51.872008 str-7260cx3-acs-1 INFO systemd-udevd[12173]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Oct  7 18:43:51.874828 str-7260cx3-acs-1 INFO kernel: [  524.277440] Bridge: port 114(neigh_adv-1000) entered blocking state
Oct  7 18:43:51.874853 str-7260cx3-acs-1 INFO kernel: [  524.277443] Bridge: port 114(neigh_adv-1000) entered disabled state
Oct  7 18:43:51.874855 str-7260cx3-acs-1 INFO kernel: [  524.278070] device neigh_adv-1000 entered promiscuous mode
Oct  7 18:43:51.893539 str-7260cx3-acs-1 INFO kernel: [  524.295032] Bridge: port 114(neigh_adv-1000) entered blocking state
Oct  7 18:43:51.893552 str-7260cx3-acs-1 INFO kernel: [  524.295036] Bridge: port 114(neigh_adv-1000) entered forwarding state
Oct  7 18:43:51.893852 str-7260cx3-acs-1 ERR systemd[1]: Failed to start Telemetry container.
Oct  7 18:43:51.895978 str-7260cx3-acs-1 WARNING swss#orchagent: :- createTunnelHw: creation src = 0
Oct  7 18:43:51.895978 str-7260cx3-acs-1 NOTICE swss#orchagent: :- create_tunnel: create_tunnel:encapmaplist[0]=0x290000000019c8
Oct  7 18:43:51.895978 str-7260cx3-acs-1 NOTICE swss#orchagent: :- create_tunnel: create_tunnel:encapmaplist[1]=0x290000000019ca
Oct  7 18:43:52.243588 str-7260cx3-acs-1 NOTICE syncd#syncd: :- threadFunction: time span 173 ms for 'create:SAI_OBJECT_TYPE_TUNNEL_MAP_ENTRY:oid:0x3b0000000019cd'
Oct  7 18:43:52.854634 str-7260cx3-acs-1 NOTICE swss#orchagent: :- addOperation: Vxlan tunnel map entry 'map_1' for tunnel 'neigh_adv' was created
Oct  7 18:43:52.863105 str-7260cx3-acs-1 ERR swss#orchagent: :- handleSaiGetStatus: Encountered failure in get operation, SAI API: SAI_API_MIRROR, status: SAI_STATUS_INVALID_PARAMETER
Oct  7 18:43:52.886935 str-7260cx3-acs-1 NOTICE swss#orchagent: :- attach: Attached next hop observer of route 192.168.8.0/25 for destination IP 192.168.8.1
Oct  7 18:43:52.887226 str-7260cx3-acs-1 NOTICE swss#orchagent: :- updateNextHop: Updating mirror session neighbor_advertiser with route 192.168.8.0/25
Oct  7 18:43:52.887226 str-7260cx3-acs-1 NOTICE swss#orchagent: :- updateNextHop:     next hop IPs: 10.0.0.33@PortChannel101,10.0.0.35@PortChannel102,10.0.0.37@PortChannel103,10.0.0.39@PortChannel104
Oct  7 18:43:52.887497 str-7260cx3-acs-1 NOTICE swss#orchagent: :- updateNextHop: Updated mirror session state db neighbor_advertiser nexthop to 10.0.0.33@PortChannel101
Oct  7 18:43:52.887497 str-7260cx3-acs-1 NOTICE swss#orchagent: :- getNeighborInfo: Mirror session neighbor_advertiser neighbor is PortChannel101
Oct  7 18:43:52.890866 str-7260cx3-acs-1 NOTICE swss#orchagent: :- activateSession: Activated mirror session neighbor_advertiser
Oct  7 18:43:52.891029 str-7260cx3-acs-1 NOTICE swss#orchagent: :- createEntry: Created mirror session neighbor_advertiser
Oct  7 18:43:52.892196 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:52.892219 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:52.892437 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:52.892437 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:52.892504 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:52.892572 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:52.892572 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:52.892755 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:52.892789 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:52.892819 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:53.158619 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:53.158619 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:53.468052 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:53.468079 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:54.158617 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:54.158617 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:54.468096 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:54.468096 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:54.519625 str-7260cx3-acs-1 INFO syncd#syncd: [none] SAI_API_FDB:_brcm_sai_fdb_event_cb:212 fdbEvent: add (1) for mac 72-06-00-01-05-08 vid:0x3e8, port:0x11 lagid:0x0 flags:0x10440 flags2:0x0 lag:false station flags 0x0
Oct  7 18:43:54.519833 str-7260cx3-acs-1 INFO syncd#syncd: [none] SAI_API_NEIGHBOR:brcm_sai_xgs_common_neighbor_mac_db_update:104 Mac entry not found - skip nbr processing. add 1, dir 4
Oct  7 18:43:54.519984 str-7260cx3-acs-1 INFO syncd#syncd: [none] SAI_API_FDB:_brcm_sai_fdb_event_cb:212 fdbEvent: add (1) for mac 72-06-00-01-06-09 vid:0x3e8, port:0x33 lagid:0x0 flags:0x10440 flags2:0x0 lag:false station flags 0x0
Oct  7 18:43:54.519984 str-7260cx3-acs-1 INFO syncd#syncd: [none] SAI_API_NEIGHBOR:brcm_sai_xgs_common_neighbor_mac_db_update:104 Mac entry not found - skip nbr processing. add 1, dir 4
Oct  7 18:43:54.520534 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:54.520841 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:54.520841 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:54.520841 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:54.520993 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:54.521096 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:54.521096 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:54.521120 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:54.521440 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:54.521457 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:54.521578 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:54.521651 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:54.521950 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:54.522006 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:54.522060 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:54.522060 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:54.558309 str-7260cx3-acs-1 NOTICE swss#orchagent: :- addNeighbor: Created neighbor ip 192.168.1.254, 72:06:00:01:05:08 on Vlan1000
Oct  7 18:43:54.563929 str-7260cx3-acs-1 NOTICE swss#orchagent: :- addNextHop: Created next hop 192.168.1.254 on Vlan1000
Oct  7 18:43:54.564205 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:54.564248 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:54.688020 str-7260cx3-acs-1 NOTICE swss#orchagent: :- addNeighbor: Created neighbor ip 192.168.2.99, 72:06:00:01:06:09 on Vlan1000
Oct  7 18:43:54.691448 str-7260cx3-acs-1 NOTICE swss#orchagent: :- addNextHop: Created next hop 192.168.2.99 on Vlan1000
Oct  7 18:43:54.691560 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:54.691637 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:54.895228 str-7260cx3-acs-1 NOTICE swss#orchagent: :- add: Successfully created ACL rule rule_arp in table EVERFLOW
Oct  7 18:43:54.895760 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:54.895760 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:54.904555 str-7260cx3-acs-1 NOTICE swss#orchagent: :- add: Successfully created ACL rule rule_nd in table EVERFLOW
Oct  7 18:43:54.904938 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:54.904938 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:54.917359 str-7260cx3-acs-1 NOTICE admin: Pausing orchagent ...
Oct  7 18:43:55.003186 str-7260cx3-acs-1 ERR systemd[1]: Failed to start Telemetry container.
Oct  7 18:43:55.044774 str-7260cx3-acs-1 NOTICE swss#orchagent_restart_check: :- main: Wait time for response from orchagent set to 2000 milliseconds
Oct  7 18:43:55.044774 str-7260cx3-acs-1 NOTICE swss#orchagent_restart_check: :- main: Number of retries for the request to orchagent is set to 5
Oct  7 18:43:55.045123 str-7260cx3-acs-1 INFO swss#orchagent_restart_check: :- subscribe: subscribed to RESTARTCHECKREPLY
Oct  7 18:43:55.045123 str-7260cx3-acs-1 NOTICE swss#orchagent_restart_check: :- main: requested orchagent to do warm restart state check, retry count: 0
Oct  7 18:43:55.045293 str-7260cx3-acs-1 NOTICE swss#orchagent: :- doTask: RESTARTCHECK notification for orchagent 
Oct  7 18:43:55.045293 str-7260cx3-acs-1 NOTICE swss#orchagent: :- doTask: orchagent|NoFreeze:false|SkipPendingTaskCheck:false
Oct  7 18:43:55.045390 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:55.045406 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:55.045427 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck: WarmRestart check found pending tasks: 
Oct  7 18:43:55.045441 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck:     VXLAN_REMOTE_VNI_TABLE:Vlan1000:192.168.8.1|SET|vni:1000
Oct  7 18:43:55.045448 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck: Restart check result: NOT_READY
Oct  7 18:43:55.045566 str-7260cx3-acs-1 NOTICE swss#orchagent_restart_check: :- main: RESTARTCHECK failed, orchagent is not ready for warm restart with status NOT_READY
Oct  7 18:43:55.045582 str-7260cx3-acs-1 NOTICE swss#orchagent_restart_check: :- main: requested orchagent to do warm restart state check, retry count: 1
Oct  7 18:43:55.045661 str-7260cx3-acs-1 NOTICE swss#orchagent: :- doTask: RESTARTCHECK notification for orchagent 
Oct  7 18:43:55.045673 str-7260cx3-acs-1 NOTICE swss#orchagent: :- doTask: orchagent|NoFreeze:false|SkipPendingTaskCheck:false
Oct  7 18:43:55.045722 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:55.045734 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:55.045818 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck: WarmRestart check found pending tasks: 
Oct  7 18:43:55.045850 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck:     VXLAN_REMOTE_VNI_TABLE:Vlan1000:192.168.8.1|SET|vni:1000
Oct  7 18:43:55.045878 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck: Restart check result: NOT_READY
Oct  7 18:43:55.045878 str-7260cx3-acs-1 NOTICE swss#orchagent_restart_check: :- main: RESTARTCHECK failed, orchagent is not ready for warm restart with status NOT_READY
Oct  7 18:43:55.045900 str-7260cx3-acs-1 NOTICE swss#orchagent_restart_check: :- main: requested orchagent to do warm restart state check, retry count: 2
Oct  7 18:43:55.046002 str-7260cx3-acs-1 NOTICE swss#orchagent: :- doTask: RESTARTCHECK notification for orchagent 
Oct  7 18:43:55.046002 str-7260cx3-acs-1 NOTICE swss#orchagent: :- doTask: orchagent|NoFreeze:false|SkipPendingTaskCheck:false
Oct  7 18:43:55.046141 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:55.046191 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:55.046191 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck: WarmRestart check found pending tasks: 
Oct  7 18:43:55.046191 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck:     VXLAN_REMOTE_VNI_TABLE:Vlan1000:192.168.8.1|SET|vni:1000
Oct  7 18:43:55.046191 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck: Restart check result: NOT_READY
Oct  7 18:43:55.046211 str-7260cx3-acs-1 NOTICE swss#orchagent_restart_check: :- main: RESTARTCHECK failed, orchagent is not ready for warm restart with status NOT_READY
Oct  7 18:43:55.046211 str-7260cx3-acs-1 NOTICE swss#orchagent_restart_check: :- main: requested orchagent to do warm restart state check, retry count: 3
Oct  7 18:43:55.046348 str-7260cx3-acs-1 NOTICE swss#orchagent: :- doTask: RESTARTCHECK notification for orchagent 
Oct  7 18:43:55.046348 str-7260cx3-acs-1 NOTICE swss#orchagent: :- doTask: orchagent|NoFreeze:false|SkipPendingTaskCheck:false
Oct  7 18:43:55.046442 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:55.046442 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:55.046480 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck: WarmRestart check found pending tasks: 
Oct  7 18:43:55.046480 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck:     VXLAN_REMOTE_VNI_TABLE:Vlan1000:192.168.8.1|SET|vni:1000
Oct  7 18:43:55.046560 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck: Restart check result: NOT_READY
Oct  7 18:43:55.046578 str-7260cx3-acs-1 NOTICE swss#orchagent_restart_check: :- main: RESTARTCHECK failed, orchagent is not ready for warm restart with status NOT_READY
Oct  7 18:43:55.046593 str-7260cx3-acs-1 NOTICE swss#orchagent_restart_check: :- main: requested orchagent to do warm restart state check, retry count: 4
Oct  7 18:43:55.046684 str-7260cx3-acs-1 NOTICE swss#orchagent: :- doTask: RESTARTCHECK notification for orchagent 
Oct  7 18:43:55.046684 str-7260cx3-acs-1 NOTICE swss#orchagent: :- doTask: orchagent|NoFreeze:false|SkipPendingTaskCheck:false
Oct  7 18:43:55.046736 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:55.046736 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:55.046809 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck: WarmRestart check found pending tasks: 
Oct  7 18:43:55.046826 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck:     VXLAN_REMOTE_VNI_TABLE:Vlan1000:192.168.8.1|SET|vni:1000
Oct  7 18:43:55.046826 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck: Restart check result: NOT_READY
Oct  7 18:43:55.046956 str-7260cx3-acs-1 NOTICE swss#orchagent_restart_check: :- main: RESTARTCHECK failed, orchagent is not ready for warm restart with status NOT_READY
Oct  7 18:43:55.046956 str-7260cx3-acs-1 NOTICE swss#orchagent_restart_check: :- main: requested orchagent to do warm restart state check, retry count: 5
Oct  7 18:43:55.046985 str-7260cx3-acs-1 NOTICE swss#orchagent: :- doTask: RESTARTCHECK notification for orchagent 
Oct  7 18:43:55.047000 str-7260cx3-acs-1 NOTICE swss#orchagent: :- doTask: orchagent|NoFreeze:false|SkipPendingTaskCheck:false
Oct  7 18:43:55.047041 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:55.047053 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:55.047074 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck: WarmRestart check found pending tasks: 
Oct  7 18:43:55.047090 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck:     VXLAN_REMOTE_VNI_TABLE:Vlan1000:192.168.8.1|SET|vni:1000
Oct  7 18:43:55.047090 str-7260cx3-acs-1 NOTICE swss#orchagent: :- warmRestartCheck: Restart check result: NOT_READY
Oct  7 18:43:55.047287 str-7260cx3-acs-1 NOTICE swss#orchagent_restart_check: :- main: RESTARTCHECK failed, orchagent is not ready for warm restart with status NOT_READY
Oct  7 18:43:55.053655 str-7260cx3-acs-1 NOTICE admin: warm-reboot failure (10) cleanup ...
Oct  7 18:43:55.058739 str-7260cx3-acs-1 NOTICE admin: Tearing down control plane assistant: 10.64.247.17 ...
Oct  7 18:43:55.158736 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:55.158736 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:55.210556 str-7260cx3-acs-1 NOTICE swss#orchagent: :- remove: Successfully deleted ACL rule rule_arp in table EVERFLOW
Oct  7 18:43:55.210556 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:55.210556 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:55.213256 str-7260cx3-acs-1 NOTICE swss#orchagent: :- remove: Successfully deleted ACL rule rule_nd in table EVERFLOW
Oct  7 18:43:55.213281 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:55.213288 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:55.214870 str-7260cx3-acs-1 WARNING kernel: [  527.614422] linux-bcm-knet (2731): Unsupported command (type=1, opcode=52)
Oct  7 18:43:55.468034 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:55.468034 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:56.158839 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:56.158914 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:56.468302 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:56.468302 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:57.095922 str-7260cx3-acs-1 ALERT dhcp_relay#dhcpmon[47]: dhcpmon detected disparity in DHCP Relay behavior. Duration: 378 (sec) for vlan: 'Agg-Vlan1000'
Oct  7 18:43:57.095922 str-7260cx3-acs-1 NOTICE dhcp_relay#dhcpmon[47]: [    Agg-Vlan1000-Snapshot rx/tx] Discover:       998/        0, Offer:         0/        0, Request:         0/        0, ACK:         0/        0
Oct  7 18:43:57.095922 str-7260cx3-acs-1 NOTICE dhcp_relay#dhcpmon[47]: [    Agg-Vlan1000- Current rx/tx] Discover:      1053/        0, Offer:         0/        0, Request:         0/        0, ACK:         0/        0
Oct  7 18:43:57.158941 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:57.158941 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:57.468201 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:57.468201 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:58.158844 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:58.158899 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:58.226899 str-7260cx3-acs-1 INFO kernel: [  530.624862] Bridge: port 114(neigh_adv-1000) entered disabled state
Oct  7 18:43:58.230023 str-7260cx3-acs-1 NOTICE swss#orchagent: :- deactivateSession: Deactivated mirror session neighbor_advertiser
Oct  7 18:43:58.230233 str-7260cx3-acs-1 NOTICE swss#orchagent: :- detach: Detached next hop observer for destination IP 192.168.8.1
Oct  7 18:43:58.230539 str-7260cx3-acs-1 NOTICE swss#orchagent: :- deleteEntry: Removed mirror session neighbor_advertiser
Oct  7 18:43:58.230832 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:58.230986 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:58.234817 str-7260cx3-acs-1 INFO kernel: [  530.634999] device neigh_adv-1000 left promiscuous mode
Oct  7 18:43:58.234836 str-7260cx3-acs-1 INFO kernel: [  530.635021] Bridge: port 114(neigh_adv-1000) entered disabled state
Oct  7 18:43:58.244055 str-7260cx3-acs-1 NOTICE syncd#syncd: :- threadFunction: time span 11 ms for 'remove:SAI_OBJECT_TYPE_TUNNEL_MAP_ENTRY:oid:0x3b0000000019cd'
Oct  7 18:43:58.251226 str-7260cx3-acs-1 ERR systemd[1]: Failed to start Telemetry container.
Oct  7 18:43:58.287772 str-7260cx3-acs-1 NOTICE swss#vxlanmgrd: :- doVxlanTunnelDeleteTask: Delete vxlan tunnel neigh_adv
Oct  7 18:43:58.394301 str-7260cx3-acs-1 NOTICE swss#orchagent: :- delOperation: vni count = 0
Oct  7 18:43:58.398438 str-7260cx3-acs-1 NOTICE swss#orchagent: :- delOperation: Vxlan tunnel map entry 'map_1' for tunnel 'neigh_adv' was removed
Oct  7 18:43:58.398554 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:58.398554 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:58.399020 str-7260cx3-acs-1 NOTICE swss#orchagent: :- delOperation: Vxlan tunnel 'neigh_adv' was removed
Oct  7 18:43:58.399124 str-7260cx3-acs-1 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Oct  7 18:43:58.399124 str-7260cx3-acs-1 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Oct  7 18:43:58.399583 str-7260cx3-acs-1 WARNING swss#orchagent: :- delOperation: RemoteVniDel getTunnelPort Fails: 192.168.8.1
Oct  7 18:43:58.759667 str-7260cx3-acs-1 NOTICE admin: Cancel warm-reboot: code (0)

Describe the results you expected:

CPA creation should succeed and warmboot should proceed.

Output of show version:

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

arlakshm commented 2 years ago

The EVPN-VXLAN feature seems to be interfering with CPA feature. This is causing a warmboot problem with CPA @yxieca to start thread with @adyeung on this issue.

adyeung commented 1 year ago

@skbhava from BRCM will followup

skbhava commented 1 year ago

@vaibhavhd can you please share techsupport for this issue to debug further the reason for tunnel mapping failure at OA

vaibhavhd commented 1 year ago

@vaibhavhd can you please share techsupport for this issue to debug further the reason for tunnel mapping failure at OA

This is tech support file: sonic_dump_str2-7050cx3-acs-02_20221208_184456.tar.gz

Issue snippet from the logs to show you where the issue happened:

Dec  8 18:25:22.412756 str2-7050cx3-acs-02 NOTICE admin: Setting up control plane assistant: 10.64.246.125 ...

Dec  8 18:25:22.768842 str2-7050cx3-acs-02 NOTICE swss#vxlanmgrd: :- doVxlanTunnelCreateTask: Create vxlan tunnel neigh_adv
Dec  8 18:25:22.769421 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- addOperation: Vxlan tunnel 'neigh_adv' was added
Dec  8 18:25:22.777758 str2-7050cx3-acs-02 WARNING swss#orchagent: :- createTunnelHw: creation src = 0
Dec  8 18:25:22.778243 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- create_tunnel: create_tunnel:encapmaplist[0]=0x29000000000710
Dec  8 18:25:22.778243 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- create_tunnel: create_tunnel:encapmaplist[1]=0x29000000000712
Dec  8 18:25:22.779032 str2-7050cx3-acs-02 INFO syncd#syncd: [none] SAI_API_TUNNEL:brcm_sai_tnl_mp_create_tunnel:3485 Setting peer_mode to 1
Dec  8 18:25:22.786426 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- addOperation: Vxlan tunnel map entry 'map_1' for tunnel 'neigh_adv' was created

Dec  8 18:25:22.795888 str2-7050cx3-acs-02 ERR swss#orchagent: :- handleSaiGetStatus: Encountered failure in get operation, SAI API: SAI_API_MIRROR, status: SAI_STATUS_INVALID_PARAMETER

Dec  8 18:25:22.804808 str2-7050cx3-acs-02 INFO systemd-udevd[25462]: Using default interface naming scheme 'v247'.
Dec  8 18:25:22.805009 str2-7050cx3-acs-02 INFO systemd-udevd[25462]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.

Dec  8 18:25:22.809420 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- attach: Attached next hop observer of route 192.168.8.0/25 for destination IP 192.168.8.1

Dec  8 18:25:22.811321 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- updateNextHop: Updating mirror session neighbor_advertiser with route 192.168.8.0/25
Dec  8 18:25:22.812640 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- updateNextHop:     next hop IPs: 10.0.0.57@PortChannel101,10.0.0.59@PortChannel102,10.0.0.61@PortChannel103,10.0.0.63@PortChannel104
Dec  8 18:25:22.814270 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- updateNextHop: Updated mirror session state db neighbor_advertiser nexthop to 10.0.0.57@PortChannel101

Dec  8 18:25:22.814411 str2-7050cx3-acs-02 INFO kernel: [ 1768.652470] Bridge: port 26(neigh_adv-1000) entered blocking state
Dec  8 18:25:22.814423 str2-7050cx3-acs-02 INFO kernel: [ 1768.652478] Bridge: port 26(neigh_adv-1000) entered disabled state

Dec  8 18:25:22.816209 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- getNeighborInfo: Mirror session neighbor_advertiser neighbor is PortChannel101
Dec  8 18:25:22.818334 str2-7050cx3-acs-02 INFO kernel: [ 1768.656777] device neigh_adv-1000 entered promiscuous mode

Dec  8 18:25:22.831113 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- activateSession: Activated mirror session neighbor_advertiser
Dec  8 18:25:22.832194 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- createEntry: Created mirror session neighbor_advertiser

Dec  8 18:25:22.835432 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Dec  8 18:25:22.835687 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Dec  8 18:25:22.837280 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Dec  8 18:25:22.837280 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1

Dec  8 18:25:22.846406 str2-7050cx3-acs-02 INFO kernel: [ 1768.685277] Bridge: port 26(neigh_adv-1000) entered blocking state
Dec  8 18:25:22.846442 str2-7050cx3-acs-02 INFO kernel: [ 1768.685285] Bridge: port 26(neigh_adv-1000) entered forwarding state

Dec  8 18:25:22.968908 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Dec  8 18:25:22.968908 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Dec  8 18:25:22.968908 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Dec  8 18:25:22.968960 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Dec  8 18:25:22.971045 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Dec  8 18:25:22.971045 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Dec  8 18:25:23.182183 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Dec  8 18:25:23.182183 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1

Dec  8 18:25:24.006192 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Dec  8 18:25:24.006192 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Dec  8 18:25:24.006324 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Dec  8 18:25:24.006428 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Dec  8 18:25:24.182317 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Dec  8 18:25:24.182317 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1

Dec  8 18:25:25.068175 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Dec  8 18:25:25.068175 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Dec  8 18:25:25.068225 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Dec  8 18:25:25.068285 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Dec  8 18:25:25.182263 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Dec  8 18:25:25.182263 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Dec  8 18:25:25.796644 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- add: Successfully created ACL rule rule_arp in table EVERFLOW
Dec  8 18:25:25.798483 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Dec  8 18:25:25.798483 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Dec  8 18:25:25.822953 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- add: Successfully created ACL rule rule_nd in table EVERFLOW
Dec  8 18:25:25.863219 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Dec  8 18:25:25.863219 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1

Dec  8 18:25:25.868860 str2-7050cx3-acs-02 NOTICE admin: Pausing orchagent ...

Dec  8 18:25:25.975167 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Dec  8 18:25:25.975167 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Dec  8 18:25:25.975206 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Dec  8 18:25:25.975232 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1

Dec  8 18:25:26.048098 str2-7050cx3-acs-02 NOTICE swss#orchagent_restart_check: :- main: Wait time for response from orchagent set to 2000 milliseconds
Dec  8 18:25:26.048098 str2-7050cx3-acs-02 NOTICE swss#orchagent_restart_check: :- main: Number of retries for the request to orchagent is set to 5
Dec  8 18:25:26.048796 str2-7050cx3-acs-02 INFO swss#orchagent_restart_check: :- subscribe: subscribed to RESTARTCHECKREPLY
Dec  8 18:25:26.048796 str2-7050cx3-acs-02 NOTICE swss#orchagent_restart_check: :- main: requested orchagent to do warm restart state check, retry count: 0

Dec  8 18:25:26.049122 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- doTask: RESTARTCHECK notification for orchagent 
Dec  8 18:25:26.049122 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- doTask: orchagent|NoFreeze:false|SkipPendingTaskCheck:false
Dec  8 18:25:26.049331 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Dec  8 18:25:26.049356 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Dec  8 18:25:26.049356 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- warmRestartCheck: WarmRestart check found pending tasks: 
Dec  8 18:25:26.049396 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- warmRestartCheck:     VXLAN_REMOTE_VNI_TABLE:Vlan1000:192.168.8.1|SET|vni:1000
Dec  8 18:25:26.049424 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- warmRestartCheck: Restart check result: NOT_READY
Dec  8 18:25:26.049514 str2-7050cx3-acs-02 NOTICE swss#orchagent_restart_check: :- main: RESTARTCHECK failed, orchagent is not ready for warm restart with status NOT_READY
Dec  8 18:25:26.049552 str2-7050cx3-acs-02 NOTICE swss#orchagent_restart_check: :- main: requested orchagent to do warm restart state check, retry count: 1

Dec  8 18:25:26.049703 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- doTask: RESTARTCHECK notification for orchagent 
Dec  8 18:25:26.049703 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- doTask: orchagent|NoFreeze:false|SkipPendingTaskCheck:false
Dec  8 18:25:26.050026 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Dec  8 18:25:26.050026 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Dec  8 18:25:26.050026 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- warmRestartCheck: WarmRestart check found pending tasks: 
Dec  8 18:25:26.050026 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- warmRestartCheck:     VXLAN_REMOTE_VNI_TABLE:Vlan1000:192.168.8.1|SET|vni:1000
Dec  8 18:25:26.050026 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- warmRestartCheck: Restart check result: NOT_READY
Dec  8 18:25:26.050026 str2-7050cx3-acs-02 NOTICE swss#orchagent_restart_check: :- main: RESTARTCHECK failed, orchagent is not ready for warm restart with status NOT_READY
Dec  8 18:25:26.050026 str2-7050cx3-acs-02 NOTICE swss#orchagent_restart_check: :- main: requested orchagent to do warm restart state check, retry count: 2

Dec  8 18:25:26.050156 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- doTask: RESTARTCHECK notification for orchagent 
Dec  8 18:25:26.050191 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- doTask: orchagent|NoFreeze:false|SkipPendingTaskCheck:false
Dec  8 18:25:26.050268 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Dec  8 18:25:26.050406 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Dec  8 18:25:26.050406 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- warmRestartCheck: WarmRestart check found pending tasks: 
Dec  8 18:25:26.050439 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- warmRestartCheck:     VXLAN_REMOTE_VNI_TABLE:Vlan1000:192.168.8.1|SET|vni:1000
Dec  8 18:25:26.050439 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- warmRestartCheck: Restart check result: NOT_READY
Dec  8 18:25:26.050506 str2-7050cx3-acs-02 NOTICE swss#orchagent_restart_check: :- main: RESTARTCHECK failed, orchagent is not ready for warm restart with status NOT_READY
Dec  8 18:25:26.050564 str2-7050cx3-acs-02 NOTICE swss#orchagent_restart_check: :- main: requested orchagent to do warm restart state check, retry count: 3

Dec  8 18:25:26.050873 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- doTask: RESTARTCHECK notification for orchagent 
Dec  8 18:25:26.050873 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- doTask: orchagent|NoFreeze:false|SkipPendingTaskCheck:false
Dec  8 18:25:26.050906 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Dec  8 18:25:26.050906 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Dec  8 18:25:26.050944 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- warmRestartCheck: WarmRestart check found pending tasks: 
Dec  8 18:25:26.050944 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- warmRestartCheck:     VXLAN_REMOTE_VNI_TABLE:Vlan1000:192.168.8.1|SET|vni:1000
Dec  8 18:25:26.050981 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- warmRestartCheck: Restart check result: NOT_READY
Dec  8 18:25:26.051008 str2-7050cx3-acs-02 NOTICE swss#orchagent_restart_check: :- main: RESTARTCHECK failed, orchagent is not ready for warm restart with status NOT_READY
Dec  8 18:25:26.051058 str2-7050cx3-acs-02 NOTICE swss#orchagent_restart_check: :- main: requested orchagent to do warm restart state check, retry count: 4

Dec  8 18:25:26.051377 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- doTask: RESTARTCHECK notification for orchagent 
Dec  8 18:25:26.051377 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- doTask: orchagent|NoFreeze:false|SkipPendingTaskCheck:false
Dec  8 18:25:26.051409 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Dec  8 18:25:26.051409 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Dec  8 18:25:26.051450 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- warmRestartCheck: WarmRestart check found pending tasks: 
Dec  8 18:25:26.051500 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- warmRestartCheck:     VXLAN_REMOTE_VNI_TABLE:Vlan1000:192.168.8.1|SET|vni:1000
Dec  8 18:25:26.051548 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- warmRestartCheck: Restart check result: NOT_READY
Dec  8 18:25:26.051607 str2-7050cx3-acs-02 NOTICE swss#orchagent_restart_check: :- main: RESTARTCHECK failed, orchagent is not ready for warm restart with status NOT_READY
Dec  8 18:25:26.051656 str2-7050cx3-acs-02 NOTICE swss#orchagent_restart_check: :- main: requested orchagent to do warm restart state check, retry count: 5

Dec  8 18:25:26.051718 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- doTask: RESTARTCHECK notification for orchagent 
Dec  8 18:25:26.051836 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- doTask: orchagent|NoFreeze:false|SkipPendingTaskCheck:false
Dec  8 18:25:26.051871 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1
Dec  8 18:25:26.052170 str2-7050cx3-acs-02 WARNING swss#orchagent: :- addOperation: Vxlan tunnelPort doesn't exist: 192.168.8.1
Dec  8 18:25:26.052170 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- warmRestartCheck: WarmRestart check found pending tasks: 
Dec  8 18:25:26.052170 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- warmRestartCheck:     VXLAN_REMOTE_VNI_TABLE:Vlan1000:192.168.8.1|SET|vni:1000
Dec  8 18:25:26.052219 str2-7050cx3-acs-02 NOTICE swss#orchagent: :- warmRestartCheck: Restart check result: NOT_READY
Dec  8 18:25:26.052439 str2-7050cx3-acs-02 NOTICE swss#orchagent_restart_check: :- main: RESTARTCHECK failed, orchagent is not ready for warm restart with status NOT_READY
Dec  8 18:25:26.064588 str2-7050cx3-acs-02 NOTICE admin: warm-reboot failure (10) cleanup ...

Dec  8 18:25:26.075823 str2-7050cx3-acs-02 NOTICE admin: Tearing down control plane assistant: 10.64.246.125 ...
skbhava commented 1 year ago

nt: :- addTunnelUser: Unable to find EVPN VTEP. user=0 remote_vtep=192.168.8.1 Dec 8 18:25:26.052170 str2-7050cx3-acs-02 WARNING swss#orchage

Thanks. Will analyze the techsupport and get back on this.

vaibhavhd commented 1 year ago

@skbhava is there an update? Do you know if this is a SAI issue?

skbhava commented 1 year ago

@vaibhavhd Thanks for sharing the tech support. This would not be a SAI issue. From the logs, it looks like the nvo object doesnt seems to be created in the orch-agent which results the remote vni-vlan map addition failing. But at this point, not sure whether the problem is at the vxlanmgr or vxlanorch as the vxlan config seems to be removed at the time of tech support collection and unable to confirm whether the nvo objects exists in config db/app db during problematic state. Are you able to consistently reproduce the issue in your local setup. Can you please share the config you are using to recreate this issue. Also, if the issue consistently seen, is it possible to collect the tech support before removing the vxlan configs and share the same.

srj102 commented 1 year ago

@vaibhavhd was VXLAN_EVPN_NVO present in the CONFIG_DB ?

dgsudharsan commented 1 year ago

@prsunny to follow up on providing inputs from MSFT