Controlplane part of `forwarder-vpp` leaks

[NikitaSkrynnik]

Description

forwarder-vpp has two interface leaks:

1. timeout chain element doesn't call Close for expired connections. Therefore, vpp interfaces are not deleted
2. Even if we call Close in forwarder-vpp it doesn't delete vxlan interfaces

Tasks

• [x] Investigate why timeout doesn't call Close - 8h
  • [x] Make timeout chain element close connetions faster - 1h
  • [x] Add logs to timeout chain element - 1h
  • [x] Run tests with the modified timeout - 3h
  • [x] Check collected logs - 3h
  • [x] Fix timeout chain element - 24h
  • [x] Investigate why forwarder-vpp doesn't close vxlan interfaces - 6h
  • [x] Run scaling tests - 3h
  • [x] Check collected logs - 3h
• [x] Fix vxlan problem - 16h

Total: 54h

[NikitaSkrynnik]

All changes that fix tap interface leaks are in these PRs:

1. https://github.com/networkservicemesh/sdk/pull/1643
2. https://github.com/networkservicemesh/sdk-vpp/pull/835

[denis-tingaikin]

It seems like the problem is still existing and the forwarder is leaking.

The picture shows mem consumption for forwarder vpp.

forwarder-vpp_goroutineprofiles_20240920082027.tar.gz forwarder-vpp_memprofiles_20240920082112.tar.gz

[NikitaSkrynnik]

Current plan

• [x] Test 20 clients and 20 endpoint on Azure cluster with 2 nodes for 24 hours. Collect memory, goroutine profiles, vpp memory profiles and kubectl top after 10 min of testing and after 24 hours of testing.
• [ ] Test 40 clients and 1 endpoint on Azure cluster with 2 nodes for 24 hours. Scale clients for 0 to 40 every 60 seconds. Collect memory, goroutine profiles, vpp memory profiles and kubectl top at the beginning, after 10 min of testing and after 24 hours of testing.

[NikitaSkrynnik]

After fixing context leaks in nsmonitor and timeout chain element forwarder shows the next results on 30 NSCs and 10 NSEs:

3h of testing:

nsm-system    forwarder-vpp-gr729                   135m         363Mi           
nsm-system    forwarder-vpp-zjd5x                   200m         390Mi           
nsm-system    nsmgr-25vvt                           167m         92Mi            
nsm-system    nsmgr-f6rjf                           92m          71Mi

18h of testing:

nsm-system    forwarder-vpp-gr729                   132m         401Mi           
nsm-system    forwarder-vpp-zjd5x                   199m         438Mi           
nsm-system    nsmgr-25vvt                           175m         97Mi            
nsm-system    nsmgr-f6rjf                           92m          74Mi 

3d20h of testing:

nsm-system    forwarder-vpp-gr729                   272m         421Mi           
nsm-system    forwarder-vpp-zjd5x                   310m         443Mi           
nsm-system    nsmgr-25vvt                           197m         103Mi           
nsm-system    nsmgr-f6rjf                           101m         78Mi   

4d5h of testing:

nsm-system    forwarder-vpp-gr729                   228m         417Mi           
nsm-system    forwarder-vpp-zjd5x                   203m         448Mi           
nsm-system    nsmgr-25vvt                           213m         101Mi           
nsm-system    nsmgr-f6rjf                           110m         83Mi  

30m after testing:

nsm-system    forwarder-vpp-gr729                   66m          414Mi           
nsm-system    forwarder-vpp-zjd5x                   63m          428Mi           
nsm-system    nsmgr-25vvt                           49m          87Mi            
nsm-system    nsmgr-f6rjf                           46m          73Mi 

memory and goroutine profiles don't show any leaks 30m after testing. Here are all profiles: traces.zip

[denis-tingaikin]

@NikitaSkrynnik attached archive is empty, could you resend it?