search

networkservicemesh / deployments-k8s

Apache License 2.0
42 stars 34 forks source link

Policy Based Routing; empty routing tables on NSC reconnect #9639

Closed ljkiraly closed 1 year ago

ljkiraly commented 1 year ago

Expected Behavior

The routes set by policy based routing should not interfere.

Current Behavior

In a Kernel2VXLAN2Kernel use case some routing tables belongin to policy rules are missing. The NSC container is collocated with other application containers in a pod. The NSC application is killed. The connection remains open since the data path healing is disabled. When the NSC container restarts a new connection is established by the forwarder - two parallel connection coexist for the same data-path. After the 10 minutes timeout the 'ghost' connection is closed by the forwarder and at this point the routing table is flushed for the related routing policy. The other connection is sane and remains, but the missing rule are never updated since the forwarder stores the policy keyed by the connection ID and thinks it is already set into the application pod's namespace. Most probably this should not happen if data path healing is enabled or NSC reuse the connection ID requested from nsmgr, but the forwarder might tolerate these deviations.

Context

Failure Logs

The last request from the killed NSC:

Jul  7 09:32:41.364ESC[37m [TRAC] [id:e91c291a-057e-4722-906d-1376c88c701c] [type:networkService] ESC[0m(1.1)   request={"connection":{"id":"e91c291a-057e-4722-906d-1376c88c701c","network_service":"proxy.sc2","mechanism":{"cls":"LOCAL","type":"KERNEL","parameters":{"inodeURL":"inode://4/4026538469","name":"nsm-0"}},"context":{"ip_context":{"src_ip_addrs":["172.16.17.12/24","214.14.132.65/32","214.14.132.66/32"],"dst_ip_addrs":["172.16.17.13/24"],"excluded_prefixes":["214.14.176.1/32","214.14.176.2/31","214.14.176.33/32","214.14.176.34/31"],"extra_prefixes":["172.16.17.1/24"],"policies":[{"from":"214.14.132.65/32","routes":[{"prefix":"0.0.0.0/0","nextHop":"172.16.17.1"}]},{"from":"214.14.132.66/32","routes":[{"prefix":"0.0.0.0/0","nextHop":"172.16.17.1"}]}]}},"labels":{"nodeName":"pool1-n106-vpod6-pool1-n7"},"path":{"index":1,"path_segments":[{"name":"8fb4cf9c6-sxdxz","id":"8fb4cf9c6-sxdxz-proxy.sc2-0","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9zZW5wLmluZnJhL25zL2NuZHNjMi9wb2QvZXJpYy1kc2MtZmRyLThmYjRjZjljNi1zeGR4eiIsImF1ZCI6WyJzcGlmZmU6Ly9zZW5wLmluZnJhL25zL3NlbnAvcG9kL2VyaWMtdG0tc2VucC1pbmZyYS1uc21nci01NGhmbSJdLCJleHAiOjE2ODg3MjQyNjJ9.lkuEX3ErXuBR_CR5bvlvrqiAwgKMjGD3z5XEmceUO8a7ncGuxgj7bY1Q5lI0ywmT9e4v7oRnS-JnZrCMz96PTQ","expires":{"seconds":1688724262}},{"name":"nsmgr-54hfm","id":"e91c291a-057e-4722-906d-1376c88c701c","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9zZW5wLmluZnJhL25zL3NlbnAvcG9kL2VyaWMtdG0tc2VucC1pbmZyYS1uc21nci01NGhmbSIsImF1ZCI6WyJzcGlmZmU6Ly9zZW5wLmluZnJhL25zL2NuZHNjMi9wb2QvZXJpYy1kc2MtZmRyLThmYjRjZjljNi1zeGR4eiJdLCJleHAiOjE2ODg3MjI5NjF9.rJ-fmuUxO43yExsabL2rVKGdrWKvni8FmdteXMtRPehO8nWdyKXtvfV6ENEpoRunTvnbkWjmsbJbEeVT3kWNcw","expires":{"seconds":1688722961,"nanos":321418856}},{"name":"forwarder-vpp-2l2mb","id":"577dc06a-6e49-49f3-a21a-b95e32e6c896","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9zZW5wLmluZnJhL25zL3NlbnAvcG9kL2VyaWMtdG0tc2VucC1pbmZyYS1mb3J3YXJkZXItdnBwLTJsMm1iIiwiYXVkIjpbInNwaWZmZTovL3NlbnAuaW5mcmEvbnMvY25kc2MyL3BvZC9lcmljLXRtLXNlbnAtbnZpcC1wcm94eS12cG4yLWp6OHMyIl0sImV4cCI6MTY4ODcyMjk2MH0.FxDZz3RUf2hpauXl6sUGaaAOYdU29KGQbj1sY2meLAbl84-VdvQ1qb6Zk_E3yNDv4CbVGfEg4QC_UmcSxcOk-Q","expires":{"seconds":1688722960,"nanos":861077586},"metrics":{"client_drops":"1","client_rx_bytes":"4503502","client_rx_packets":"17859","client_tx_bytes":"5737662","client_tx_packets":"32185","server_drops":"0","server_rx_bytes":"5737662","server_rx_packets":"32185","server_tx_bytes":"4503412","server_tx_packets":"17858"}},{"name":"proxy-vpn2-jz8s2","id":"195059cd-1fe8-4b56-a33c-d33b01b07ceb","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9zZW5wLmluZnJhL25zL2NuZHNjMi9wb2QvZXJpYy10bS1zZW5wLW52aXAtcHJveHktdnBuMi1qejhzMiIsImF1ZCI6WyJzcGlmZmU6Ly9zZW5wLmluZnJhL25zL3NlbnAvcG9kL2VyaWMtdG0tc2VucC1pbmZyYS1mb3J3YXJkZXItdnBwLTJsMm1iIl0sImV4cCI6MTY4ODcyMjk2MH0.svszR0NJpyknDhmgI3E02l9txN5V6fYsz1mYbezlStd480iOjDGo1D9DPbKrc3RJnhgwUJyYqOdVqsM_j1Za7g","expires":{"seconds":1688722960,"nanos":861879128}}]},"payload":"ETHERNET"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"inodeURL":"inode://4/4026538469","name":"proxy.vpn2"}}]}

The request for the new connection:

Jul  7 09:32:47.645ESC[37m [TRAC] [id:40154849-6578-4425-9f70-b5988dc4c1c5] [type:networkService] ESC[0m(1.1)   request={"connection":{"id":"40154849-6578-4425-9f70-b5988dc4c1c5","network_service":"proxy.sc2","mechanism":{"cls":"LOCAL","type":"KERNEL","parameters":{"inodeURL":"inode://4/4026538469","name":"nsm-0"}},"context":{"ip_context":{"src_ip_addrs":["172.16.17.12/24","214.14.132.65/32","214.14.132.66/32"],"dst_ip_addrs":["172.16.17.13/24"],"excluded_prefixes":["214.14.176.1/32","214.14.176.2/31","214.14.176.33/32","214.14.176.34/31"],"extra_prefixes":["172.16.17.1/24"],"policies":[{"from":"214.14.132.65/32","routes":[{"prefix":"0.0.0.0/0","nextHop":"172.16.17.1"}]},{"from":"214.14.132.66/32","routes":[{"prefix":"0.0.0.0/0","nextHop":"172.16.17.1"}]}]}},"labels":{"nodeName":"pool1-n106-vpod6-pool1-n7"},"path":{"index":1,"path_segments":[{"name":"8fb4cf9c6-sxdxz","id":"8fb4cf9c6-sxdxz-proxy.sc2-0","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9zZW5wLmluZnJhL25zL2NuZHNjMi9wb2QvZXJpYy1kc2MtZmRyLThmYjRjZjljNi1zeGR4eiIsImF1ZCI6WyJzcGlmZmU6Ly9zZW5wLmluZnJhL25zL3NlbnAvcG9kL2VyaWMtdG0tc2VucC1pbmZyYS1uc21nci01NGhmbSJdLCJleHAiOjE2ODg3MjQyNjJ9.skXRejpQSe2u8LgZYVSDwJLyFEOp4UFg8lgWtfJZbAALVTDOKCk_a00iIsYWXh9oWwgN0KqJQTMhrODH49yxGA","expires":{"seconds":1688724262}},{"name":"nsmgr-54hfm","id":"40154849-6578-4425-9f70-b5988dc4c1c5","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9zZW5wLmluZnJhL25zL3NlbnAvcG9kL2VyaWMtdG0tc2VucC1pbmZyYS1uc21nci01NGhmbSIsImF1ZCI6WyJzcGlmZmU6Ly9zZW5wLmluZnJhL25zL2NuZHNjMi9wb2QvZXJpYy1kc2MtZmRyLThmYjRjZjljNi1zeGR4eiJdLCJleHAiOjE2ODg3MjI5Njd9.du0cImznAsN5MAt5VjaQJ_4-3i4WZf5s-AyN1th_8jVrRjkhiUiBV8BxtmOtggFm-zRO10xjRJxbRX5YV3jb0g","expires":{"seconds":1688722967,"nanos":521465619}},{"name":"forwarder-vpp-2l2mb","id":"1f814d0b-b113-47a7-81cb-c4066582d6f1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9zZW5wLmluZnJhL25zL3NlbnAvcG9kL2VyaWMtdG0tc2VucC1pbmZyYS1mb3J3YXJkZXItdnBwLTJsMm1iIiwiYXVkIjpbInNwaWZmZTovL3NlbnAuaW5mcmEvbnMvY25kc2MyL3BvZC9lcmljLXRtLXNlbnAtbnZpcC1wcm94eS12cG4yLWp6OHMyIl0sImV4cCI6MTY4ODcyMjk2NH0.95Ht-eE8AEOapSlwd1tMLGf_yGd-Yva0i92NJ-KoS8c-AdcWaNMPw2VOL73Yig3MxyPw7R0uHrMijIvieUXKpg","expires":{"seconds":1688722964,"nanos":672197373},"metrics":{"client_drops":"0","client_rx_bytes":"0","client_rx_packets":"0","client_tx_bytes":"0","client_tx_packets":"0","server_drops":"0","server_rx_bytes":"0","server_rx_packets":"0","server_tx_bytes":"0","server_tx_packets":"0"}},{"name":"proxy-vpn2-jz8s2","id":"3651ac20-ee91-45bd-bc3e-6d551cb88cfc","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9zZW5wLmluZnJhL25zL2NuZHNjMi9wb2QvZXJpYy10bS1zZW5wLW52aXAtcHJveHktdnBuMi1qejhzMiIsImF1ZCI6WyJzcGlmZmU6Ly9zZW5wLmluZnJhL25zL3NlbnAvcG9kL2VyaWMtdG0tc2VucC1pbmZyYS1mb3J3YXJkZXItdnBwLTJsMm1iIl0sImV4cCI6MTY4ODcyMjk2NH0.m0FLyl1skiyrCPb7aNQfyc1KBWD9xzeobWpB5IzKQUfVkWv420fDOeQpuhK1NEp6Uilg97yOyi4u0Kl1STaDgw","expires":{"seconds":1688722964,"nanos":672892012}}]},"payload":"ETHERNET"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"inodeURL":"inode://4/4026538469","name":"proxy.vpn2"}}]}

Jul  7 09:32:47.789[37m [TRAC] [id:1f814d0b-b113-47a7-81cb-c4066582d6f1] [type:networkService] [0m(85.1) request-response={"id":"1f814d0b-b113-47a7-81cb-c4066582d6f1","network_service":"proxy.vpn2","mechanism":{"cls":"LOCAL","type":"KERNEL","parameters":{"inodeURL":"inode://4/4026538613","name":"proxy.vpn2-3651"}},"context":{"ip_context":{"src_ip_addrs":["172.16.17.12/24","214.14.132.65/32","214.14.132.66/32"],"dst_ip_addrs":["172.16.17.13/24"],"excluded_prefixes":["214.14.176.1/32","214.14.176.2/31","214.14.176.33/32","214.14.176.34/31"],"extra_prefixes":["172.16.17.1/24"],"policies":[{"from":"214.14.132.65/32","routes":[{"prefix":"0.0.0.0/0","nextHop":"172.16.17.1"}]},{"from":"214.14.132.66/32","routes":[{"prefix":"0.0.0.0/0","nextHop":"172.16.17.1"}]}]},"MTU":1500},"labels":{"nodeName":"pool1-n106-n7"},"path":{"index":2,"path_segments":[{"name":"8fb4cf9c6-sxdxz","id":"8fb4cf9c6-sxdxz-proxy.vpn2-0","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9zZW5wLmluZnJhL25zL2NuZHNjMi9wb2QvZXJpYy1kc2MtZmRyLThmYjRjZjljNi1zeGR4eiIsImF1ZCI6WyJzcGlmZmU6Ly9zZW5wLmluZnJhL25zL3NlbnAvcG9kL2VyaWMtdG0tc2VucC1pbmZyYS1uc21nci01NGhmbSJdLCJleHAiOjE2ODg3MjQyNjJ9.skXRejpQSe2u8LgZYVSDwJLyFEOp4UFg8lgWtfJZbAALVTDOKCk_a00iIsYWXh9oWwgN0KqJQTMhrODH49yxGA","expires":{"seconds":1688724262}},{"name":"nsmgr-54hfm","id":"40154849-6578-4425-9f70-b5988dc4c1c5","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9zZW5wLmluZnJhL25zL3NlbnAvcG9kL2VyaWMtdG0tc2VucC1pbmZyYS1uc21nci01NGhmbSIsImF1ZCI6WyJzcGlmZmU6Ly9zZW5wLmluZnJhL25zL3NlbnAvcG9kL2VyaWMtdG0tc2VucC1pbmZyYS1mb3J3YXJkZXItdnBwLTJsMm1iIl0sImV4cCI6MTY4ODcyMjk2N30.WtI5-6sxmbTC-j26bainHSwwKfY7Q8mx7m0SslaWBoj0Vh7WvhEXDzcNoiXHX3JtzbHy8731-E2kx6UR-k0-sw","expires":{"seconds":1688722967,"nanos":644830687}},{"name":"forwarder-vpp-2l2mb","id":"1f814d0b-b113-47a7-81cb-c4066582d6f1","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9zZW5wLmluZnJhL25zL3NlbnAvcG9kL2VyaWMtdG0tc2VucC1pbmZyYS1mb3J3YXJkZXItdnBwLTJsMm1iIiwiYXVkIjpbInNwaWZmZTovL3NlbnAuaW5mcmEvbnMvY25kc2MyL3BvZC9lcmljLXRtLXNlbnAtbnZpcC1wcm94eS12cG4yLWp6OHMyIl0sImV4cCI6MTY4ODcyMjk2N30.pyHOWUJg_KtVCjv8Ly-sjIhKC3m8unbnqm_f7wxY3RWfprpL99dSNFuj2c7qrgkeUaKJUhVgg6CGa74iK9H38Q","expires":{"seconds":1688722967,"nanos":783779203},"metrics":{"client_drops":"0","client_rx_bytes":"0","client_rx_packets":"0","client_tx_bytes":"0","client_tx_packets":"0","server_drops":"0","server_rx_bytes":"0","server_rx_packets":"0","server_tx_bytes":"0","server_tx_packets":"0"}},{"name":"proxy-vpn2-jz8s2","id":"3651ac20-ee91-45bd-bc3e-6d551cb88cfc","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9zZW5wLmluZnJhL25zL2NuZHNjMi9wb2QvZXJpYy10bS1zZW5wLW52aXAtcHJveHktdnBuMi1qejhzMiIsImF1ZCI6WyJzcGlmZmU6Ly9zZW5wLmluZnJhL25zL3NlbnAvcG9kL2VyaWMtdG0tc2VucC1pbmZyYS1mb3J3YXJkZXItdnBwLTJsMm1iIl0sImV4cCI6MTY4ODcyMjk2N30.lY18nZwwgxVdZS_1YM7TSHGFPr6TJpr7Rpr-Kz5qUmNwTelawzkVVfXgFq6K9R76YclbSCJUK4MLhAlXO5eFuQ","expires":{"seconds":1688722967,"nanos":784512935}}]},"network_service_endpoint_name":"proxy-vpn2-jz8s2","payload":"ETHERNET"}

The close after timeout:

Jul  7 09:42:21.845ESC[37m [TRAC] [id:577dc06a-6e49-49f3-a21a-b95e32e6c896] [type:networkService] ESC[0m(2.1)    close={"id":"577dc06a-6e49-49f3-a21a-b95e32e6c896","network_service":"proxy.sc2","mechanism":{"cls":"LOCAL","type":"KERNEL","parameters":{"inodeURL":"inode://4/4026538469","name":"nsm-0"}},"context":{"ip_context":{"src_ip_addrs":["172.16.17.12/24","214.14.132.65/32","214.14.132.66/32"],"dst_ip_addrs":["172.16.17.13/24"],"excluded_prefixes":["214.14.176.1/32","214.14.176.2/31","214.14.176.33/32","214.14.176.34/31"],"extra_prefixes":["172.16.17.1/24"],"policies":[{"from":"214.14.132.65/32","routes":[{"prefix":"0.0.0.0/0","nextHop":"172.16.17.1"}]},{"from":"214.14.132.66/32","routes":[{"prefix":"0.0.0.0/0","nextHop":"172.16.17.1"}]}]},"MTU":1500},"labels":{"nodeName":"pool1-n106-n7"},"path":{"index":2,"path_segments":[{"name":"8fb4cf9c6-sxdxz","id":"8fb4cf9c6-sxdxz-proxy.sc2-0","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9zZW5wLmluZnJhL25zL2NuZHNjMi9wb2QvZXJpYy1kc2MtZmRyLThmYjRjZjljNi1zeGR4eiIsImF1ZCI6WyJzcGlmZmU6Ly9zZW5wLmluZnJhL25zL3NlbnAvcG9kL2VyaWMtdG0tc2VucC1pbmZyYS1uc21nci01NGhmbSJdLCJleHAiOjE2ODg3MjQyNjJ9.lkuEX3ErXuBR_CR5bvlvrqiAwgKMjGD3z5XEmceUO8a7ncGuxgj7bY1Q5lI0ywmT9e4v7oRnS-JnZrCMz96PTQ","expires":{"seconds":1688724262}},{"name":"nsmgr-54hfm","id":"e91c291a-057e-4722-906d-1376c88c701c","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9zZW5wLmluZnJhL25zL3NlbnAvcG9kL2VyaWMtdG0tc2VucC1pbmZyYS1uc21nci01NGhmbSIsImF1ZCI6WyJzcGlmZmU6Ly9zZW5wLmluZnJhL25zL3NlbnAvcG9kL2VyaWMtdG0tc2VucC1pbmZyYS1mb3J3YXJkZXItdnBwLTJsMm1iIl0sImV4cCI6MTY4ODcyMjk2MX0.2Pp1JefNzozeBJig34EI1EDJ4HysDXh9D23GcB1Ib2digwSYyI_Lka2PCnOjNLxGTauvHYmR2r4kMkgmWjdNWA","expires":{"seconds":1688722961,"nanos":364294020}},{"name":"forwarder-vpp-2l2mb","id":"577dc06a-6e49-49f3-a21a-b95e32e6c896","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9zZW5wLmluZnJhL25zL3NlbnAvcG9kL2VyaWMtdG0tc2VucC1pbmZyYS1mb3J3YXJkZXItdnBwLTJsMm1iIiwiYXVkIjpbInNwaWZmZTovL3NlbnAuaW5mcmEvbnMvY25kc2MyL3BvZC9lcmljLXRtLXNlbnAtbnZpcC1wcm94eS12cG4yLWp6OHMyIl0sImV4cCI6MTY4ODcyMjk2MX0.6nCsHfvY3SMqK40z8ePSq6xw_VVlcR2rI_eJ5QOnin5Jhj8AdUh403JnwqRrk6INB2Jdf6x_pFZIa0buHXY37w","expires":{"seconds":1688722961,"nanos":465053732},"metrics":{"client_drops":"1","client_rx_bytes":"4503502","client_rx_packets":"17859","client_tx_bytes":"5737662","client_tx_packets":"32185","server_drops":"0","server_rx_bytes":"5737662","server_rx_packets":"32185","server_tx_bytes":"4503412","server_tx_packets":"17858"}},{"name":"proxy-vpn2-jz8s2","id":"195059cd-1fe8-4b56-a33c-d33b01b07ceb","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJzcGlmZmU6Ly9zZW5wLmluZnJhL25zL2NuZHNjMi9wb2QvZXJpYy10bS1zZW5wLW52aXAtcHJveHktdnBuMi1qejhzMiIsImF1ZCI6WyJzcGlmZmU6Ly9zZW5wLmluZnJhL25zL3NlbnAvcG9kL2VyaWMtdG0tc2VucC1pbmZyYS1mb3J3YXJkZXItdnBwLTJsMm1iIl0sImV4cCI6MTY4ODcyMjk2MX0.jLAWyQXlaKUfKeYWTMpZ_5Q0w_1Mv7PJ0hLb4w5rD9XwGHyyOTZOG8UoWoX35nE0gT7Pz_90yjqFcborPVKI3g","expires":{"seconds":1688722961,"nanos":465772612}}]},"network_service_endpoint_name":"proxy-vpn2-jz8s2","payload":"ETHERNET"}

Jul  7 09:42:21.880[37m [DEBU] [id:577dc06a-6e49-49f3-a21a-b95e32e6c896] [Route:{Ifindex: 7 Dst: <nil> Src: <nil> Gw: 172.16.17.1 Flags: [onlink] Table: 1 Realm: 0}] [netlink:RouteDel] [type:networkService] [0m(22.1)                        completed

Jul  7 09:42:21.880[37m [DEBU] [id:577dc06a-6e49-49f3-a21a-b95e32e6c896] [netlink:flushTable] [tableID:1] [type:networkService] [0m(22.2)                        completed

Jul  7 09:42:21.880[37m [DEBU] [id:577dc06a-6e49-49f3-a21a-b95e32e6c896] [DstPort:] [From:214.14.132.65/32] [IPProto:] [SrcPort:] [duration:22.638µs] [netlink:RuleDel] [type:networkService] [0m(22.3)                        completed

Jul  7 09:42:21.880[37m [DEBU] [id:577dc06a-6e49-49f3-a21a-b95e32e6c896] [Route:{Ifindex: 7 Dst: <nil> Src: <nil> Gw: 172.16.17.1 Flags: [onlink] Table: 2 Realm: 0}] [netlink:RouteDel] [type:networkService] [0m(22.4)                        completed

Jul  7 09:42:21.880[37m [DEBU] [id:577dc06a-6e49-49f3-a21a-b95e32e6c896] [netlink:flushTable] [tableID:2] [type:networkService] [0m(22.5)                        completed

Jul  7 09:42:21.880[37m [DEBU] [id:577dc06a-6e49-49f3-a21a-b95e32e6c896] [DstPort:] [From:214.14.132.66/32] [IPProto:] [SrcPort:] [duration:15.588µs] [netlink:RuleDel] [type:networkService] [0m(22.6)                        completed

The rules and routing configuration in the application pod:

# ip rule
0:    from all lookup local
32762:    from 214.14.132.66 lookup 4
32763:    from 214.14.132.65 lookup 3
32764:    from 214.14.131.113 lookup 2
32765:    from 214.14.131.114 lookup 1
32766:    from all lookup main
32767:    from all lookup default
bash-4.4# ip route show table all
default via 172.16.16.1 dev nsm-1 table 3 onlink 
default via 172.16.16.1 dev nsm-1 table 4 onlink 
default via 169.254.1.1 dev eth0 
169.254.1.1 dev eth0 scope link 
172.16.1.0/24 dev nsm-0 proto kernel scope link src 172.16.1.12 
172.16.16.0/24 dev nsm-1 proto kernel scope link src 172.16.16.12 
.....
ljkiraly commented 1 year ago

Fixed with https://github.com/networkservicemesh/sdk-kernel/pull/604