Container to container networking performance degradation - Ubuntu Jammy stemcell

Issue

Sometime ago I posted a message on Slack regarding timeouts observed in C2C when moving from Bionic to Jammy [1]. Since then we have updated our test environment to cf-deployment [v27.2.0] and the Jammy stemcell to 1.95 which improved the situation but we still observe slow connections from / to the containers.

Please note that this issue can only be observed when the Diego cells are migrated from Bionic to Jammy and cannot be reproduced on Bionic stemcells. As additional information, we've also tested in different environments with and without dynamic ASGs.

Steps to Reproduce

Install cf-deployment [v27.2.0] on Jammy stemcell
Push two apps and add a network-policy enabling traffic from app A to app B
cf add-network-policy app-a app-b --protocol tcp --port 8080
ssh into app-a and try to reach app-b

Expected result

Successful connections from app-a to app-b.

Current result

Sporadic timeouts and slow connections from app-a to app-b.

[backend-wgnnmafs]: Hello!
real    0m2.035s
user    0m0.000s
sys     0m0.007s
[backend-wgnnmafs]: Hello!
real    0m0.018s
user    0m0.000s
sys     0m0.007s
[backend-wgnnmafs]: Hello!
real    0m1.039s
user    0m0.000s
sys     0m0.007s

Additional Context

[1] https://cloudfoundry.slack.com/archives/C01ABMVNE9E/p1678891121644269

cloudfoundry / cf-networking-release