search

networkservicemesh / deployments-k8s

Apache License 2.0
42 stars 35 forks source link

Unstable CI #12544

Open NikitaSkrynnik opened 1 day ago

NikitaSkrynnik commented 1 day ago

Description

This issue aggregates all the issues that relate to CI problems and provides a decomposition for them.

sdk

  1. https://github.com/networkservicemesh/sdk/issues/1627
  2. https://github.com/networkservicemesh/sdk/issues/1615
  3. https://github.com/networkservicemesh/sdk/issues/1593
  4. https://github.com/networkservicemesh/sdk/issues/1592
  5. https://github.com/networkservicemesh/sdk/issues/1591
  6. https://github.com/networkservicemesh/sdk/issues/1575
  7. https://github.com/networkservicemesh/sdk/issues/1574
  8. https://github.com/networkservicemesh/sdk/issues/1573
  9. https://github.com/networkservicemesh/sdk/issues/1444
  10. https://github.com/networkservicemesh/sdk/issues/839
  11. Test_DiscoverForwarder_ChangeForwarderOnClose
  12. Test_DiscoverForwarder_ChangeForwarderOnDeath_LostHeal
  13. Test_DiscoverForwarder_CloseAfterError
  14. Test_DNSUsecase
  15. TestNSMGR_HealEndpoint/Local_New
  16. TestNSMGR_HealEndpoint/Remote_New
  17. TestNSMGRHealEndpoint_DataPlaneBroken_CtrlPlaneBroken
  18. https://github.com/networkservicemesh/sdk/issues/1695

sdk-k8s

  1. https://github.com/networkservicemesh/sdk-k8s/issues/514
  2. https://github.com/networkservicemesh/sdk-k8s/issues/402

integration tests

  1. https://github.com/networkservicemesh/integration-k8s-kind/pull/1035 (First we need to fix the issue with pod deletion)
  2. https://github.com/networkservicemesh/integration-k8s-kind/issues/1008
  3. https://github.com/networkservicemesh/integration-k8s-kind/issues/1007
  4. https://github.com/networkservicemesh/integration-k8s-kind/issues/905
  5. https://github.com/networkservicemesh/integration-k8s-kind/issues/904
  6. https://github.com/networkservicemesh/integration-k8s-kind/issues/872
  7. https://github.com/networkservicemesh/integration-k8s-kind/issues/839
  8. https://github.com/networkservicemesh/integration-k8s-kind/issues/776
  9. https://github.com/networkservicemesh/integration-k8s-kind/issues/671
  10. https://github.com/networkservicemesh/integration-k8s-kind/issues/633
  11. https://github.com/networkservicemesh/integration-k8s-kind/issues/627
  12. https://github.com/networkservicemesh/integration-k8s-kind/issues/625

public clusters

  1. Packet: https://github.com/networkservicemesh/integration-k8s-packet/issues/405
  2. AWS: https://github.com/networkservicemesh/integration-k8s-aws/pull/423

decomposition

sdk and sdk-k8s

  1. Check if test is still unstable - 2h
  2. Fix test if it's unstuble - 2h (positive), 10h (negative) Total negative: (2h + 10h) 18 = 216h or 31d Total positive: (2h + 2h) 18 = 68h or 10d

integration tests

  1. Fix the issue with proper pod deletion - 2d (positive), 5d (negative)
  2. Each integration test (11 tests) - 1.5d (positive), 3d (negative) Total negative: 5d + 11 3d = 38d Total positive: 2d + 11 1.5d = 18.5d

public clusters

  1. Packet - 2d (positive), 5d (negative)
  2. AWS - 2d (positive), 5d (negative)

Total positive: 10d + 18.5d + 4d = 32.5d Total negative: 31d + 38d + 10d = 79d

NikitaSkrynnik commented 9 hours ago

Tests by priority

HIGH

  1. TestNSMGRHealEndpoint_DatapathHealthy_CtrlPlaneBroken (4h to 12h)
  2. Test_NSC_ConnectsTo_vl3NSE (4h to 12h)
  3. TestNSMGR_HealEndpoint/Local_New (4h to 12h)
  4. TestPassThrough/TestDeleteToDown (4h to 12h)
  5. The pod deletion issue in integration-k8s-kind (2d to 5d)
  6. Packet cluster (2d to 5d)

Total positive: 16h + 2d + 2d = 6d Total negative: 48h + 5d + 5d = 17d

MEDIUM TODO