search

spidernet-io / spiderpool

Underlay and RDMA network solution of the Kubernetes, for bare metal, VM and any public cloud
https://spidernet-io.github.io/spiderpool/
Apache License 2.0
505 stars 72 forks source link

After restarting Pods in batches, curling the Pod IP address from the node fails. #3562

Open github-actions[bot] opened 1 month ago

github-actions[bot] commented 1 month ago

action url: https://github.com/spidernet-io/spiderpool/actions/runs/9356525896

ty-dc commented 1 month ago

After rebuilding Pods in batches, checking network connectivity fails.

  [FAILED] Failed to execute the command curl --retry 10 -m 1 -g http://[fc00:f853:ccd:e793:f::1dc]:80 --insecure on the node:    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  curl: (28) Connection timed out after 1001 milliseconds
  Warning: Problem : timeout. Will retry in 1 seconds. 10 retries left.

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  curl: (28) Connection timed out after 1001 milliseconds
  Warning: Problem : timeout. Will retry in 2 seconds. 9 retries left.

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  curl: (28) Connection timed out after 1000 milliseconds
  Warning: Problem : timeout. Will retry in 4 seconds. 8 retries left.

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  curl: (28) Connection timed out after 1000 milliseconds
  Warning: Problem : timeout. Will retry in 8 seconds. 7 retries left.

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  curl: (28) Connection timed out after 1001 milliseconds
  Warning: Problem : timeout. Will retry in 16 seconds. 6 retries left.

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  curl: (28) Connection timed out after 1000 milliseconds
  Warning: Problem : timeout. Will retry in 32 seconds. 5 retries left.

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  curl: (28) Connection timed out after 1001 milliseconds
  Warning: Problem : timeout. Will retry in 64 seconds. 4 retries left.

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  curl: (28) Connection timed  out after 1000 milliseconds
  Warning: Problem : timeout. Will retry in 128 seconds. 3 retries left.

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  curl: (28) Connection timed out after 1001 milliseconds
  Warning: Problem : timeout. Will retry in 256 seconds. 2 retries left.out after 1000 milliseconds
  Warning: Problem : timeout. Will retry in 128 seconds. 3 retries left.

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  curl: (28) Connection timed out after 1001 milliseconds
  Warning: Problem : timeout. Will retry in 256 seconds. 2 retries left.

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  curl: (28) Connection timed out after 1000 milliseconds
  Warning: Problem : timeout. Will retry in 512 seconds. 1 retries left.

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  curl: (28) Connection timed out after 1000 milliseconds

  Unexpected error:
      <*exec.ExitError | 0xc000546b20>: 
      signal: killed
      {
          ProcessState: {
              pid: 184631,
              status: 9,
              rusage: {
                  Utime: {Sec: 0, Usec: 976},
                  Stime: {Sec: 0, Usec: 0},
                  Maxrss: 112688,
                  Ixrss: 0,
                  Idrss: 0,
                  Isrss: 0,
                  Minflt: 67,
                  Majflt: 0,
                  Nswap: 0,
                  Inblock: 0,
                  Oublock: 0,
                  Msgsnd: 0,
                  Msgrcv: 0,
                  Nsignals: 0,
                  Nvcsw: 3,
                  Nivcsw: 0,
              },
          },
          Stderr: nil,
      }
  occurred
  In [It] at: /home/runner/work/spiderpool/spiderpool/test/e2e/performance/performance_test.go:155 @ 06/03/24 21:02:08.607

Solution: Check the logs and confirm the problem.

ubuntu-latest-ipv6-e2e_test_underlay-v1.29.0-debuglog.txt.zip

ty-dc commented 1 month ago

Additional log supplement:

ubuntu-latest-dual-e2e_test_underlay-v1.23.17-debuglog.txt.zip

ty-dc commented 2 weeks ago

We will first try to reproduce the problem locally and troubleshoot the issue. If the problem is not resolved, we will add logs to e2e.