Open ritvik-statsig opened 2 years ago
Not sure what happened here, but it took ~13 hours for NebulaServer to update specs in NodeServers
I0302 00:58:42.259829 1 NebulaServer.cpp:495] Nebula server listening on 0.0.0.0:9190 I0302 00:58:52.817811 1 NebulaServer.cpp:560] Loading nebula cluster config: /tmp/nebula.kpu3Kg, old sign: , new sign: 2873_18208199511650023245, size: 2873 I0302 00:59:03.102922 1 ConnectionPool.cpp:64] Creating a channel to 10.240.0.219:9199 I0302 00:59:03.105432 1 ConnectionPool.cpp:64] Creating a channel to 10.240.0.181:9199 I0302 00:59:03.107265 1 ConnectionPool.cpp:64] Creating a channel to 10.240.0.108:9199 I0302 00:59:03.108273 1 ConnectionPool.cpp:64] Creating a channel to 10.240.0.58:9199 I0302 00:59:03.110499 1 ConnectionPool.cpp:64] Creating a channel to 10.240.0.66:9199 I0302 01:31:54.084906 1 ConnectionPool.cpp:64] Creating a channel to 10.240.0.156:9199 E0302 01:32:24.383826 1 NodeClient.cpp:169] RPC failed to 10.240.0.219: code=14, msg=Connect Failed I0302 01:32:24.383872 1 ConnectionPool.cpp:81] Removing a channel to 10.240.0.219:9199 I0302 02:24:42.695111 1 NebulaServer.cpp:594] Nebula Server updated itself 500 times. I0302 02:25:23.819861 1 ConnectionPool.cpp:64] Creating a channel to 10.240.1.56:9199 I0302 03:50:18.518813 1 NebulaServer.cpp:594] Nebula Server updated itself 1000 times. I0302 05:15:55.254267 1 NebulaServer.cpp:594] Nebula Server updated itself 1500 times. I0302 06:41:31.268491 1 NebulaServer.cpp:594] Nebula Server updated itself 2000 times. I0302 08:07:07.238289 1 NebulaServer.cpp:594] Nebula Server updated itself 2500 times. I0302 09:32:44.070459 1 NebulaServer.cpp:594] Nebula Server updated itself 3000 times. I0302 10:58:21.663203 1 NebulaServer.cpp:594] Nebula Server updated itself 3500 times. I0302 12:23:56.903872 1 NebulaServer.cpp:594] Nebula Server updated itself 4000 times. I0302 13:49:32.284674 1 NebulaServer.cpp:594] Nebula Server updated itself 4500 times. I0302 14:02:22.524799 1 SpecRepo.cpp:369] Updating 77 specs: brandnew=77, renew=0, removed=18446744073709551539, count=77 I0302 14:02:22.564625 1 NodeSync.cpp:178] Communicated tasks=77 to nodes=5 using ms=323 I0302 14:02:32.860934 1 SpecRepo.cpp:369] Updating 181 specs: brandnew=104, renew=0, removed=18446744073709551512, count=181 I0302 14:02:32.984542 1 NodeSync.cpp:178] Communicated tasks=181 to nodes=5 using ms=404
Not sure what happened here, but it took ~13 hours for NebulaServer to update specs in NodeServers
I0302 00:58:42.259829 1 NebulaServer.cpp:495] Nebula server listening on 0.0.0.0:9190 I0302 00:58:52.817811 1 NebulaServer.cpp:560] Loading nebula cluster config: /tmp/nebula.kpu3Kg, old sign: , new sign: 2873_18208199511650023245, size: 2873 I0302 00:59:03.102922 1 ConnectionPool.cpp:64] Creating a channel to 10.240.0.219:9199 I0302 00:59:03.105432 1 ConnectionPool.cpp:64] Creating a channel to 10.240.0.181:9199 I0302 00:59:03.107265 1 ConnectionPool.cpp:64] Creating a channel to 10.240.0.108:9199 I0302 00:59:03.108273 1 ConnectionPool.cpp:64] Creating a channel to 10.240.0.58:9199 I0302 00:59:03.110499 1 ConnectionPool.cpp:64] Creating a channel to 10.240.0.66:9199 I0302 01:31:54.084906 1 ConnectionPool.cpp:64] Creating a channel to 10.240.0.156:9199 E0302 01:32:24.383826 1 NodeClient.cpp:169] RPC failed to 10.240.0.219: code=14, msg=Connect Failed I0302 01:32:24.383872 1 ConnectionPool.cpp:81] Removing a channel to 10.240.0.219:9199 I0302 02:24:42.695111 1 NebulaServer.cpp:594] Nebula Server updated itself 500 times. I0302 02:25:23.819861 1 ConnectionPool.cpp:64] Creating a channel to 10.240.1.56:9199 I0302 03:50:18.518813 1 NebulaServer.cpp:594] Nebula Server updated itself 1000 times. I0302 05:15:55.254267 1 NebulaServer.cpp:594] Nebula Server updated itself 1500 times. I0302 06:41:31.268491 1 NebulaServer.cpp:594] Nebula Server updated itself 2000 times. I0302 08:07:07.238289 1 NebulaServer.cpp:594] Nebula Server updated itself 2500 times. I0302 09:32:44.070459 1 NebulaServer.cpp:594] Nebula Server updated itself 3000 times. I0302 10:58:21.663203 1 NebulaServer.cpp:594] Nebula Server updated itself 3500 times. I0302 12:23:56.903872 1 NebulaServer.cpp:594] Nebula Server updated itself 4000 times. I0302 13:49:32.284674 1 NebulaServer.cpp:594] Nebula Server updated itself 4500 times. I0302 14:02:22.524799 1 SpecRepo.cpp:369] Updating 77 specs: brandnew=77, renew=0, removed=18446744073709551539, count=77 I0302 14:02:22.564625 1 NodeSync.cpp:178] Communicated tasks=77 to nodes=5 using ms=323 I0302 14:02:32.860934 1 SpecRepo.cpp:369] Updating 181 specs: brandnew=104, renew=0, removed=18446744073709551512, count=181 I0302 14:02:32.984542 1 NodeSync.cpp:178] Communicated tasks=181 to nodes=5 using ms=404