search

tarantool / cartridge

Out-of-the-box cluster manager for Tarantool with a modern web UI
https://www.tarantool.io/en/cartridge/
BSD 2-Clause "Simplified" License
94 stars 30 forks source link

Flaky test integration.failover_eventual.test_all_rw_failover #1715

Closed yngvar-antonsson closed 2 years ago

yngvar-antonsson commented 2 years ago

Tests with errors: 408

409

410 1) integration.failover_eventual.test_all_rw_failover 411 ...ge/cartridge/test/integration/failover_eventual_test.lua:88: "localhost:13303": Instance state is ConfiguringRoles, can't apply config in this state 412 stack traceback: 413 ...rk/cartridge/cartridge/cartridge/test-helpers/server.lua:194: in function 'graphql' 414 ...ge/cartridge/test/integration/failover_eventual_test.lua:88: in function 'set_all_rw' 415 ...ge/cartridge/test/integration/failover_eventual_test.lua:603: in function 'integration.failover_eventual.test_all_rw_failover' 416 ... 417 [C]: in function 'xpcall' 418

419 Captured stdout: 420 router-1 | 2022-01-21 20:46:45.149 [4260] main/235/http/127.0.0.1:41508 twophase.lua:533 W> Updating config clusterwide... 421 router-1 | 2022-01-21 20:46:45.150 [4260] main/235/http/127.0.0.1:41508 twophase.lua:409 W> (2PC) patch_clusterwide upload phase... 422 router-1 | 2022-01-21 20:46:45.155 [4260] main/235/http/127.0.0.1:41508 twophase.lua:422 W> (2PC) patch_clusterwide prepare phase... 423 router-1 | 2022-01-21 20:46:45.159 [4260] main/235/http/127.0.0.1:41508 twophase.lua:431 W> Prepared for patch_clusterwide at localhost:13301 424 router-1 | 2022-01-21 20:46:45.159 [4260] main/235/http/127.0.0.1:41508 twophase.lua:431 W> Prepared for patch_clusterwide at localhost:13302 425 router-1 | 2022-01-21 20:46:45.159 [4260] main/235/http/127.0.0.1:41508 twophase.lua:431 W> Prepared for patch_clusterwide at localhost:13303 426 router-1 | 2022-01-21 20:46:45.159 [4260] main/235/http/127.0.0.1:41508 twophase.lua:431 W> Prepared for patch_clusterwide at localhost:13304 427 router-1 | 2022-01-21 20:46:45.159 [4260] main/235/http/127.0.0.1:41508 twophase.lua:455 W> (2PC) patch_clusterwide commit phase... 428 storage-1 | 2022-01-21 20:46:45.160 [4384] main/116/main I> Backup of active config created: "/tmp/tmp.cartridge.s2xzk_W8Ml-r/localhost-13302/config.backup" 429 storage-1 | 2022-01-21 20:46:45.160 [4384] main/116/main I> Instance state changed: RolesConfigured -> ConfiguringRoles 430 storage-1 | 2022-01-21 20:46:45.161 [4384] main/116/main I> connecting to 3 replicas 431 storage-1 | 2022-01-21 20:46:45.162 [4384] main/116/main C> failed to connect to 3 out of 3 replicas 432 router-1 | 2022-01-21 20:46:45.163 [4260] main/213/main I> Backup of active config created: "/tmp/tmp.cartridge.s2xzk_W8Ml-r/localhost-13301/config.backup" 433 router-1 | 2022-01-21 20:46:45.164 [4260] main/213/main I> Instance state changed: RolesConfigured -> ConfiguringRoles 434 router-1 | 2022-01-21 20:46:45.164 [4260] main/213/main I> Eventual failover enabled 435 router-1 | 2022-01-21 20:46:45.164 [4260] main/213/main I> --- apply_config({is_master = true}) 436 router-1 | 2022-01-21 20:46:45.164 [4260] main/213/main I> Roles configuration finished 437 router-1 | 2022-01-21 20:46:45.164 [4260] main/213/main I> Instance state changed: ConfiguringRoles -> RolesConfigured 438 storage-3 | 2022-01-21 20:46:45.164 [4284] main/155/main I> Backup of active config created: "/tmp/tmp.cartridge.s2xzk_W8Ml-r/localhost-13304/config.backup" 439 storage-3 | 2022-01-21 20:46:45.165 [4284] main/155/main I> Instance state changed: RolesConfigured -> ConfiguringRoles 440 storage-3 | 2022-01-21 20:46:45.165 [4284] main/155/main I> connecting to 3 replicas 441 storage-3 | 2022-01-21 20:46:45.165 [4284] main/155/main C> failed to connect to 3 out of 3 replicas 442 storage-2 | 2022-01-21 20:46:45.168 [4276] relay/127.0.0.1:45704/101/main coio.cc:379 !> SystemError unexpected EOF when reading from socket, called on fd 484, aka 127.0.0.1:13303, peer of 127.0.0.1:45704: Broken pipe 443 storage-2 | 2022-01-21 20:46:45.168 [4276] relay/127.0.0.1:45704/101/main C> exiting the relay loop 444 storage-2 | 2022-01-21 20:46:45.168 [4276] main/122/main I> Backup of active config created: "/tmp/tmp.cartridge.s2xzk_W8Ml-r/localhost-13303/config.backup" 445 storage-1 | 2022-01-21 20:46:45.168 [4384] main/116/main C> leaving orphan mode 446 storage-1 | 2022-01-21 20:46:45.168 [4384] main/116/main I> set 'replication' configuration option to ["admin@localhost:13302","admin@localhost:13303","admin@localhost:13304"] 447 storage-1 | 2022-01-21 20:46:45.168 [4384] main/116/main I> Eventual failover enabled 448 storage-3 | 2022-01-21 20:46:45.169 [4284] main/155/main C> leaving orphan mode 449 storage-3 | 2022-01-21 20:46:45.169 [4284] main/155/main I> set 'replication' configuration option to ["admin@localhost:13302","admin@localhost:13303","admin@localhost:13304"] 450 storage-3 | 2022-01-21 20:46:45.169 [4284] main/155/main I> Eventual failover enabled 1151 storage-2 | stack traceback: 1152 storage-2 | ...e/runner/work/cartridge/cartridge/cartridge/twophase.lua:116: in function <...e/runner/work/cartridge/cartridge/cartridge/twophase.lua:95> 1153 storage-2 | [C]: in function 'xpcall' 1154 storage-2 | ...rk/cartridge/cartridge/.rocks/share/tarantool/errors.lua:145: in function <...rk/cartridge/cartridge/.rocks/share/tarantool/errors.lua:139> 1155 storage-2 | [C]: at 0x004fd2d0 1156 storage-2 | 2022-01-21 20:46:45.769 [4276] main/150/localhost:13302 (net.box) I> connected to localhost:13302 1157 storage-2 | 2022-01-21 20:46:45.769 [4276] main/303/lua I> Old replicaset and replica objects are outdated. 1158 storage-2 | 2022-01-21 20:46:45.770 [4276] main/301/cartridge.failover.task I> --- apply_config({is_master = false}) 1159 storage-2 | 2022-01-21 20:46:45.770 [4276] main/301/cartridge.failover.task I> Failover step finished 1160 storage-2 | 2022-01-21 20:46:45.770 [4276] main/301/cartridge.failover.task I> Instance state changed: ConfiguringRoles -> RolesConfigured 1161 router-1 | 2022-01-21 20:46:45.773 [4260] main/249/http/127.0.0.1:41708 twophase.lua:431 W> Prepared for patch_clusterwide at localhost:13301 1162 router-1 | 2022-01-21 20:46:45.773 [4260] main/249/http/127.0.0.1:41708 twophase.lua:431 W> Prepared for patch_clusterwide at localhost:13302 1163 router-1 | 2022-01-21 20:46:45.773 [4260] main/249/http/127.0.0.1:41708 twophase.lua:431 W> Prepared for patch_clusterwide at localhost:13304 1164 router-1 | 2022-01-21 20:46:45.773 [4260] main/249/http/127.0.0.1:41708 twophase.lua:441 E> Error preparing for patch_clusterwide at localhost:13303: 1165 router-1 | Prepare2pcError: "localhost:13303": Instance state is ConfiguringRoles, can't apply config in this state 1166 router-1 | stack traceback: 1167 router-1 | ...e/runner/work/cartridge/cartridge/cartridge/twophase.lua:116: in function <...e/runner/work/cartridge/cartridge/cartridge/twophase.lua:95> 1168 router-1 | [C]: in function 'xpcall' 1169 router-1 | ...rk/cartridge/cartridge/.rocks/share/tarantool/errors.lua:145: in function <...rk/cartridge/cartridge/.rocks/share/tarantool/errors.lua:139> 1170 router-1 | [C]: at 0x004fd2d0 1171 router-1 | during async net.box call to localhost:13303, function "_G.__cartridge_clusterwide_config_prepare_2pc" 1172 router-1 | stack traceback: 1173 router-1 | /home/runner/work/cartridge/cartridge/cartridge/pool.lua:248: in function 'map_call' 1174 router-1 | ...e/runner/work/cartridge/cartridge/cartridge/twophase.lua:424: in function 'twophase_commit' 1175 router-1 | ...e/runner/work/cartridge/cartridge/cartridge/twophase.lua:599: in function <...e/runner/work/cartridge/cartridge/cartridge/twophase.lua:525> 1176 router-1 | [C]: in function 'xpcall' 1177 router-1 | ...rk/cartridge/cartridge/.rocks/share/tarantool/errors.lua:145: in function 'pcall' 1178 router-1 | ...e/runner/work/cartridge/cartridge/cartridge/twophase.lua:625: in function 'patch_clusterwide' 1179 router-1 | .../cartridge/cartridge/cartridge/lua-api/edit-topology.lua:361: in function 'edit_topology' 1180 router-1 | ...ork/cartridge/cartridge/cartridge/lua-api/deprecated.lua:220: in function 'call' 1181 router-1 | /home/runner/work/cartridge/cartridge/cartridge/graphql.lua:38: in function 'defaultResolver' 1182 router-1 | ...dge/cartridge/.rocks/share/tarantool/graphql/execute.lua:341: in function 'getFieldEntry' 1183 router-1 | ... 1184 router-1 | /home/runner/work/cartridge/cartridge/cartridge/graphql.lua:300: in function </home/runner/work/cartridge/cartridge/cartridge/graphql.lua:226> 1185 router-1 | [C]: in function 'xpcall' 1186 router-1 | ...rk/cartridge/cartridge/.rocks/share/tarantool/errors.lua:145: in function 'pcall' 1187 router-1 | /home/runner/work/cartridge/cartridge/cartridge/graphql.lua:335: in function 'sub' 1188 router-1 | ...rtridge/cartridge/.rocks/share/tarantool/http/server.lua:717: in function <...rtridge/cartridge/.rocks/share/tarantool/http/server.lua:695> 1189 router-1 | [C]: in function 'pcall' 1190 router-1 | ...rtridge/cartridge/.rocks/share/tarantool/http/server.lua:795: in function 'process_client' 1191 router-1 | ...rtridge/cartridge/.rocks/share/tarantool/http/server.lua:1262: in function <...rtridge/cartridge/.rocks/share/tarantool/http/server.lua:1261> 1192 router-1 | [C]: in function 'pcall' 1193 router-1 | builtin/socket.lua:1081: in function <builtin/socket.lua:1079> 1194 router-1 | 2022-01-21 20:46:45.773 [4260] main/249/http/127.0.0.1:41708 twophase.lua:480 W> (2PC) patch_clusterwide abort phase... 1195 router-1 | 2022-01-21 20:46:45.774 [4260] main/249/http/127.0.0.1:41708 twophase.lua:489 W> Aborted patch_clusterwide at localhost:13301

yngvar-antonsson commented 2 years ago

Closed until #1687 would be merged.