tarantool / vshard

The new generation of sharding based on virtual buckets
Other
100 stars 30 forks source link

Test storage/storage.test.lua is flaky #335

Closed ligurio closed 1 year ago

ligurio commented 2 years ago

Test sometimes fails after changes not related to vshard, see log in https://github.com/tarantool/tarantool/runs/6613304486?check_suite_focus=true

Full archive with logs: logs_487018.zip

storage/scheduler.test.lua                                      [ pass ]
storage/storage.test.lua                        memtx           [ pass ]
storage/storage.test.lua                        vinyl           

[Instance "storage_1_a" returns with non-zero exit code: 1]

Last 15 lines of Tarantool Log file [Instance "storage_1_a"][/home/runner/work/tarantool/tarantool/test/var/001_storage/storage_1_a.log]:
2022-05-26 17:02:15.972 [6592] main/103/storage_1_a I> set 'listen' configuration option to "storage:storage@127.0.0.1:3301"
2022-05-26 17:02:15.972 [6592] main/103/storage_1_a I> set 'replication_connect_quorum' configuration option to 0
2022-05-26 17:02:15.972 [6592] main/103/storage_1_a I> set 'replication' configuration option to ["storage@127.0.0.1:3302","storage@127.0.0.1:3301"]
2022-05-26 17:02:15.972 [6592] main/126/applier/storage@127.0.0.1:3302 I> remote master 3de2e3e1-9ebe-4d0d-abb1-26d301b84633 at 127.0.0.1:3302 running Tarantool 2.11.0
2022-05-26 17:02:15.972 [6592] main/[125](https://github.com/tarantool/tarantool/runs/6613304486?check_suite_focus=true#step:7:126)/applier/storage@127.0.0.1:3301 I> remote master 8a274925-a26d-47fc-9e1b-af88ce939412 at 127.0.0.1:3301 running Tarantool 2.11.0
2022-05-26 17:02:15.973 [6592] main/103/storage_1_a I> set 'replicaset_uuid' configuration option to "cbf06940-0790-498b-948d-042b62cf3d29"
2022-05-26 17:02:15.974 [6592] main/125/applier/storage@127.0.0.1:3301 I> leaving orphan mode
2022-05-26 17:02:15.974 [6592] main/[126](https://github.com/tarantool/tarantool/runs/6613304486?check_suite_focus=true#step:7:127)/applier/storage@[127](https://github.com/tarantool/tarantool/runs/6613304486?check_suite_focus=true#step:7:128).0.0.1:3302 I> authenticated
2022-05-26 17:02:15.984 [6592] main I> entering the event loop
2022-05-26 17:02:15.984 [6592] main/126/applier/storage@127.0.0.1:3302 I> subscribed
2022-05-26 17:02:15.984 [6592] main/126/applier/storage@127.0.0.1:3302 I> remote vclock {1: [140](https://github.com/tarantool/tarantool/runs/6613304486?check_suite_focus=true#step:7:141)} local vclock {1: 140}
2022-05-26 17:02:15.985 [6592] main/126/applier/storage@127.0.0.1:3302 I> RAFT: message {term: 1, state: follower} from 2
2022-05-26 17:02:15.985 [6592] main/126/applier/storage@127.0.0.1:3302 I> leaving orphan mode
2022-05-26 17:02:16.025 [6592] main/113/main F> cfg_get('read_only')
2022-05-26 17:02:16.025 [6592] main/113/main F> cfg_get('read_only')
[ fail ]
Reproduce file /home/runner/work/tarantool/tarantool/test/var/reproduce/001_storage.list.yaml
---

I suppose it should be disabled for integration testing with Tarantool or fixed.

Serpentian commented 1 year ago

Fails due to panic on getting read_only on IPROTO_VOTE

Backtrace:

2023-05-15 09:24:43.681 [96156] main/115/applier/storage@127.0.0.1:3302 I> leaving orphan mode
2023-05-15 09:24:43.702 [96156] main/111/main I> #1  0x556401fb3256 in backtrace_collect+114
#2  0x556401f7b4bc in cfg_get+210
#3  0x556401f7b70b in cfg_geti+28
#4  0x556401e6c9ec in box_process_vote(ballot*)+28
#5  0x556401d54ae5 in tx_process_misc(cmsg*)+612
#6  0x556401fb64c1 in cmsg_deliver+52
#7  0x556401fb763a in fiber_pool_f+442
#8  0x556401d49b4a in fiber_cxx_invoke(int (*)(__va_list_tag*), __va_list_tag*)+34
#9  0x556401faf8d9 in fiber_loop+149
#10 0x55640230b8b0 in coro_init+120

2023-05-15 09:24:43.703 [96156] main/111/main I> IS_FUNC: function
2023-05-15 09:24:43.703 [96156] main/111/main I> BUFFER: return box.cfg.read_only
2023-05-15 09:24:43.703 [96156] main/111/main F> cfg_get('read_only')
2023-05-15 09:24:43.703 [96156] main/111/main F> cfg_get('read_only')                                      
Serpentian commented 1 year ago

Blocked with https://github.com/tarantool/tarantool/issues/8655

Serpentian commented 1 year ago

Reproducer:

local server = require('luatest.server')
local msgpack = require('msgpack')
local socket = require('socket')
local t = require('luatest')

local g = t.group()

g.before_all(function(g)
    g.server = server:new{
        alias   = 'server',
    }

    g.server:start()
end)

g.after_all(function(g)
    g.server:drop()
end)

local IPROTO_REQUEST_TYPE = 0x00
local IPROTO_VOTE = 0x44
-- local IPROTO_BALLOT = 0x29

g.test_iproto_vote_panic = function(g)
    g.server:exec(function()
        -- Simulate unconfigured box
        rawset(_G, 'old_box_cfg', box.cfg)
        box.cfg = function(...) return _G.old_box_cfg(...) end
    end)

    local header = msgpack.encode({
        [IPROTO_REQUEST_TYPE] = IPROTO_VOTE
    })

    local len = header:len()
    local s = socket.tcp_connect('unix/', g.server.net_box_uri)

    -- Skip greeting
    local data = s:read(128)
    t.assert_equals(#data, 128)

    -- Send IPROTO_VOTE
    data = '\xce\00\00\00' .. string.char(len) .. header
    t.assert_equals(s:write(data), #data) 

    g.server:exec(function()
        box.cfg = _G.old_box_cfg
    end)
end