github.com/cockroachdb/pebble/internal/metamorphic: TestMeta failed

[cockroach-teamcity]

github.com/cockroachdb/pebble/internal/metamorphic.TestMeta failed with artifacts on refs/heads/master @ e955da9b332e:

        // INFO: [JOB 54] flushed 1 memtable to L0 [000036] (1.8 K), in 0.0s (0.0s total), output rate 1.2 M/s
        // INFO: [JOB 54] WAL deleted 000031
        db.Flush() // <nil> #1681
        db.Merge("mknhohj@6", "funptwfz") // <nil> #1682
        db.DeleteRange("dysmcnuhjwn@2", "wrmqviywssi@2") // <nil> #1686
        db.Set("rvqv@1", "etdzgadkw") // <nil> #1688
        db.Delete("rruegrk") // <nil> #1691
        db.Merge("oyuyotbzeb@6", "pqrrukdgbsgzmewtiho") // <nil> #1695
        iter16.Last() // [true,"ytrfrx",<no point>,["ytrfrx","zgsxfhutlam")=>{"@6"="xjqumhdzrykpkppatth"}] <nil> #1701
        db.Get("ucilgl@1") // [""] pebble: not found #1704
        db.Set("eeuh@6", "nkrndopgwvhui") // <nil> #1706
        snap7.Get("bglaxt@2") // [""] pebble: not found #1707
        batch11 = db.NewIndexedBatch() #1708
        iter16.SeekGE("xanvezhtjby@3", "") // [true,"xanvezhtjby@3",<no point>,["wrmqviywssi","xuggptw")=>{"@6"="xjqumhdzrykpkppatth","@4"="osktptrbkqoyt","@3"="kgsoijcmfriuwzxfnmo","@2"="kgjdsnhgcvajkfusyq","@1"="lelorelrvcid"}*] <nil> #1709
        iter16.Prev("") // [true,"wrmqviywssi",<no point>,["wrmqviywssi","xuggptw")=>{"@6"="xjqumhdzrykpkppatth","@4"="osktptrbkqoyt","@3"="kgsoijcmfriuwzxfnmo","@2"="kgjdsnhgcvajkfusyq","@1"="lelorelrvcid"}] <nil> #1710
        batch11.Merge("csgo@4", "cwupharj") // <nil> #1712
        batch11.Set("tybti@7", "gikl") // <nil> #1714
        batch11.Merge("jfksua@7", "rzogkqnrryx") // <nil> #1718
        snap7.Get("mxaao@2") // [""] pebble: not found #1711
        db.DeleteRange("hdgfqbqzif@1", "ucilgl@1") // <nil> #1715
        // INFO: [JOB 56] ingesting: sstable created 000037
        // INFO: [JOB 57] WAL created 000038
        // INFO: [JOB 58] flushing 1 memtable to L0
        // INFO: [JOB 58] flushing: sstable created 000039
        // INFO: [JOB 58] flushed 1 memtable to L0 [000039] (1.1 K), in 0.0s (0.0s total), output rate 1.2 M/s
        // INFO: [JOB 58] WAL deleted 000035
        // INFO: [JOB 56] ingested L0:000037 (1.0 K)
        db.Ingest(batch11) // <nil> #1721
        db.Get("sbvagefdfxh@4") // [""] pebble: not found #1723
        iter16.SeekPrefixGE("ayxohrbysop@3") // [false] <nil> #1724
        db.Delete("reupvf") // <nil> #1725
        // INFO: [JOB 60] compacting(default) L0 [000036 000039] (2.9 K) + L6 [000033] (6.6 K)
        // INFO: [JOB 60] compacting: sstable created 000040
        iter17 = snap7.NewIter("", "", 2 /* key types */, 0, 0, false /* use L6 filters */, "@2" /* masking suffix */) // <nil> #1716
        db.Delete("lfqjbid@1") // <nil> #1730
        db.Merge("zgsxfhutlam@4", "vcc") // <nil> #1733
        db.Set("rruegrk@1", "vvltiswxlnrfhhjnq") // <nil> #1734
        db.Merge("jdwdrlnbb@1", "dsobrb") // <nil> #1736
        // INFO: [JOB 60] compacted(default) L0 [000036 000039] (2.9 K) + L6 [000033] (6.6 K) -> L6 [000040] (7.7 K), in 0.0s (0.0s total), output rate 430 K/s
        iter17.SeekPrefixGE("vgxdltjef@1") // [true,"vgxdltjef@1",<no point>,["vgxdltjef","vgxdltjef\x00")=>{""="ebnnlyntazwgflep","@4"="osktptrbkqoyt"}*] <nil> #1717
        iter17.SetOptions("", "xuggptw@3", 2 /* key types */, 0, 0, false /* use L6 filters */, "@2" /* masking suffix */) // <nil> #1731
        snap5.Get("hcofnji@4") // [""] pebble: not found #1737
        iter17.Last() // [true,"wrmqviywssi",<no point>,["wrmqviywssi","xuggptw@3")=>{"@4"="osktptrbkqoyt","@3"="kgsoijcmfriuwzxfnmo","@2"="kgjdsnhgcvajkfusyq"}*] <nil> #1732
        iter17.SeekPrefixGE("hqnhz@2") // [false] <nil> #1735
        db.Merge("muevmcv@6", "plvz") // <nil> #1738
        db.Get("cojvidjyw@2") // [""] pebble: not found #1740
        db.DeleteRange("mknhohj@6", "sbvagefdfxh@4") // <nil> #1741
        iter16.Next("") // [false] <nil> #1744
        snap5.Get("oufajim@4") // [""] pebble: not found #1742
        --- FAIL: TestMeta/execution/standard-000 (1.18s)

Help

To reproduce, try: ```bash go test -tags 'invariants' -exec 'stress -p 1' -timeout 0 -test.v -run TestMeta$ ./internal/metamorphic -seed 1682657602877739979 -ops "uniform:5000-10000" ```

This test on roachdash | Improve this report!

[nicktrav]

        ===== SEED =====
        1682657602877739979
        ===== ERR =====
        exit status 2
        ===== OUT =====
        unexpected fault address 0x200004d9f000
        fatal error: fault
        runtime: g 197: unexpected return pc for runtime.sigpanic called from 0x0
        stack: frame={sp:0x7ffc48d0e508, fp:0x7ffc48d0e558} stack=[0xc0004f8000,0xc0004fc000)

        fatal error: unknown caller pc

        runtime stack:
        runtime.throw({0x111fac5?, 0x172e420?})
            GOROOT/src/runtime/panic.go:1047 +0x5d fp=0x7ffc48d0e210 sp=0x7ffc48d0e1e0 pc=0x455f3d
        runtime.gentraceback(0x4278b3?, 0x26c4808?, 0x7ffc48d0e5b8?, 0x7ffc48d0e5b8?, 0x0, 0x0, 0x7fffffff, 0x7ffc48d0e5d0, 0x11188b4?, 0x0)
            GOROOT/src/runtime/traceback.go:269 +0x1cf7 fp=0x7ffc48d0e580 sp=0x7ffc48d0e210 pc=0x47f597
        runtime.addOneOpenDeferFrame.func1()
            GOROOT/src/runtime/panic.go:645 +0x6b fp=0x7ffc48d0e5f8 sp=0x7ffc48d0e580 pc=0x4550cb
        runtime.systemstack()
            src/runtime/asm_amd64.s:492 +0x49 fp=0x7ffc48d0e600 sp=0x7ffc48d0e5f8 pc=0x48a089

        goroutine 197 [running]:
        runtime: g 197: unexpected return pc for runtime.systemstack_switch called from 0x0
        stack: frame={sp:0x7ffc48d0e400, fp:0x7ffc48d0e408} stack=[0xc0004f8000,0xc0004fc000)

        runtime.systemstack_switch()
            src/runtime/asm_amd64.s:459 fp=0x7ffc48d0e408 sp=0x7ffc48d0e400 pc=0x48a020
        created by github.com/cockroachdb/pebble.(*DB).maybeCollectTableStatsLocked
            github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_stats.go:46 +0xa5

That smells more like a runtime issue.

[nicktrav]

Side node: the reproduction command isn't accurate. It needs the -trace flag too.

$ go test -race -tags 'invariants' -exec 'stress -p 1' -timeout 0 -test.v -run TestMeta$ ./internal/metamorphic -seed 1682657602877739979 -ops "uniform:5000-10000"

[nicktrav]

Was trying to stress this on my Linux box with the same Go version and same SHA, but after 2 hours, nothing.

I did find this issue in Go, which is similar.