search

matrixorigin / matrixone

Hyperconverged cloud-edge native database
https://docs.matrixorigin.cn/en
Apache License 2.0
1.79k stars 277 forks source link

[Bug]: ut fails in TestDelete3 #18521

Open badboynt1 opened 2 months ago

badboynt1 commented 2 months ago

Is there an existing issue for the same bug?

Branch Name

main

Commit ID

lastest

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

================== 2024-09-04T07:00:14.0939819Z WARNING: DATA RACE 2024-09-04T07:00:14.0939976Z Write at 0x00c003849eb8 by goroutine 4307: 2024-09-04T07:00:14.0940099Z runtime.slicecopy() 2024-09-04T07:00:14.0940448Z /opt/hostedtoolcache/go/1.23.0/x64/src/runtime/slice.go:355 +0x0 2024-09-04T07:00:14.0940828Z github.com/matrixorigin/matrixone/pkg/container/vector.(Vector).CloneWindowTo() 2024-09-04T07:00:14.0941288Z /home/runner/work/matrixone/matrixone/pkg/container/vector/vector.go:3459 +0x3b2 2024-09-04T07:00:14.0941802Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/containers.(vectorWrapper).CloneWindowWithPool() 2024-09-04T07:00:14.0942291Z /home/runner/work/matrixone/matrixone/pkg/vm/engine/tae/containers/vector.go:406 +0xce 2024-09-04T07:00:14.0942753Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tables.(memoryNode).getDataWindowLocked() 2024-09-04T07:00:14.0943215Z /home/runner/work/matrixone/matrixone/pkg/vm/engine/tae/tables/mnode.go:221 +0x683 2024-09-04T07:00:14.0943578Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tables.(memoryNode).Scan() 2024-09-04T07:00:14.0944020Z /home/runner/work/matrixone/matrixone/pkg/vm/engine/tae/tables/mnode.go:321 +0x24b 2024-09-04T07:00:14.0944367Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tables.(baseObject).Scan() 2024-09-04T07:00:14.0944880Z /home/runner/work/matrixone/matrixone/pkg/vm/engine/tae/tables/base.go:455 +0x1d4 2024-09-04T07:00:14.0945276Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tables.(aobject).Scan() 2024-09-04T07:00:14.0945429Z :1 +0x108 2024-09-04T07:00:14.0945807Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tables.HybridScanByBlock() 2024-09-04T07:00:14.0946291Z /home/runner/work/matrixone/matrixone/pkg/vm/engine/tae/tables/table_scan.go:43 +0x1c2 2024-09-04T07:00:14.0946718Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/txn/txnimpl.(txnObject).HybridScan() 2024-09-04T07:00:14.0947214Z /home/runner/work/matrixone/matrixone/pkg/vm/engine/tae/txn/txnimpl/object.go:253 +0x3b7 2024-09-04T07:00:14.0947769Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tables/jobs.(flushTableTailTask).prepareAObjSortedData() 2024-09-04T07:00:14.0948354Z /home/runner/work/matrixone/matrixone/pkg/vm/engine/tae/tables/jobs/flushTableTail.go:500 +0x283 2024-09-04T07:00:14.0948835Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tables/jobs.(flushTableTailTask).mergeAObjs() 2024-09-04T07:00:14.0949423Z /home/runner/work/matrixone/matrixone/pkg/vm/engine/tae/tables/jobs/flushTableTail.go:608 +0xb44 2024-09-04T07:00:14.0949881Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tables/jobs.(flushTableTailTask).Execute() 2024-09-04T07:00:14.0950631Z /home/runner/work/matrixone/matrixone/pkg/vm/engine/tae/tables/jobs/flushTableTail.go:356 +0x9b0 2024-09-04T07:00:14.0951064Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tasks/ops.(Op).OnExec() 2024-09-04T07:00:14.0951565Z /home/runner/work/matrixone/matrixone/pkg/vm/engine/tae/tasks/ops/ops.go:103 +0xe1 2024-09-04T07:00:14.0952013Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tables/jobs.(flushTableTailTask).OnExec() 2024-09-04T07:00:14.0952165Z :1 +0x5d 2024-09-04T07:00:14.0952621Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/db.(ScheduledTxnTask).Execute() 2024-09-04T07:00:14.0953047Z /home/runner/work/matrixone/matrixone/pkg/vm/engine/tae/db/task.go:64 +0x317 2024-09-04T07:00:14.0953384Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tasks/ops.(Op).OnExec() 2024-09-04T07:00:14.0953849Z /home/runner/work/matrixone/matrixone/pkg/vm/engine/tae/tasks/ops/ops.go:103 +0xe1 2024-09-04T07:00:14.0954223Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/db.(ScheduledTxnTask).OnExec() 2024-09-04T07:00:14.0954376Z :1 +0x5d 2024-09-04T07:00:14.0954756Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tasks/worker.(OpWorker).onOp() 2024-09-04T07:00:14.0955285Z /home/runner/work/matrixone/matrixone/pkg/vm/engine/tae/tasks/worker/worker.go:210 +0x58 2024-09-04T07:00:14.0955763Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tasks/worker.(OpWorker).onOp-fm() 2024-09-04T07:00:14.0955907Z :1 +0x47 2024-09-04T07:00:14.0956500Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tasks.(poolHandler).doHandle.(poolHandler).doHandle.func1.func2() 2024-09-04T07:00:14.0956986Z /home/runner/work/matrixone/matrixone/pkg/vm/engine/tae/tasks/poolhandler.go:60 +0x66 2024-09-04T07:00:14.0957199Z github.com/panjf2000/ants/v2.(goWorker).run.func1() 2024-09-04T07:00:14.0957622Z /home/runner/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.4/worker.go:67 +0xdb 2024-09-04T07:00:14.0957633Z 2024-09-04T07:00:14.0957816Z Previous read at 0x00c003849ebc by goroutine 4243: 2024-09-04T07:00:14.0958125Z github.com/matrixorigin/matrixone/pkg/container/types.(TS).Compare() 2024-09-04T07:00:14.0958561Z /home/runner/work/matrixone/matrixone/pkg/container/types/txnts.go:54 +0x12f9 2024-09-04T07:00:14.0958866Z github.com/matrixorigin/matrixone/pkg/container/types.(TS).Greater() 2024-09-04T07:00:14.0959295Z /home/runner/work/matrixone/matrixone/pkg/container/types/txnts.go:80 +0x134c 2024-09-04T07:00:14.0959857Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tables.(persistedNode).FillBlockTombstones() 2024-09-04T07:00:14.0960384Z /home/runner/work/matrixone/matrixone/pkg/vm/engine/tae/tables/pnode.go:329 +0xf05 2024-09-04T07:00:14.0960838Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tables.(baseObject).FillBlockTombstones() 2024-09-04T07:00:14.0961284Z /home/runner/work/matrixone/matrixone/pkg/vm/engine/tae/tables/base.go:469 +0x1d4 2024-09-04T07:00:14.0961720Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tables.(aobject).FillBlockTombstones() 2024-09-04T07:00:14.0961866Z :1 +0xa4 2024-09-04T07:00:14.0962233Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tables.HybridScanByBlock() 2024-09-04T07:00:14.0962715Z /home/runner/work/matrixone/matrixone/pkg/vm/engine/tae/tables/table_scan.go:56 +0x3b2 2024-09-04T07:00:14.0963129Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/txn/txnimpl.(txnObject).HybridScan() 2024-09-04T07:00:14.0963624Z /home/runner/work/matrixone/matrixone/pkg/vm/engine/tae/txn/txnimpl/object.go:253 +0x3b7 2024-09-04T07:00:14.0964058Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/db/testutil.ForEachColumnView.func1() 2024-09-04T07:00:14.0964545Z /home/runner/work/matrixone/matrixone/pkg/vm/engine/tae/db/testutil/funcs.go:238 +0x223 2024-09-04T07:00:14.0964915Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/db/testutil.forEachObject() 2024-09-04T07:00:14.0965404Z /home/runner/work/matrixone/matrixone/pkg/vm/engine/tae/db/testutil/funcs.go:270 +0x134 2024-09-04T07:00:14.0965833Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/db/testutil.ForEachObject() 2024-09-04T07:00:14.0966320Z /home/runner/work/matrixone/matrixone/pkg/vm/engine/tae/db/testutil/funcs.go:258 +0xb1 2024-09-04T07:00:14.0966714Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/db/testutil.ForEachColumnView() 2024-09-04T07:00:14.0967197Z /home/runner/work/matrixone/matrixone/pkg/vm/engine/tae/db/testutil/funcs.go:234 +0x35 2024-09-04T07:00:14.0967661Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/db/testutil.GetColumnRowsByScan() 2024-09-04T07:00:14.0968148Z /home/runner/work/matrixone/matrixone/pkg/vm/engine/tae/db/testutil/funcs.go:222 +0xb8 2024-09-04T07:00:14.0968561Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/db/testutil.CheckAllColRowsByScan() 2024-09-04T07:00:14.0969052Z /home/runner/work/matrixone/matrixone/pkg/vm/engine/tae/db/testutil/funcs.go:215 +0x104 2024-09-04T07:00:14.0969507Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/db/testutil.(TestEngine).CheckRowsByScan() 2024-09-04T07:00:14.0970006Z /home/runner/work/matrixone/matrixone/pkg/vm/engine/tae/db/testutil/engine.go:136 +0x164 2024-09-04T07:00:14.0970331Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/db/test.TestDelete3() 2024-09-04T07:00:14.0970811Z /home/runner/work/matrixone/matrixone/pkg/vm/engine/tae/db/test/db_test.go:3382 +0x5b4 2024-09-04T07:00:14.0970926Z testing.tRunner() 2024-09-04T07:00:14.0971289Z /opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:1690 +0x226 2024-09-04T07:00:14.0971421Z testing.(T).Run.gowrap1() 2024-09-04T07:00:14.0971782Z /opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:1743 +0x44 2024-09-04T07:00:14.0971789Z 2024-09-04T07:00:14.0971920Z Goroutine 4307 (running) created at: 2024-09-04T07:00:14.0972111Z github.com/panjf2000/ants/v2.(goWorker).run() 2024-09-04T07:00:14.0972540Z /home/runner/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.4/worker.go:48 +0xb1 2024-09-04T07:00:14.0972789Z github.com/panjf2000/ants/v2.(Pool).retrieveWorker.func1() 2024-09-04T07:00:14.0973206Z /home/runner/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.4/pool.go:337 +0xa7 2024-09-04T07:00:14.0973416Z github.com/panjf2000/ants/v2.(Pool).retrieveWorker() 2024-09-04T07:00:14.0973843Z /home/runner/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.4/pool.go:348 +0x198 2024-09-04T07:00:14.0974019Z github.com/panjf2000/ants/v2.(Pool).Submit() 2024-09-04T07:00:14.0974506Z /home/runner/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.4/pool.go:222 +0x47 2024-09-04T07:00:14.0974939Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tasks.(poolHandler).doHandle() 2024-09-04T07:00:14.0975432Z /home/runner/work/matrixone/matrixone/pkg/vm/engine/tae/tasks/poolhandler.go:65 +0x1e4 2024-09-04T07:00:14.0975909Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tasks.(poolHandler).doHandle-fm() 2024-09-04T07:00:14.0976061Z :1 +0x47 2024-09-04T07:00:14.0976482Z github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tasks/worker.(OpWorker).Start.func1() 2024-09-04T07:00:14.0977004Z /home/runner/work/matrixone/matrixone/pkg/vm/engine/tae/tasks/worker/worker.go:142 +0x16a 2024-09-04T07:00:14.0977010Z 2024-09-04T07:00:14.0977139Z Goroutine 4243 (running) created at: 2024-09-04T07:00:14.0977255Z testing.(T).Run() 2024-09-04T07:00:14.0977616Z /opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:1743 +0x825 2024-09-04T07:00:14.0977754Z testing.runTests.func1() 2024-09-04T07:00:14.0978103Z /opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:2168 +0x85 2024-09-04T07:00:14.0978223Z testing.tRunner() 2024-09-04T07:00:14.0978580Z /opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:1690 +0x226 2024-09-04T07:00:14.0978696Z testing.runTests() 2024-09-04T07:00:14.0979046Z /opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:2166 +0x8be 2024-09-04T07:00:14.0979155Z testing.(*M).Run() 2024-09-04T07:00:14.0979498Z /opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:2034 +0xf17 2024-09-04T07:00:14.0979666Z main.main() 2024-09-04T07:00:14.0979835Z _testmain.go:431 +0x164 2024-09-04T07:00:14.0979928Z ==================

Expected Behavior

No response

Steps to Reproduce

https://github.com/matrixorigin/matrixone/actions/runs/10696401422/job/29651524858?pr=18514

Additional information

No response

XuPeng-SH commented 2 months ago

fixed

jensenojs commented 1 month ago

repro : https://github.com/matrixorigin/matrixone/actions/runs/11048889917/job/30693268409?pr=19031

image
sukki37 commented 1 month ago

repro：https://github.com/matrixorigin/matrixone/actions/runs/11107305179/job/30857596327?pr=18969 log： job-logs.txt.zip

2024-09-30T14:14:43.6778930Z 2024/09/30 13:56:05.662700 +0000 INFO dbutils/mem.go:107 HeapInfo:TotalAlloc:2108MB Sys:175MB HeapAlloc:83MB HeapSys:165MB HeapIdle:76MB HeapReleased:37MB HeapInuse:88MB NextGC:130MB NumGC:38 PauseNs:70557355
2024-09-30T14:14:43.6779310Z 2024/09/30 13:56:05.663311 +0000 INFO catalog/catalog.go:124 GC Catalog 1727704565184466794-0
2024-09-30T14:14:43.6779532Z     db_test.go:3476: <Empty Tree>
2024-09-30T14:14:43.6779816Z 2024/09/30 13:56:05.666618 +0000 INFO v2/diskcleaner.go:55 JobFactory is start
2024-09-30T14:14:43.6780104Z 2024/09/30 13:56:05.666712 +0000 INFO db/open.go:313 LogtailStats: Compact[0|0]
2024-09-30T14:14:43.6780467Z 2024/09/30 13:56:05.666779 +0000 INFO catalog/catalog.go:124 GC Catalog 1727704565184466794-0
2024-09-30T14:14:43.6780608Z     engine.go:206: 
2024-09-30T14:14:43.6781224Z            Error Trace:    /home/runner/work/matrixone/matrixone/pkg/vm/engine/tae/db/testutil/engine.go:206
2024-09-30T14:14:43.6782036Z                                        /home/runner/work/matrixone/matrixone/pkg/vm/engine/tae/db/test/db_test.go:3494
2024-09-30T14:14:43.6782269Z            Error:          Received unexpected error:
2024-09-30T14:14:43.6782527Z                            w-w conflict
2024-09-30T14:14:43.6782695Z            Test:           TestDelete3
XuPeng-SH commented 1 month ago

@jiangxinmeng1

jiangxinmeng1 commented 3 weeks ago

还没进展

jiangxinmeng1 commented 2 weeks ago

fixed by #19774

badboynt1 commented 4 days ago

repro in https://github.com/matrixorigin/matrixone/actions/runs/11908023271/job/33182708184?pr=20169

w-zr commented 1 day ago

还在尝试复现