search

matrixorigin / matrixone

Hyperconverged cloud-edge native database
https://docs.matrixorigin.cn/en
Apache License 2.0
1.79k stars 276 forks source link

[Bug]: data race in CI (in mysql_buffer.go) #18880

Closed ouyuanning closed 1 month ago

ouyuanning commented 1 month ago

Is there an existing issue for the same bug?

Branch Name

main

Commit ID

main

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

https://github.com/matrixorigin/matrixone/actions/runs/10952357425/job/30410915911?pr=18650

2024/09/20 03:02:06.132900 +0000 INFO logutil/action.go:35 start to close components {"service": "1-cn-3", "action": "close", "components": "cnservice/frontend"}
==================
WARNING: DATA RACE
Write at 0x00c0a275fe00 by goroutine 8066:
  github.com/matrixorigin/matrixone/pkg/frontend.(*Conn).Close.func1.1()
      /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_buffer.go:153 +0x172
  runtime.deferreturn()
      /opt/hostedtoolcache/go/1.23.0/x64/src/runtime/panic.go:605 +0x5d
  sync.(*Once).doSlow()
      /opt/hostedtoolcache/go/1.23.0/x64/src/sync/once.go:76 +0xe1
  sync.(*Once).Do()
      /opt/hostedtoolcache/go/1.23.0/x64/src/sync/once.go:67 +0x44
  github.com/matrixorigin/matrixone/pkg/frontend.(*Conn).Close()
      /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_buffer.go:148 +0x8f
  github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).Stop()
      /home/runner/work/matrixone/matrixone/pkg/frontend/server.go:117 +0x38b
  github.com/matrixorigin/matrixone/pkg/cnservice.(*service).serverShutdown()
      /home/runner/work/matrixone/matrixone/pkg/cnservice/server.go:516 +0xb1
  github.com/matrixorigin/matrixone/pkg/cnservice.(*service).stopFrontend()
      /home/runner/work/matrixone/matrixone/pkg/cnservice/server.go:345 +0xb2
  github.com/matrixorigin/matrixone/pkg/cnservice.(*service).Close()
      /home/runner/work/matrixone/matrixone/pkg/cnservice/server.go:263 +0xfb
  github.com/matrixorigin/matrixone/pkg/embed.(*operator).Close()
      /home/runner/work/matrixone/matrixone/pkg/embed/operator.go:129 +0xf0
  github.com/matrixorigin/matrixone/pkg/embed.(*cluster).Close()
      /home/runner/work/matrixone/matrixone/pkg/embed/cluster.go:145 +0x166
  github.com/matrixorigin/matrixone/pkg/embed.TestMultiClusterCanWork()
      /home/runner/work/matrixone/matrixone/pkg/embed/cluster_test.go:61 +0xc2
  testing.tRunner()
      /opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:1743 +0x44

Previous read at 0x00c0a275fe00 by goroutine 13452:
  github.com/matrixorigin/matrixone/pkg/frontend.(*Conn).BeginPacket()
      /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_buffer.go:546 +0x51
  github.com/matrixorigin/matrixone/pkg/frontend.(*MysqlProtocolImpl).beginPacket()
      /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_protocol.go:2733 +0x56
  github.com/matrixorigin/matrixone/pkg/frontend.(*MysqlProtocolImpl).appendPacket()
      /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_protocol.go:2743 +0x40
  github.com/matrixorigin/matrixone/pkg/frontend.(*MysqlProtocolImpl).SendColumnDefinitionPacket()
      /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_protocol.go:2175 +0xec
  github.com/matrixorigin/matrixone/pkg/frontend.(*MysqlProtocolImpl).WriteColumnDef()
      /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_protocol.go:437 +0x58
  github.com/matrixorigin/matrixone/pkg/frontend.(*MysqlResp).respColumnDefsWithoutFlush()
      /home/runner/work/matrixone/matrixone/pkg/frontend/result_row_stmt.go:239 +0x4e2
  github.com/matrixorigin/matrixone/pkg/frontend.(*MysqlResp).RespPreMeta()
      /home/runner/work/matrixone/matrixone/pkg/frontend/resp_client.go:120 +0xc5
  github.com/matrixorigin/matrixone/pkg/frontend.executeResultRowStmt()
      /home/runner/work/matrixone/matrixone/pkg/frontend/result_row_stmt.go:51 +0x5cf
  github.com/matrixorigin/matrixone/pkg/frontend.executeStmt()
      /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_cmd_executor.go:2695 +0x1912
  github.com/matrixorigin/matrixone/pkg/frontend.dispatchStmt()
      /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_cmd_executor.go:2558 +0x4a8
  github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithIncrStmt()
      /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_cmd_executor.go:2527 +0x3cc
  github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithWorkspace()
      /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_cmd_executor.go:2484 +0x966

Expected Behavior

No response

Steps to Reproduce

https://github.com/matrixorigin/matrixone/actions/runs/10952357425/job/30410915911?pr=18650

Additional information

No response

ouyuanning commented 1 month ago

job-logs (7).txt.zip log

sukki37 commented 1 month ago

repro：https://github.com/matrixorigin/matrixone/actions/runs/10953064595/job/30412774046?pr=18882

log： job-logs.txt.zip

2024-09-20T04:29:29.7900908Z ==================
2024-09-20T04:29:29.7901355Z WARNING: DATA RACE
2024-09-20T04:29:29.7902002Z Write at 0x00c083fed110 by goroutine 2497:
2024-09-20T04:29:29.7903074Z   github.com/matrixorigin/matrixone/pkg/frontend.(*Conn).Close.func1.1()
2024-09-20T04:29:29.7904571Z       /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_buffer.go:153 +0x172
2024-09-20T04:29:29.7905598Z   runtime.deferreturn()
2024-09-20T04:29:29.7906570Z       /opt/hostedtoolcache/go/1.23.0/x64/src/runtime/panic.go:605 +0x5d
2024-09-20T04:29:29.7907459Z   sync.(*Once).doSlow()
2024-09-20T04:29:29.7908320Z       /opt/hostedtoolcache/go/1.23.0/x64/src/sync/once.go:76 +0xe1
2024-09-20T04:29:29.7909318Z   sync.(*Once).Do()
2024-09-20T04:29:29.7910151Z       /opt/hostedtoolcache/go/1.23.0/x64/src/sync/once.go:67 +0x44
2024-09-20T04:29:29.7911238Z   github.com/matrixorigin/matrixone/pkg/frontend.(*Conn).Close()
2024-09-20T04:29:29.7912666Z       /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_buffer.go:148 +0x8f
2024-09-20T04:29:29.7913962Z   github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).Stop()
2024-09-20T04:29:29.7915367Z       /home/runner/work/matrixone/matrixone/pkg/frontend/server.go:117 +0x38b
2024-09-20T04:29:29.7916721Z   github.com/matrixorigin/matrixone/pkg/cnservice.(*service).serverShutdown()
2024-09-20T04:29:29.7918239Z       /home/runner/work/matrixone/matrixone/pkg/cnservice/server.go:516 +0xb1
2024-09-20T04:29:29.7919786Z   github.com/matrixorigin/matrixone/pkg/cnservice.(*service).stopFrontend()
2024-09-20T04:29:29.7921253Z       /home/runner/work/matrixone/matrixone/pkg/cnservice/server.go:345 +0xb2
2024-09-20T04:29:29.7922508Z   github.com/matrixorigin/matrixone/pkg/cnservice.(*service).Close()
2024-09-20T04:29:29.7923928Z       /home/runner/work/matrixone/matrixone/pkg/cnservice/server.go:263 +0xfb
2024-09-20T04:29:29.7925167Z   github.com/matrixorigin/matrixone/pkg/embed.(*operator).Close()
2024-09-20T04:29:29.7926535Z       /home/runner/work/matrixone/matrixone/pkg/embed/operator.go:129 +0xf0
2024-09-20T04:29:29.7927749Z   github.com/matrixorigin/matrixone/pkg/embed.(*cluster).Close()
2024-09-20T04:29:29.7929275Z       /home/runner/work/matrixone/matrixone/pkg/embed/cluster.go:145 +0x166
2024-09-20T04:29:29.7930569Z   github.com/matrixorigin/matrixone/pkg/embed.TestMultiClusterCanWork()
2024-09-20T04:29:29.7932062Z       /home/runner/work/matrixone/matrixone/pkg/embed/cluster_test.go:60 +0x93
2024-09-20T04:29:29.7933033Z   testing.tRunner()
2024-09-20T04:29:29.7934110Z       /opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:1690 +0x226
2024-09-20T04:29:29.7935031Z   testing.(*T).Run.gowrap1()
2024-09-20T04:29:29.7936199Z       /opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:1743 +0x44
2024-09-20T04:29:29.7936917Z 
2024-09-20T04:29:29.7937237Z Previous read at 0x00c083fed110 by goroutine 7924:
2024-09-20T04:29:29.7938312Z   github.com/matrixorigin/matrixone/pkg/frontend.(*Conn).BeginPacket()
2024-09-20T04:29:29.7940023Z       /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_buffer.go:546 +0x51
2024-09-20T04:29:29.7941503Z   github.com/matrixorigin/matrixone/pkg/frontend.(*MysqlProtocolImpl).beginPacket()
2024-09-20T04:29:29.7943143Z       /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_protocol.go:2733 +0x56
2024-09-20T04:29:29.7944659Z   github.com/matrixorigin/matrixone/pkg/frontend.(*MysqlProtocolImpl).appendPacket()
2024-09-20T04:29:29.7946294Z       /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_protocol.go:2743 +0x40
2024-09-20T04:29:29.7947904Z   github.com/matrixorigin/matrixone/pkg/frontend.(*MysqlProtocolImpl).WriteEOFIFAndNoFlush()
2024-09-20T04:29:29.7950007Z       /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_protocol.go:419 +0x6d
2024-09-20T04:29:29.7951576Z   github.com/matrixorigin/matrixone/pkg/frontend.(*MysqlResp).respColumnDefsWithoutFlush()
2024-09-20T04:29:29.7953319Z       /home/runner/work/matrixone/matrixone/pkg/frontend/result_row_stmt.go:254 +0x538
2024-09-20T04:29:29.7954880Z   github.com/matrixorigin/matrixone/pkg/frontend.(*MysqlResp).RespPreMeta()
2024-09-20T04:29:29.7956531Z       /home/runner/work/matrixone/matrixone/pkg/frontend/resp_client.go:120 +0xc5
2024-09-20T04:29:29.7957922Z   github.com/matrixorigin/matrixone/pkg/frontend.executeResultRowStmt()
2024-09-20T04:29:29.7959669Z       /home/runner/work/matrixone/matrixone/pkg/frontend/result_row_stmt.go:51 +0x5cf
2024-09-20T04:29:29.7960961Z   github.com/matrixorigin/matrixone/pkg/frontend.executeStmt()
2024-09-20T04:29:29.7962464Z       /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_cmd_executor.go:2695 +0x1912
2024-09-20T04:29:29.7963820Z   github.com/matrixorigin/matrixone/pkg/frontend.dispatchStmt()
2024-09-20T04:29:29.7965326Z       /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_cmd_executor.go:2558 +0x4a8
2024-09-20T04:29:29.7966791Z   github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithIncrStmt()
2024-09-20T04:29:29.7968412Z       /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_cmd_executor.go:2527 +0x3cc
2024-09-20T04:29:29.7970118Z   github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithWorkspace()
2024-09-20T04:29:29.7971738Z       /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_cmd_executor.go:2484 +0x966
2024-09-20T04:29:29.7973160Z   github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithTxn()
2024-09-20T04:29:29.7974723Z       /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_cmd_executor.go:2374 +0x213
2024-09-20T04:29:29.7976196Z   github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithResponse()
2024-09-20T04:29:29.7977839Z       /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_cmd_executor.go:2338 +0x5d5
2024-09-20T04:29:29.7979315Z   github.com/matrixorigin/matrixone/pkg/frontend.doComQuery()
2024-09-20T04:29:29.7980846Z       /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_cmd_executor.go:2945 +0x37ed
2024-09-20T04:29:29.7982189Z   github.com/matrixorigin/matrixone/pkg/frontend.ExecRequest()
2024-09-20T04:29:29.7983674Z       /home/runner/work/matrixone/matrixone/pkg/frontend/mysql_cmd_executor.go:3020 +0xd91
2024-09-20T04:29:29.7985134Z   github.com/matrixorigin/matrixone/pkg/frontend.(*Routine).handleRequest()
2024-09-20T04:29:29.7986640Z       /home/runner/work/matrixone/matrixone/pkg/frontend/routine.go:290 +0xb39
2024-09-20T04:29:29.7987971Z   github.com/matrixorigin/matrixone/pkg/frontend.(*RoutineManager).Handler()
2024-09-20T04:29:29.7989764Z       /home/runner/work/matrixone/matrixone/pkg/frontend/routine_manager.go:383 +0x55e
2024-09-20T04:29:29.7991389Z   github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).handleRequest()
2024-09-20T04:29:29.7993001Z       /home/runner/work/matrixone/matrixone/pkg/frontend/server.go:471 +0x284
2024-09-20T04:29:29.7994357Z   github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).handleMessage()
2024-09-20T04:29:29.7995799Z       /home/runner/work/matrixone/matrixone/pkg/frontend/server.go:440 +0x4b
2024-09-20T04:29:29.7997120Z   github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).handleLoop()
2024-09-20T04:29:29.7998567Z       /home/runner/work/matrixone/matrixone/pkg/frontend/server.go:203 +0x4b
2024-09-20T04:29:29.8000049Z   github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).handleConn()
2024-09-20T04:29:29.8001562Z       /home/runner/work/matrixone/matrixone/pkg/frontend/server.go:199 +0x5dc
2024-09-20T04:29:29.8002978Z   github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).startAccept.gowrap2()
2024-09-20T04:29:29.8004496Z       /home/runner/work/matrixone/matrixone/pkg/frontend/server.go:167 +0x4f
2024-09-20T04:29:29.8005239Z 
2024-09-20T04:29:29.8005476Z Goroutine 2497 (running) created at:
2024-09-20T04:29:29.8006121Z   testing.(*T).Run()
2024-09-20T04:29:29.8007098Z       /opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:1743 +0x825
2024-09-20T04:29:29.8007987Z   testing.runTests.func1()
2024-09-20T04:29:29.8009161Z       /opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:2168 +0x85
2024-09-20T04:29:29.8010187Z   testing.tRunner()
2024-09-20T04:29:29.8011181Z       /opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:1690 +0x226
2024-09-20T04:29:29.8012055Z   testing.runTests()
2024-09-20T04:29:29.8013124Z       /opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:2166 +0x8be
2024-09-20T04:29:29.8014007Z   testing.(*M).Run()
2024-09-20T04:29:29.8014981Z       /opt/hostedtoolcache/go/1.23.0/x64/src/testing/testing.go:2034 +0xf17
2024-09-20T04:29:29.8015847Z   main.main()
2024-09-20T04:29:29.8016377Z       _testmain.go:73 +0x164
2024-09-20T04:29:29.8016738Z 
2024-09-20T04:29:29.8016968Z Goroutine 7924 (running) created at:
2024-09-20T04:29:29.8017987Z   github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).startAccept()
2024-09-20T04:29:29.8019674Z       /home/runner/work/matrixone/matrixone/pkg/frontend/server.go:167 +0x1a8
2024-09-20T04:29:29.8021088Z   github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).startListener.gowrap1()
2024-09-20T04:29:29.8022652Z       /home/runner/work/matrixone/matrixone/pkg/frontend/server.go:139 +0x4f
2024-09-20T04:29:29.8023562Z ==================
2
daviszhen commented 1 month ago

it was fixed