search

xitongsys / parquet-go

pure golang library for reading/writing parquet file
Apache License 2.0
1.24k stars 294 forks source link

race condition #559

Open hangxie opened 1 year ago

hangxie commented 1 year ago

Got race condition in an older version so I upgraded to latest master branch but still get:`

WARNING: DATA RACE
Read at 0x00c000a5dc50 by goroutine 714:
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).flushObjs()
      /Users/xiehang/go/pkg/mod/github.com/xitongsys/parquet-go@v1.6.3-0.20230806034711-b6d7d8771e28/writer/writer.go:341 +0x520
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).Flush()
      /Users/xiehang/go/pkg/mod/github.com/xitongsys/parquet-go@v1.6.3-0.20230806034711-b6d7d8771e28/writer/writer.go:370 +0x34
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).WriteStop()
      /Users/xiehang/go/pkg/mod/github.com/xitongsys/parquet-go@v1.6.3-0.20230806034711-b6d7d8771e28/writer/writer.go:149 +0x60
  github.com/hangxie/parquet-tools/cmd.ImportCmd.importJSON()
      /Users/xiehang/dev/parquet/parquet-tools/cmd/import.go:116 +0x238
  github.com/hangxie/parquet-tools/cmd.Test_ImportCmd_importJSON_schema_mismatch()
      /Users/xiehang/dev/parquet/parquet-tools/cmd/import_test.go:247 +0xfc
  testing.tRunner()
      /opt/homebrew/Cellar/go/1.21.1/libexec/src/testing/testing.go:1595 +0x194
  testing.(*T).Run.func1()
      /opt/homebrew/Cellar/go/1.21.1/libexec/src/testing/testing.go:1648 +0x40

Previous write at 0x00c000a5dc50 by goroutine 723:
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).flushObjs.func1.1()
      /Users/xiehang/go/pkg/mod/github.com/xitongsys/parquet-go@v1.6.3-0.20230806034711-b6d7d8771e28/writer/writer.go:297 +0x1c8
  runtime.gopanic()
      /opt/homebrew/Cellar/go/1.21.1/libexec/src/runtime/panic.go:920 +0x268
  github.com/xitongsys/parquet-go/encoding.WritePlainINT32()
      /Users/xiehang/go/pkg/mod/github.com/xitongsys/parquet-go@v1.6.3-0.20230806034711-b6d7d8771e28/encoding/encodingwrite.go:75 +0x3bc
  github.com/xitongsys/parquet-go/encoding.WritePlain()
      /Users/xiehang/go/pkg/mod/github.com/xitongsys/parquet-go@v1.6.3-0.20230806034711-b6d7d8771e28/encoding/encodingwrite.go:43 +0x408
  github.com/xitongsys/parquet-go/layout.(*Page).EncodingValues()
      /Users/xiehang/go/pkg/mod/github.com/xitongsys/parquet-go@v1.6.3-0.20230806034711-b6d7d8771e28/layout/page.go:189 +0x140
  github.com/xitongsys/parquet-go/layout.(*Page).DataPageCompress()
      /Users/xiehang/go/pkg/mod/github.com/xitongsys/parquet-go@v1.6.3-0.20230806034711-b6d7d8771e28/layout/page.go:216 +0x360
  github.com/xitongsys/parquet-go/layout.TableToDataPages()
      /Users/xiehang/go/pkg/mod/github.com/xitongsys/parquet-go@v1.6.3-0.20230806034711-b6d7d8771e28/layout/page.go:129 +0xb58
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).flushObjs.func1()
      /Users/xiehang/go/pkg/mod/github.com/xitongsys/parquet-go@v1.6.3-0.20230806034711-b6d7d8771e28/writer/writer.go:328 +0x3cc
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).flushObjs.func2()
      /Users/xiehang/go/pkg/mod/github.com/xitongsys/parquet-go@v1.6.3-0.20230806034711-b6d7d8771e28/writer/writer.go:336 +0x60

Goroutine 714 (running) created at:
  testing.(*T).Run()
      /opt/homebrew/Cellar/go/1.21.1/libexec/src/testing/testing.go:1648 +0x5d8
  testing.runTests.func1()
      /opt/homebrew/Cellar/go/1.21.1/libexec/src/testing/testing.go:2054 +0x80
  testing.tRunner()
      /opt/homebrew/Cellar/go/1.21.1/libexec/src/testing/testing.go:1595 +0x194
  testing.runTests()
      /opt/homebrew/Cellar/go/1.21.1/libexec/src/testing/testing.go:2052 +0x6d8
  testing.(*M).Run()
      /opt/homebrew/Cellar/go/1.21.1/libexec/src/testing/testing.go:1925 +0x908
  main.main()
      _testmain.go:273 +0x2b4

Goroutine 723 (finished) created at:
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).flushObjs()
      /Users/xiehang/go/pkg/mod/github.com/xitongsys/parquet-go@v1.6.3-0.20230806034711-b6d7d8771e28/writer/writer.go:289 +0x294
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).Flush()
      /Users/xiehang/go/pkg/mod/github.com/xitongsys/parquet-go@v1.6.3-0.20230806034711-b6d7d8771e28/writer/writer.go:370 +0x34
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).WriteStop()
      /Users/xiehang/go/pkg/mod/github.com/xitongsys/parquet-go@v1.6.3-0.20230806034711-b6d7d8771e28/writer/writer.go:149 +0x60
  github.com/hangxie/parquet-tools/cmd.ImportCmd.importJSON()
      /Users/xiehang/dev/parquet/parquet-tools/cmd/import.go:116 +0x238
  github.com/hangxie/parquet-tools/cmd.Test_ImportCmd_importJSON_schema_mismatch()
      /Users/xiehang/dev/parquet/parquet-tools/cmd/import_test.go:247 +0xfc
  testing.tRunner()
      /opt/homebrew/Cellar/go/1.21.1/libexec/src/testing/testing.go:1595 +0x194
  testing.(*T).Run.func1()
      /opt/homebrew/Cellar/go/1.21.1/libexec/src/testing/testing.go:1648 +0x40
hangxie commented 1 year ago

Added a unit test to reproduce this problem in https://github.com/xitongsys/parquet-go/pull/560

=== RUN   TestWriteStopRaceConditionOnError
==================
WARNING: DATA RACE
Read at 0x00c0004b3fc0 by goroutine 40:
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).flushObjs()
      github.com/xitongsys/parquet-go/writer/writer.go:341 +0x628
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).Flush()
      github.com/xitongsys/parquet-go/writer/writer.go:370 +0x78
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).WriteStop()
      github.com/xitongsys/parquet-go/writer/writer.go:149 +0xb4
  github.com/xitongsys/parquet-go/writer.TestWriteStopRaceConditionOnError()
      github.com/xitongsys/parquet-go/writer/writer_test.go:251 +0x19c
  testing.tRunner()
      testing/testing.go:1595 +0x194
  testing.(*T).Run.func1()
      testing/testing.go:1648 +0x40

Previous write at 0x00c0004b3fc0 by goroutine 41:
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).flushObjs.func1.1()
      github.com/xitongsys/parquet-go/writer/writer.go:297 +0x210
  runtime.gopanic()
      runtime/panic.go:920 +0x268
  github.com/xitongsys/parquet-go/encoding.WritePlainINT64()
      github.com/xitongsys/parquet-go/encoding/encodingwrite.go:81 +0x308
  github.com/xitongsys/parquet-go/encoding.WritePlain()
      github.com/xitongsys/parquet-go/encoding/encodingwrite.go:45 +0x354
  github.com/xitongsys/parquet-go/layout.(*Page).EncodingValues()
      github.com/xitongsys/parquet-go/layout/page.go:189 +0x140
  github.com/xitongsys/parquet-go/layout.(*Page).DataPageCompress()
      github.com/xitongsys/parquet-go/layout/page.go:216 +0x360
  github.com/xitongsys/parquet-go/layout.TableToDataPages()
      github.com/xitongsys/parquet-go/layout/page.go:129 +0xb58
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).flushObjs.func1()
      github.com/xitongsys/parquet-go/writer/writer.go:328 +0x468
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).flushObjs.func2()
      github.com/xitongsys/parquet-go/writer/writer.go:336 +0x60

Goroutine 40 (running) created at:
  testing.(*T).Run()
      testing/testing.go:1648 +0x5d8
  testing.runTests.func1()
      testing/testing.go:2054 +0x80
  testing.tRunner()
      testing/testing.go:1595 +0x194
  testing.runTests()
      testing/testing.go:2052 +0x6d8
  testing.(*M).Run()
      testing/testing.go:1925 +0x908
  main.main()
      _testmain.go:103 +0x2b4

Goroutine 41 (finished) created at:
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).flushObjs()
      github.com/xitongsys/parquet-go/writer/writer.go:289 +0x328
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).Flush()
      github.com/xitongsys/parquet-go/writer/writer.go:370 +0x78
  github.com/xitongsys/parquet-go/writer.(*ParquetWriter).WriteStop()
      github.com/xitongsys/parquet-go/writer/writer.go:149 +0xb4
  github.com/xitongsys/parquet-go/writer.TestWriteStopRaceConditionOnError()
      github.com/xitongsys/parquet-go/writer/writer_test.go:251 +0x19c
  testing.tRunner()
      testing/testing.go:1595 +0x194
  testing.(*T).Run.func1()
      testing/testing.go:1648 +0x40
==================
    testing.go:1465: race detected during execution of test
--- FAIL: TestWriteStopRaceConditionOnError (0.00s)
=== NAME
    testing.go:1465: race detected during execution of test
FAIL
coverage: 77.8% of statements
FAIL    github.com/xitongsys/parquet-go/writer  0.272s
FAIL