search

go-enry / go-license-detector

Reliable project licenses detector.
Other
127 stars 36 forks source link

Data race on licensedb.Detect() #9

Closed g4s8 closed 3 years ago

g4s8 commented 3 years ago

Hi there. I tried to integrate this library into Gitaly component of Gitlab to replace Ruby licensee version with Go implementation: https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3313

But the CI failed on race detection phase (with go test -race). So I tried to reproduce it on go-enry/go-license-detector tests with:

go test -race -v ./licensedb

And it fails with error:

=== RUN   TestDataset
==================
WARNING: DATA RACE
Write at 0x000001378b30 by goroutine 944:
  github.com/hhatto/gorst.initParser()
      /home/g4s8/go/pkg/mod/github.com/hhatto/gorst@v0.0.0-20181029133204-ca9f730cac5b/parser.leg.go:12772 +0x3b8
  github.com/hhatto/gorst.NewParser()
      /home/g4s8/go/pkg/mod/github.com/hhatto/gorst@v0.0.0-20181029133204-ca9f730cac5b/rst.go:63 +0x3a8
  github.com/go-enry/go-license-detector/v4/licensedb/internal/processors.RestructuredText()
      /home/g4s8/projects/github.com/g4s8/go-license-detector/licensedb/internal/processors/markup.go:20 +0x3b
  github.com/go-enry/go-license-detector/v4/licensedb/internal.ExtractReadmeFiles()
      /home/g4s8/projects/github.com/g4s8/go-license-detector/licensedb/internal/investigation.go:137 +0x3b4
  github.com/go-enry/go-license-detector/v4/licensedb.Detect()
      /home/g4s8/projects/github.com/g4s8/go-license-detector/licensedb/licensedb.go:46 +0x2a7
  github.com/go-enry/go-license-detector/v4/licensedb.TestDataset.func1()
      /home/g4s8/projects/github.com/g4s8/go-license-detector/licensedb/dataset_test.go:28 +0x128

Previous write at 0x000001378b30 by goroutine 928:
  [failed to restore the stack]

Goroutine 944 (running) created at:
  github.com/go-enry/go-license-detector/v4/licensedb.TestDataset()
      /home/g4s8/projects/github.com/g4s8/go-license-detector/licensedb/dataset_test.go:26 +0x2fc
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1123 +0x202

Goroutine 928 (running) created at:
  github.com/go-enry/go-license-detector/v4/licensedb.TestDataset()
      /home/g4s8/projects/github.com/g4s8/go-license-detector/licensedb/dataset_test.go:26 +0x2fc
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1123 +0x202
==================
==================
WARNING: DATA RACE
Write at 0x0000013444a0 by goroutine 944:
  github.com/hhatto/gorst.initParser()
      /home/g4s8/go/pkg/mod/github.com/hhatto/gorst@v0.0.0-20181029133204-ca9f730cac5b/parser.leg.go:12773 +0x3dc
  github.com/hhatto/gorst.NewParser()
      /home/g4s8/go/pkg/mod/github.com/hhatto/gorst@v0.0.0-20181029133204-ca9f730cac5b/rst.go:63 +0x3a8
  github.com/go-enry/go-license-detector/v4/licensedb/internal/processors.RestructuredText()
      /home/g4s8/projects/github.com/g4s8/go-license-detector/licensedb/internal/processors/markup.go:20 +0x3b
  github.com/go-enry/go-license-detector/v4/licensedb/internal.ExtractReadmeFiles()
      /home/g4s8/projects/github.com/g4s8/go-license-detector/licensedb/internal/investigation.go:137 +0x3b4
  github.com/go-enry/go-license-detector/v4/licensedb.Detect()
      /home/g4s8/projects/github.com/g4s8/go-license-detector/licensedb/licensedb.go:46 +0x2a7
  github.com/go-enry/go-license-detector/v4/licensedb.TestDataset.func1()
      /home/g4s8/projects/github.com/g4s8/go-license-detector/licensedb/dataset_test.go:28 +0x128

Previous write at 0x0000013444a0 by goroutine 928:
  [failed to restore the stack]

Goroutine 944 (running) created at:
  github.com/go-enry/go-license-detector/v4/licensedb.TestDataset()
      /home/g4s8/projects/github.com/g4s8/go-license-detector/licensedb/dataset_test.go:26 +0x2fc
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1123 +0x202

Goroutine 928 (running) created at:
  github.com/go-enry/go-license-detector/v4/licensedb.TestDataset()
      /home/g4s8/projects/github.com/g4s8/go-license-detector/licensedb/dataset_test.go:26 +0x2fc
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1123 +0x202
==================
==================
WARNING: DATA RACE
Write at 0x00c000222098 by goroutine 125:
  regexp.(*Regexp).Longest()
      /usr/lib/go/src/regexp/regexp.go:166 +0x64
  github.com/jdkato/prose/chunk.Locate()
      /home/g4s8/go/pkg/mod/github.com/jdkato/prose@v1.1.0/chunk/chunk.go:65 +0x4f
  github.com/jdkato/prose/chunk.Chunk()
      /home/g4s8/go/pkg/mod/github.com/jdkato/prose@v1.1.0/chunk/chunk.go:50 +0x7c
  github.com/go-enry/go-license-detector/v4/licensedb/internal.investigateReadmeFile()
      /home/g4s8/projects/github.com/g4s8/go-license-detector/licensedb/internal/nlp.go:67 +0x469
  github.com/go-enry/go-license-detector/v4/licensedb/internal.(*database).QueryReadmeText()
      /home/g4s8/projects/github.com/g4s8/go-license-detector/licensedb/internal/db.go:443 +0x404
  github.com/go-enry/go-license-detector/v4/licensedb/internal.InvestigateReadmeText()
      /home/g4s8/projects/github.com/g4s8/go-license-detector/licensedb/internal/investigation.go:157 +0xaa
  github.com/go-enry/go-license-detector/v4/licensedb/internal.InvestigateReadmeTexts.func1()
      /home/g4s8/projects/github.com/g4s8/go-license-detector/licensedb/internal/investigation.go:150 +0x68
  github.com/go-enry/go-license-detector/v4/licensedb/internal.investigateCandidates()
      /home/g4s8/projects/github.com/g4s8/go-license-detector/licensedb/internal/investigation.go:71 +0x3b8
  github.com/go-enry/go-license-detector/v4/licensedb/internal.InvestigateReadmeTexts()
      /home/g4s8/projects/github.com/g4s8/go-license-detector/licensedb/internal/investigation.go:149 +0x77
  github.com/go-enry/go-license-detector/v4/licensedb.Detect()
      /home/g4s8/projects/github.com/g4s8/go-license-detector/licensedb/licensedb.go:50 +0x2fd
  github.com/go-enry/go-license-detector/v4/licensedb.TestDataset.func1()
      /home/g4s8/projects/github.com/g4s8/go-license-detector/licensedb/dataset_test.go:28 +0x128

Previous write at 0x00c000222098 by goroutine 777:
  regexp.(*Regexp).Longest()
      /usr/lib/go/src/regexp/regexp.go:166 +0x64
  github.com/jdkato/prose/chunk.Locate()
      /home/g4s8/go/pkg/mod/github.com/jdkato/prose@v1.1.0/chunk/chunk.go:65 +0x4f
  github.com/jdkato/prose/chunk.Chunk()
      /home/g4s8/go/pkg/mod/github.com/jdkato/prose@v1.1.0/chunk/chunk.go:50 +0x7c
  github.com/go-enry/go-license-detector/v4/licensedb/internal.investigateReadmeFile()
      /home/g4s8/projects/github.com/g4s8/go-license-detector/licensedb/internal/nlp.go:67 +0x469
  github.com/go-enry/go-license-detector/v4/licensedb/internal.(*database).QueryReadmeText()
      /home/g4s8/projects/github.com/g4s8/go-license-detector/licensedb/internal/db.go:443 +0x404
  github.com/go-enry/go-license-detector/v4/licensedb/internal.InvestigateReadmeText()
      /home/g4s8/projects/github.com/g4s8/go-license-detector/licensedb/internal/investigation.go:157 +0xaa
  github.com/go-enry/go-license-detector/v4/licensedb/internal.InvestigateReadmeTexts.func1()
      /home/g4s8/projects/github.com/g4s8/go-license-detector/licensedb/internal/investigation.go:150 +0x68
  github.com/go-enry/go-license-detector/v4/licensedb/internal.investigateCandidates()
      /home/g4s8/projects/github.com/g4s8/go-license-detector/licensedb/internal/investigation.go:71 +0x3b8
  github.com/go-enry/go-license-detector/v4/licensedb/internal.InvestigateReadmeTexts()
      /home/g4s8/projects/github.com/g4s8/go-license-detector/licensedb/internal/investigation.go:149 +0x77
  github.com/go-enry/go-license-detector/v4/licensedb.Detect()
      /home/g4s8/projects/github.com/g4s8/go-license-detector/licensedb/licensedb.go:50 +0x2fd
  github.com/go-enry/go-license-detector/v4/licensedb.TestDataset.func1()
      /home/g4s8/projects/github.com/g4s8/go-license-detector/licensedb/dataset_test.go:28 +0x128

Goroutine 125 (running) created at:
  github.com/go-enry/go-license-detector/v4/licensedb.TestDataset()
      /home/g4s8/projects/github.com/g4s8/go-license-detector/licensedb/dataset_test.go:26 +0x2fc
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1123 +0x202

Goroutine 777 (running) created at:
  github.com/go-enry/go-license-detector/v4/licensedb.TestDataset()
      /home/g4s8/projects/github.com/g4s8/go-license-detector/licensedb/dataset_test.go:26 +0x2fc
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1123 +0x202
==================
895 902 99%
    testing.go:1038: race detected during execution of test
--- FAIL: TestDataset (135.48s)
=== CONT  
    testing.go:1038: race detected during execution of test
FAIL
FAIL    github.com/go-enry/go-license-detector/v4/licensedb 136.905s
FAIL

Checked with go version go1.15.8 linux/amd64.

g4s8 commented 3 years ago

Submitted #10 PR with GitHub action workflow to reproduce this issue.