search

influxdata / influxdb

Scalable datastore for metrics, events, and real-time analytics
https://influxdata.com
Apache License 2.0
28.62k stars 3.54k forks source link

Data race between appendEntry and FlushAndSync tsi1.(*LogFile) on master-1.x #25181

Closed chengshiwen closed 1 month ago

chengshiwen commented 1 month ago

Steps to reproduce: List the minimal actions needed to reproduce the behaviour.

  1. In Development, run INFLUXDB_DATA_INDEX_VERSION=tsi1 INFLUXDB_DATA_WAL_FSYNC_DELAY=1ms go run -race cmd/influxd/main.go
  2. Run stress program ./influx-stress insert -f --host http://localhost:8086 -r 10m
  3. Delete measurement drop measurement ctr

Expected behaviour: No data race.

Actual behaviour:

==================
WARNING: DATA RACE
Read at 0x00c0381d6368 by goroutine 993268:
  bufio.(*Writer).Available()
      /usr/local/go/src/bufio/bufio.go:659 +0xe5
  bufio.(*Writer).Write()
      /usr/local/go/src/bufio/bufio.go:677 +0x1a
  github.com/influxdata/influxdb/tsdb/index/tsi1.(*LogFile).appendEntry()
      /Users/csw/Go/influxdata/influxdb/tsdb/index/tsi1/log_file.go:618 +0x189
  github.com/influxdata/influxdb/tsdb/index/tsi1.(*LogFile).ExecEntries()
      /Users/csw/Go/influxdata/influxdb/tsdb/index/tsi1/log_file.go:1081 +0x11a
  github.com/influxdata/influxdb/tsdb/index/tsi1.(*LogFile).Writes()
      /Users/csw/Go/influxdata/influxdb/tsdb/index/tsi1/log_file.go:1090 +0x48
  github.com/influxdata/influxdb/tsdb/index/tsi1.(*Partition).DropMeasurement()
      /Users/csw/Go/influxdata/influxdb/tsdb/index/tsi1/partition.go:628 +0x5fb
  github.com/influxdata/influxdb/tsdb/index/tsi1.(*Index).DropMeasurement.func1()
      /Users/csw/Go/influxdata/influxdb/tsdb/index/tsi1/index.go:607 +0x8c

Previous write at 0x00c0381d6368 by goroutine 993258:
  bufio.(*Writer).Flush()
      /usr/local/go/src/bufio/bufio.go:654 +0x33e
  github.com/influxdata/influxdb/tsdb/index/tsi1.(*LogFile).FlushAndSync()
      /Users/csw/Go/influxdata/influxdb/tsdb/index/tsi1/log_file.go:205 +0x7b
  github.com/influxdata/influxdb/tsdb/index/tsi1.(*LogFile).Writes()
      /Users/csw/Go/influxdata/influxdb/tsdb/index/tsi1/log_file.go:1095 +0x76
  github.com/influxdata/influxdb/tsdb/index/tsi1.(*Partition).DropMeasurement()
      /Users/csw/Go/influxdata/influxdb/tsdb/index/tsi1/partition.go:628 +0x5fb
  github.com/influxdata/influxdb/tsdb/index/tsi1.(*Index).DropMeasurement.func1()
      /Users/csw/Go/influxdata/influxdb/tsdb/index/tsi1/index.go:607 +0x8c

Goroutine 993268 (running) created at:
  github.com/influxdata/influxdb/tsdb/index/tsi1.(*Index).DropMeasurement()
      /Users/csw/Go/influxdata/influxdb/tsdb/index/tsi1/index.go:601 +0x110
  github.com/influxdata/influxdb/tsdb/index/tsi1.(*Index).DropMeasurementIfSeriesNotExist()
      /Users/csw/Go/influxdata/influxdb/tsdb/index/tsi1/index.go:897 +0x7b
  github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).deleteSeriesRange()
      /Users/csw/Go/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1783 +0x13f7
  github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).DeleteSeriesRangeWithPredicate()
      /Users/csw/Go/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1538 +0x83d
  github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).DeleteSeriesRange()
      /Users/csw/Go/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1430 +0x3f4
  github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).DeleteMeasurement()
      /Users/csw/Go/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1881 +0x2b1
  github.com/influxdata/influxdb/tsdb.(*Shard).DeleteMeasurement()
      /Users/csw/Go/influxdata/influxdb/tsdb/shard.go:790 +0x75
  github.com/influxdata/influxdb/tsdb.(*Store).DeleteMeasurement.func1()
      /Users/csw/Go/influxdata/influxdb/tsdb/store.go:967 +0x2b2
  github.com/influxdata/influxdb/tsdb.(*Store).walkShards.func1()
      /Users/csw/Go/influxdata/influxdb/tsdb/store.go:1015 +0x56
  github.com/influxdata/influxdb/tsdb.(*Store).walkShards.func2()
      /Users/csw/Go/influxdata/influxdb/tsdb/store.go:1021 +0x41

Goroutine 993258 (running) created at:
  github.com/influxdata/influxdb/tsdb/index/tsi1.(*Index).DropMeasurement()
      /Users/csw/Go/influxdata/influxdb/tsdb/index/tsi1/index.go:601 +0x110
  github.com/influxdata/influxdb/tsdb/index/tsi1.(*Index).DropMeasurementIfSeriesNotExist()
      /Users/csw/Go/influxdata/influxdb/tsdb/index/tsi1/index.go:897 +0x7b
  github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).deleteSeriesRange()
      /Users/csw/Go/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1783 +0x13f7
  github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).DeleteSeriesRangeWithPredicate()
      /Users/csw/Go/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1538 +0x83d
  github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).DeleteSeriesRange()
      /Users/csw/Go/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1430 +0x3f4
  github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).DeleteMeasurement()
      /Users/csw/Go/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1881 +0x2b1
  github.com/influxdata/influxdb/tsdb.(*Shard).DeleteMeasurement()
      /Users/csw/Go/influxdata/influxdb/tsdb/shard.go:790 +0x75
  github.com/influxdata/influxdb/tsdb.(*Store).DeleteMeasurement.func1()
      /Users/csw/Go/influxdata/influxdb/tsdb/store.go:967 +0x2b2
  github.com/influxdata/influxdb/tsdb.(*Store).walkShards.func1()
      /Users/csw/Go/influxdata/influxdb/tsdb/store.go:1015 +0x56
  github.com/influxdata/influxdb/tsdb.(*Store).walkShards.func2()
      /Users/csw/Go/influxdata/influxdb/tsdb/store.go:1021 +0x41
==================

Environment info:

Darwin 23.5.0 x86_64, macOS 14.5

influxdb branch-1.8

Config:

INFLUXDB_DATA_INDEX_VERSION=tsi1 INFLUXDB_DATA_WAL_FSYNC_DELAY=1ms

chengshiwen commented 1 month ago

This data race may also cause a short write error like #21712