kelindar / column

High-performance, columnar, in-memory store with bitmap indexing in Go
MIT License
1.44k stars 57 forks source link

Multiple goroutines cannot be read at the same time #51

Closed nodelrd closed 2 years ago

nodelrd commented 2 years ago

When i use goroutines for query data

Multiple goroutines cannot be read at the same time

nodelrd commented 2 years ago
for i := 0; i < 6000; i++ {
    // run a query over human mages
    go measure("indexed query", "human mages", func() {
        players.Query(func(txn *column.Txn) error {
            fmt.Printf("-> result = %v\n", txn.With("human", "mage").Count())
            return nil
        })
    }, 1)

    // run a query over human mages
    go measure("indexed query", "human female mages", func() {
        players.Query(func(txn *column.Txn) error {
            fmt.Printf("-> result = %v\n", txn.With("human", "female", "mage").Count())
            return nil
        })
    }, 1)
}

And memory not release

nodelrd commented 2 years ago

when i use goroutines insert cpu only used 400% , my server have 160 cpu 1646932465525

kelindar commented 2 years ago

There's an examples/bench example which runs reads/writes concurrently and measures it. The library should work (but not perfectly scale) for multiple concurrent readers and writers, especially for large collections. Unfortunately I don't have access to a 160 core machine, so won't be able to reproduce at the moment, but make sure you do not update/insert at the same time. From your screenshot I'm seeing concurrent inserts which would block the reads in this case.