search

kelindar / column

High-performance, columnar, in-memory store with bitmap indexing in Go
MIT License
1.44k stars 57 forks source link

The performance seems low... #80

Open didip opened 1 year ago

didip commented 1 year ago

I modified the million benchmark to billion and it took forever to insert.

Is it possible to expand the commit.Buffer to speed up the transaction?

Dreeseaw commented 1 year ago

Hello, I also noticed a very high bump in insert-time-per-record when bumping amount to 1B from 10M (as in, more than 100x time for 100x records). While the mass-loading pattern doesn't exactly complement a store built for concurrent DML, I agree the 1B benchmark should insert in a reasonable time regardless - especially with all data being known before hand.

kelindar commented 1 year ago

Indeed, insertion is a bit slow as it touches most columns. I have a few questions to clarify this.

  1. How did you perform the insertion (single or multiple txn)?
  2. What was the time to insert 1B?
  3. What's the expected time?