ElasticSearch bottleneck for batch record creation

spiral-project / daybed

Form validation and data storage API

http://daybed.rtfd.org/

BSD 3-Clause "New" or "Revised" License

52 stars 8 forks source link

ElasticSearch bottleneck for batch record creation #264

Open leplatrem opened 10 years ago

leplatrem commented 10 years ago

Importing a model with 3000 inline records is 20x slower with ES enabled :|

➜  time node import.js                     
node import.js  0,19s user 0,05s system 12% cpu 1,841 total
➜  import-geojson  time node import.js
node import.js  0,17s user 0,05s system 0% cpu 37,006 total

See https://github.com/spiral-project/daybed.js/pull/28

See refresh param https://github.com/spiral-project/daybed/blob/1.0.1/daybed/indexer.py#L117

See #213

leplatrem commented 10 years ago

Without index refresh, it's only 10x slower :) (13secs)

I guess some time should be invested in handling events asynchronously instead...