flush/commit every ELASTICSEARCH_FLUSH_SIZE items, and flush at end.

jayzeng / scrapy-elasticsearch

A scrapy pipeline which send items to Elastic Search server

327 stars 88 forks source link

Closed danizen closed 7 years ago

danizen commented 7 years ago

I'll work on this; pretty basic - expect pull request. If there's a way and I've missed it, let me know.

jayzeng commented 7 years ago

@danizen you can set the buffer size by specifying ELASTICSEARCH_BUFFER_LENGTH in settings.py (which is default to 500)

danizen commented 7 years ago

Thanks.