search

jayzeng / scrapy-elasticsearch

A scrapy pipeline which send items to Elastic Search server
327 stars 88 forks source link

ELASTICSEARCH_BUFFER_LENGTH #62

Closed ghost closed 7 years ago

ghost commented 7 years ago

I use scrapy-redis, my spider is waiting for input from redis queue. If i send less urls than the buffer_length they wont be ever pushed into elasticsearch.

Do you have any workarounds?

ghost commented 7 years ago

I guess overwriting spider_idle can do the work but i am not sure how to send the data while spider is idle?

ghost commented 7 years ago

Solution, overwrite spider_idle to close spider, and restart: https://gist.github.com/pythoncontrol/64e0861e5a04a0e51643ebcf5851d722