Improve batch processing

al-serebrov / scrapinghub-elasticsearch-loader

Load items from Scrapinghub to ElasticSearch

GNU General Public License v3.0

11 stars 0 forks source link

Improve batch processing #1

Closed al-serebrov closed 5 years ago

al-serebrov commented 5 years ago

Hi @serebrov !

Could you please review the following changes:

[cli_interface] Add hashbang to be able to make the file executable
[es_loader] Add a better way to split items in chunks, more verbose logging, add docstrings
[docker-compose] Add volume to store loaded data (to keep it after container restart)

I'm particularly interested in finding a better way for "leftovers" condition (es_loader.py file, index_item() method).

Thanks!

al-serebrov commented 5 years ago

Hi @serebrov ! Thank you so much for the review! I've reworked ES loader according with your review, and also added some new functionality - now it calculates buffer size depending on a job items count, maybe it's not that optimal but I think it's better to have a hardcoded buffer size :) Could you please have a look again?