jayzeng / scrapy-elasticsearch

A scrapy pipeline which send items to Elastic Search server
327 stars 88 forks source link

Suggestion of setting '_index', '_source' and other parameters directly in parser #86

Open julfes opened 4 years ago

julfes commented 4 years ago

Hi, I want to suggest to change the operation of the pipeline so that the items to be indexed are created by the user at the parser level and not via the parameters 'ELASTICSEARCH_INDEX' and 'ELASTICSEARCH_TYPE' by the pipeline. Advantages: -The user can specify different indices in elasticsearch for different parsers -The user can control the '_op_type' setting of the bulk method to change for example from 'index' to 'update'

Cheers, Julian