search

jayzeng / scrapy-elasticsearch

A scrapy pipeline which send items to Elastic Search server
327 stars 88 forks source link

Specifiy specific fields to index #56

Closed maanas closed 7 years ago

maanas commented 7 years ago

I am also storing the raw html along with the items, but do not want to send that to ES index. Can we specify the specific fields which should be send to ES for indexeing

jayzeng commented 7 years ago

hmm, I don't fully understand what you meant by 'specify the specific fields'. Can you put some examples to illustrate?

jayzeng commented 7 years ago

Ah, I know what you mean now. It sounds to me you will need to make a decision whether to index the entire raw html or certain fields (which will be structured). If raw html is something you have to do and unstructured, how will you know which fields to specify?

jayzeng commented 7 years ago

@maanas free feel to re-open if you have any good idea