Add new scrappy pipeline for text cleaning

imrenagi / rojak-pantau

Kawal Media crawler & machine learning services

http://kawalmedia.id

MIT License

1 stars 4 forks source link

Open imrenagi opened 6 years ago

imrenagi commented 6 years ago

We need to add new scrapy pipeline that will remove all html tags and js scripts in the raw_content. So the result will only contain clean text.
You might need to add another field in News class items.py. You can add new field called cleaned_content
You dont have to store this cleaned_content to mysql database. This task will be done in another new issue.

thakkardharmik commented 6 years ago

@imrenagi I would like to give it a try.