Data Pipeline: Consider using filter by engineering - Githubissues

rivernews / review-scraper-java-development-environment

An environment to develop review scraper

0 stars 1 forks source link

Data Pipeline: Consider using filter by engineering #16

Open rivernews opened 4 years ago

rivernews commented 4 years ago

Is this necessary? Do we need this? How to implement this? Do we still want the general reviews?

Some idea

Store engineering filtered reviews separatey. This is going to have some scrapping overhead. BUt when storing, we can "inject" the category "engineering" information into the review object, not necessarily need to store entirely duplicated data.
Even better - if we could get this category information directly from the html DOM, but it has to be provided there first.
Any other way to do this?

rivernews commented 4 years ago

Approach of preference

We may still do a full-scraping w/o any filter. We can then do a second round, specifically for filtered reviews.