search

abu-rayyan / crawper

A python based crawler and scraper
0 stars 0 forks source link

duplicate products, reviews and reviewers #4

Closed ghost closed 6 years ago

ghost commented 6 years ago

Some times the scraper scrapes the products, reviews or reviewers information which already exists in the database, for now we are discarding them via duplicate value constraint, in the future we need to adopt a better way of doing that.

ghost commented 6 years ago

addressed in:

  1. https://github.com/ExpertsVision/crawper/commit/259e8e8dc41883f64c32e516f1eefb20af1f1acf

  2. https://github.com/ExpertsVision/crawper/commit/88ec4d3a848be2f31487d6b1753680b7b60ed218