chore: Design the webscraping strategy to extract the information.

C10-Brazilian-e-commerce-modeling-team / brazilian-e-commerce

0 stars 6 forks source link

chore: Design the webscraping strategy to extract the information. #17

Closed alexrods closed 2 years ago

alexrods commented 2 years ago

Summary

Set the tools for data extraction through web scrapping

Acceptance Criteria

[x] Select the language to use
[x] Select the libraries to use
[x] Select the pages to which to do web scraping

GabyGO2108 commented 2 years ago

For this issue we decided to go with our favorite language, Python. The reasons, well:

It's simplicity
It has a large amount of libraries
Timely saving
The syntax is easily understood
Familiarity- the entire team is familiarized with it

As for the libraries, we first thought about using Scrapy and LXML for their power and simplicity. We'll see if they are enough, or changes will need to be made.

alexrods commented 2 years ago

The Web Scraping was made using requests and lxml libraries on Mercado Livre Brazil
The strategy is: 1- Define categories to scrape. 2- Collect the link of those categories. 3- Collect the link of each product of each category. 4- Scrape reviews for each product.