search

owaisrazaeng1 / nimble-test

0 stars 0 forks source link

[Feature] Scraping data must be dynamic #12

Open olivierobert opened 1 year ago

olivierobert commented 1 year ago

Issue

Scraping data is currently hardcoded:

https://github.com/owaisrazaeng1/nimble-test/blob/a057d841097bc3a45158ecb89a882f924e4c4f69/app/services/Scraper/google_scraper_service.rb#L21-L26

As a result, the same data would be stored for all keywords.

Expected

Scraping data must be scraped from the HTML content of the search result page. The right selectors must be used to retrieve the wanted pieces of information.

owaisrazaeng1 commented 1 year ago

I ran into issues while using the kimurai gem for scraping. As it turns out the gem does not offer support for rail 3.x. So I hard coded the results to complete the application.

olivierobert commented 1 year ago

Have you tried using https://rubygems.org/gems/nokogiri instead? This gem is the standard to parse HTML.

owaisrazaeng1 commented 1 year ago

resolved in https://github.com/owaisrazaeng1/nimble-test/commit/7aa833bd51593f376ce1c36a37c31e7b314e627f