Scraper doesn't work

[michaeldacanay]

• Black box test: attempted search for "chair" returns the error "Data couldn't be found in the database"
• According to the previous team, a set of 3 items returns data, one of which includes "laptop". Their reasoning -- the former behavior of waiting for several minutes is bad design + user experience. Their idea: run a daily batch job for scraping pre-selected/popular items and store results in a database. Then, when a user searches for one of the items, the return is much faster. However, this makes the application virtually useless for searching prices of anything else.

[LoganWilliams1]

Scraper was reworked to behave as expected. The three hard-coded search results are still added to the database on startup, but the app will now scrape any other search query entered by the user. We are still using the idea of storing results in the database for instant access by any subsequent searches. We are unsure of the previous group's experience in detail, but running for the scraper usually takes about 15-20 seconds. Once the scraper has been run for a given search term, the results are saved in the database, and repeat searches will access those results instantly. Although this is not the most optimal behavior- over time, the database should be refreshed when current results become outdated. However, rescraping every single stored search term at the same time on a daily basis is not feasible. The best solution may be some sort of hybrid approach between ours and the previous teams. Therefore, more work can be done to improve the scraper/database dynamic, but the primary issue at hand is solved.