alirezamika / autoscraper

A Smart, Automatic, Fast and Lightweight Web Scraper for Python
MIT License
6.16k stars 648 forks source link

Ignores duplicate value #83

Closed thouravi closed 1 year ago

thouravi commented 1 year ago

Hi,

I was trying to fetch from a website which had some duplicate values like item A and item B had similar price i.e. $1.0 Your AutoScraper simply ignored any duplicate values and fetched unique items to the result list.

Website had 18 items, result list had only 5, all unique. I hope if you can fix this issue, thanks.

alirezamika commented 1 year ago

You can use unique=False parameter.

thouravi commented 1 year ago

This is my code:

from autoscraper import AutoScraper

url = 'https://csgostash.com/stickers/tournament/Rio+2022?name=&sticker_type=tournament_team&rarity_contraband=0&rarity_covert=0&rarity_legendary=0&rarity_mythical=0&rarity_rare=1&container=any&sort_agg=avg&sort=default'

wanted_list = ["$0.16"]
scraper = AutoScraper()
result = scraper.build(url, wanted_list)

Where can I add this parameter?

alirezamika commented 1 year ago

You can use this parameter with get_result_similar function.

thouravi commented 1 year ago

It worked, thanks. I wish it was a default property.