search

bocchilorenzo / autoscout24_bot

Scraper for autoscout24.it that sends new listings to a Telegram chat
GNU Affero General Public License v3.0
0 stars 1 forks source link

Can this work with autoscout24.ch? #2

Open muharem321 opened 1 year ago

muharem321 commented 1 year ago

Also I am having trouble, I am not getting any messages on telegram, the console is only outputting 'scraped x pages'

muharem321 commented 1 year ago

Correction, it only prints out scrapped 1 pages, never increments, what could be the issue

bocchilorenzo commented 1 year ago

It's meant to be that way. Initially, the script scrapes the first page and sends the results to the telegram chat. Then, periodically, the script will fetch for new results and send only the new ones that came after the last of the previous call. It's not meant to be a scraper for a one off call, although it can be adapted to do that.

muharem321 commented 1 year ago

I think there is a problem with some anti-bot captcha, the listings variable gets this value Anti-Bot Captcha (and many more text)

bocchilorenzo commented 1 year ago

I tried running it for a couple of hours with only 10 seconds between the updates and couldn't reproduce the error. Unfortunately, if there is any anti-bot mechanism, the only way to bypass it for now is to either use a vpn or reducing the update frequency.

muharem321 commented 1 year ago

Okay, no more problem with the captcha, now the problem seems to be with the URL, when I go to autoscout24.it, it works, but when I go to autoscout24.ch, it won't work, listings variable is always an empty list, for example when I put this URL: https://www.autoscout24.ch/de/autos/audi--q5?fuel=271%2C15%2C246&cond=272%2C49%2C50&yearto=2020&make=5&model=23&vehtyp=10 It doesn't work

bocchilorenzo commented 1 year ago

I just checked out the link and the reason it doesn't work is that the layout is different from the italian site. I'll see if I can adapt it to the swiss one.

muharem321 commented 1 year ago

Also the problem with the swiss site, it doesn't order vehicles by their published date, it sorts them in some unknown order. But I guess, I could sort vehicles by the cheapest price, that would kinda work for my research purposes.