search

godfriedmeesters / scraper

As part of DiffScraper, one or more bots can be deployed. Ready-to-use bots are provided that can extract offers from mobile applications, mobile websites and desktop websites.
GNU General Public License v3.0
2 stars 0 forks source link

KayakWebScraper stopped working #14

Closed godfriedmeesters closed 3 years ago

godfriedmeesters commented 3 years ago

In addition to showing multiple versions of cookie popups randomly, Kayak's desktop website also show multiple versions of its website.

For example, to select one-way flight, multiple versions with different nodes are shown randomly. Most of the time, the KayakWebScraper therefore crashes.

When going directly to the url https://www.kayak.de/flights/DUS-BCN/2021-05-14?sort=bestflight_a we get detected as bot and a ReCaptcha is shown. I tried a sweatshop ReCaptcha service (2captcha), however they dont solve German or French ReCaptcha.

Very hard to fix

godfriedmeesters commented 3 years ago

If you can fix this, for sure you deserve a gold medal in scraping. Tried for 5 hours without success.

godfriedmeesters commented 3 years ago

seems working now