search

godfriedmeesters / scraper

As part of DiffScraper, one or more bots can be deployed. Ready-to-use bots are provided that can extract offers from mobile applications, mobile websites and desktop websites.
GNU General Public License v3.0
2 stars 0 forks source link

Add mobile website scraper for one vendor #4

Closed bkrumnow closed 3 years ago

bkrumnow commented 3 years ago

We would like to compare a mobile website scraped from a real phone with a desktop version of a site.

Currently in favor: Opodo

godfriedmeesters commented 3 years ago

Added KayakAppBrowserScraper.

First tried with Opodo, but the scraper crashed unpredictably many times.

godfriedmeesters commented 3 years ago

done

bkrumnow commented 3 years ago

Great, do you have one example data set scraped? I would love to visit it once and check if everything is there. Then, I would close the ticket

godfriedmeesters commented 3 years ago

[{"price":"83 €","departureTime":"9:55","arrivalTime":"11:25","origin":"CDG","destination":"FRA","airline":"Air France","screenshot":"https://scraperbox.be/screenshots/KayakAppBrowserScraper-1616016598435.png"},{"price":"63 €","departureTime":"14:25","arrivalTime":"15:35","origin":"CDG","destination":"FRA","airline":"Lufthansa","screenshot":"https://scraperbox.be/screenshots/KayakAppBrowserScraper-1616016598435.png"},{"price":"63 €","departureTime":"11:20","arrivalTime":"12:35","origin":"CDG","destination":"FRA","airline":"Lufthansa","screenshot":"https://scraperbox.be/screenshots/KayakAppBrowserScraper-1616016598435.png"},{"price":"63 €","departureTime":"9:25","arrivalTime":"10:40","origin":"CDG","destination":"FRA","airline":"Lufthansa","screenshot":"https://scraperbox.be/screenshots/KayakAppBrowserScraper-1616016598435.png"},{"price":"63 €","departureTime":"7:00","arrivalTime":"8:15","origin":"CDG","destination":"FRA","airline":"Lufthansa","screenshot":"https://scraperbox.be/screenshots/KayakAppBrowserScraper-1616016598435.png"},{"price":"83 €","departureTime":"9:55","arrivalTime":"11:25","origin":"CDG","destination":"FRA","airline":"Air France","screenshot":"https://scraperbox.be/screenshots/KayakAppBrowserScraper-1616016598435.png"},{"price":"93 €","departureTime":"18:25","arrivalTime":"19:35","origin":"CDG","destination":"FRA","airline":"Lufthansa","screenshot":"https://scraperbox.be/screenshots/KayakAppBrowserScraper-1616016598435.png"},{"price":"134 €","departureTime":"20:45","arrivalTime":"22:05","origin":"CDG","destination":"FRA","airline":"Air France","screenshot":"https://scraperbox.be/screenshots/KayakAppBrowserScraper-1616016598435.png"}]

godfriedmeesters commented 3 years ago

ts-node cli.ts scrape KayakAppBrowserScraper .\inputData.json

godfriedmeesters commented 3 years ago

Note that only German is supported

bkrumnow commented 3 years ago

Just tested it on the new phone. Looks good so far.