Scrape Fermentoren / Untappd

heikkilevanto / beertracker

Simple script to track the beers I drink

GNU General Public License v2.0

2 stars 1 forks source link

Scrape Fermentoren / Untappd #201

Closed heikkilevanto closed 2 years ago

heikkilevanto commented 3 years ago

Fermentoren, and others, use Untappd for their beer list. Would be nice to have a scraper for them. But it is not that simple:

The page defaults to showing the first 12(?) beers, and requires a click to get the rest
Prices seem not to be there

For now I have disabled the fermentoren scraper, and am working on the others.

heikkilevanto commented 3 years ago

Untappd does have an API, but the free tier explicitly does not support getting beer lists for venues. I am not going to register and pay for an API key that would give me that.

heikkilevanto commented 3 years ago

Their terms of use are pretty clear that they do not want me to scrape their site. https://untappd.com/terms

heikkilevanto commented 2 years ago

OTOH, the Fermentoren page just fetches a javascript file that contains the full beer list. https://business.untappd.com/locations/2098/themes/4868/js No API keys required, so it could be publicly available stuff. Extracting the html from the js is not hard, and it looks quite possible to parse.

heikkilevanto commented 2 years ago

Done. There was a bit of messy escaping in the html, as it was expressed as a javascript string, so double quotes etc had to be escaped. Bit of regexp unescaped them all right. The html itself was quite parsable.