search

dcsil / dream-team

Team repository for Dream Team
https://dreamtune-cdf8a.web.app/auth/login
0 stars 1 forks source link

Backend Data Collection Notes #197

Open RyanMarten opened 4 years ago

RyanMarten commented 4 years ago

One big scrape that fills the database and the front end pulls from that.

Major benefits: super automated scrapes tons of venues and gets all their information PREDICTS tariff value..

Lets flush out a better JSON object together.

We aren't going to worry about API access / crawling access --> when we are bigger we can have formal partnership. Also all the data is public.

We are also going to ASSUME that dataclef has data about which venue is acquired.

Conversation with @VMS-6511

RyanMarten commented 4 years ago

One way of searching yelp" https://www.yelp.com/biz/r_BrIgzYcwo1NAuG9dLbpg/review_feed?rl=en&sort_by=relevance_desc&q=music&start=0

Can use Google for text analysis or do a regex

RyanMarten commented 4 years ago

https://www.yelp.com/biz/pai-northern-thai-kitchen-toronto-5?q=music

^ this is the page that requests the above linl

ribhav99 commented 4 years ago

Scraper for BBB is done. Just testing further and will generalise to fit all different venue cases

ribhav99 commented 4 years ago

Created a new issue #204 Check that out

RyanMarten commented 4 years ago

This is awesome. I'll try running it on my computer

jollyredflames commented 4 years ago

Working on Yelp. See yelp.js in functions for details. Commiting on firebase-functions