search

dcsil / dream-team

Team repository for Dream Team
https://dreamtune-cdf8a.web.app/auth/login
0 stars 1 forks source link

Investigate Firebase Cloud Functions For Creating APIs and Scraping #148

Open RyanMarten opened 4 years ago

RyanMarten commented 4 years ago

https://firebase.google.com/products/functions?authuser=0

RyanMarten commented 4 years ago

https://firebase.google.com/docs/functions/?authuser=0#how_does_it_work

This looks perfect for what we are trying to do

RyanMarten commented 4 years ago

Here are some example functions provided by the firebase team: https://github.com/firebase/functions-samples

This looks like one of the most relevant use cases: https://firebase.google.com/docs/functions/use-cases?authuser=0#execute_intensive_tasks_in_the_cloud_instead_of_in_your_app

Execute intensive tasks in the cloud instead of your app

This would be our scraping of data

RyanMarten commented 4 years ago

This web crawler and search service from Algolia looks interesting and promising: https://www.algolia.com/products/crawler/

Firebase has a tutorial on how to use Algolia on their Realtime Database service https://firebase.google.com/docs/functions/use-cases?authuser=0#execute_intensive_tasks_in_the_cloud_instead_of_in_your_app

RyanMarten commented 4 years ago

https://medium.com/@vinaymavi/serverless-web-scrapping-with-google-cloud-function-24d5f1e24442

@jollyredflames

RyanMarten commented 4 years ago

https://console.firebase.google.com/u/0/project/dreamtune-cdf8a/functions =---> Console

RyanMarten commented 4 years ago

For external calls you need to use the Blaze plan: https://firebase.google.com/pricing/ https://stackoverflow.com/questions/43415759/use-firebase-cloud-function-to-send-post-request-to-non-google-server

The first 2,000,000 invocations, 400,000 GB-sec, 200,000 CPU-sec, and 5 GB of Internet egress traffic is provided for free each month.

RyanMarten commented 4 years ago

I'll upgrade the account now, I will just have to put limits on it so it doesn't charge me :(

RyanMarten commented 4 years ago

Creating budgets: https://firebase.googleblog.com/2018/03/adding-free-usage-to-blaze-pricing-plan.html

RyanMarten commented 4 years ago
Screen Shot 2020-03-24 at 11 42 22 PM

No spending :)

RyanMarten commented 4 years ago

Can use a local emulator for Firebase Functions so you don't use up the cloud calls: firebase emulators:start

See https://firebase.google.com/docs/functions/local-emulator for more!

RyanMarten commented 4 years ago

@ribhav99 the demo is not working entirely for me. You might want to try a different way of getting the text out of the HTML. Jquery might be helpful, im sure that there are web scraping libraries that also do this work.