Firebase as intermediate storage + deduplication

[krauthex]

Firebase RT DB as intermediate storage and preprocessor for BigQuery

Description

It's easier to check for duplicates in the RT database because SQL in BigQuery is a pain in the neck. So the workflow will work like the following: news API request --> nappyTools.Content instance --> GNL API request via Content --> Firebase --> CloudFunction + Trigger metadata --> BigQuery.

Proposal

• [x] Uploading the cloudFunction relevant files (main.py and requirements.txt in tl-hermes/python/BigQuerify) to GCP and test them.

• [x] A python script orchestrating the first 4 steps of the above flowchart.

  • [x] Scraping data from somewhere using any kind of news/social media API (tbd @wrongpin )
  • [x] Using the results from 1. to create a Content instance.
  • [x] Check if the data is already in RT database (dedup)
  • [x] Make the Google Natural Language API call.
  • [x] Use the firebase interface (tl-hermes/python/firebase_interface.py) to send the output of Content.jsonify to Firebase.

How to test the implementation

Do the above workflow using the python script mentioned above with real data.

Related Issues

[krauthex]

The cloud function has somehow a problem in unpacking the event dictionary - need to take a closer look at this. If only re-deploying cloud functions wouldn't take forever...

[krauthex]

The cloud function has somehow a problem in unpacking the event dictionary - need to take a closer look at this. If only re-deploying cloud functions wouldn't take forever...

welp, nailed it. ¯_(ツ)_/¯