greenpeace / gpi-tl-hermes

News Aggregations and Sentiment Analysis app

2 stars 0 forks source link

Cloud function payload exceeding maximal size #27

Closed krauthex closed 5 years ago

krauthex commented 5 years ago

Cloud Function payload exceeding maximal size

Description

When running the hermify.py script, Firebase responds with a 400 Client Error, reproducibly. The reason seems to be that the Cloud Function is triggered by any changes to the Firebase database, but the trigger always transports the whole database snapshot to the Cloud Function, see 1 and 2. So after Firebase has been populated with a certain number of entries / amount of data (a few MB), the full snapshot exceeds the payload data limit for cloud functions.

Proposal

more research on the actual issue, but no imminent idea at the moment to solve this problem.

Edit: see the comment below for a plan.

krauthex commented 5 years ago

Once this is sorted out, a better description of how to use the command line arguments for hermify is needed + cleanup of code/api-stuff.

krauthex commented 5 years ago

New solution approach: replacing the Firebase trigger with an HTTP trigger. This goes as follows:

[x] deactivate the Firebase trigger
[x] in the hermify.py script, add arequests.post(..), that sends the full node path to the new entry (whereas the ID is available by the returned reference-object by the .push method) as json-payload to the HTTP triggered new cloud function.
[x] the new cloud function can then create a reference object, extracting exactly the "delta" part of the database, which in turn can be put into BQ again.