create a gcp cloud function to clone gists

[micahstubbs]

questions

• [ ] can we run a shell script from inside a cloud function?
• [x] how does github API rate limiting interact with cloud functions?

[enjalot]

Let's try using PubSub to trigger the functions (see Tutorial)

We can send messages like this {usernames: ['enjalot', 'micahstubbs'], since: "2017-09-30T20:34:21.648Z" } to the PubSub channel with subsets of all our users and have the Cloud Functions figure out how to scale horizontally to handle it.

Potential query rules around time: since could be 15min, 20min or an ISO datetime string. If its nothing we get all blocks. Follows GitHub's query parameter for listing gists.

[enjalot]

Following tutorial: Made the blockbuilder-search-function-staging storage bucket to stage our functions.

Deploy the function gcloud functions deploy getGists --trigger-topic get-gists --memory 1024

Trigger the function gcloud pubsub topics publish get-gists --message '{"users":["micahstubbs","enjalot"]}'

[enjalot]

Developing with the cloud functions emulator

functions deploy getGists --trigger-topic get-gists
functions call getGists --data='{"users":["ivyywang","tarekrached"]}'
functions logs read

[enjalot]

note: biggest constraint is probably memory, writing to /tmp uses memory.

[enjalot]

I pushed some changes, and updated the above comments to be in step with the latest in cloud functions.

re: API rate limit, cloud functions don't get us around it, but we can rotate oath keys they use whenever they run out.

I made it so the getGists function now saves the gists it fetches in the gists namespace as gist entities. I had to modify the gist metadata slightly, to remove . from filenames (replaced with a | to make the Datastore API happy). I also store the username and github id of the user as owner_login and owner_id and delete the owner object. this way its easy to filter by the username in datastore.