pagination for all endpoints using batch operations

[brainysmurf]

Perhaps put this on a different branch and test before merging to master. No additional setup required, all that is needed is that the Plus endpoints enabled in the google console for the project. Note that this code does not technically need the Plus variable defined in Advanced Google Services, but it won't conflict if it is.

After 2 executions:

• Retrieved 3092 posts
• Retrieved 12533 replies
• Retrieved 5428 plusoners
• Retrieved 461 reshares

But then I hit the daily quota limit.

[brainysmurf]

By manually migrating the script properties (by making copy of the script and manually updating the new script's properties), I have downloaded 10,000 posts in 1/2 hour.

[brainysmurf]

Whoops. I believe for example the comments endpoint, I need to put the maxResults property inside an object and pass that to query property.

Will get to it when in front of a desktop.

[brainysmurf]

Yeah, I was right; fixed now. Those endpoints have 100 as the max for maxResults.

• Retrieved 4345 posts
• Retrieved 18501 replies
• Retrieved 7234 plusoners

That is after one run, hit quota on second run, after which:

• Retrieved 4723 posts
• Retrieved 20227 replies
• Retrieved 7951 plusoners

[brainysmurf]

Update: I've confirmed I'm hitting the 10K daily limit on a single execution. Coding it up now so that we approach but don't exceed, then ends. Suggest creating trigger to execute 24 hours later.

EDIT: Just realized: The pageToken will probably be invalid 24 hours later! Hmmmm....

[RomainVialard]

@brainysmurf I've done several exports over multiple days without issues regarding the validity of the pageToken, you shouldn't have issues with that.

[brainysmurf]

Oh okay. But it made me think hard about not depending on the pageToken between executions. I realize that we can probably get the user to export out everything in one sitting with the following:

• Don’t use nextPageToken from previous run, instead just start from scratch and only pick up posts that haven’t already been saved in fb.
• User manually runs it once, until the daily quota is used up, it ends with email to user, instructing them to make another copy of their project and rerun
• By making a project the quota is essentially reset, run again manually
• As above, it starts from scratch and “fast forwards” to where it left off, continues until daily quota is hit or no more left
• With that, it takes about 15 minutes to download all of the community

[RomainVialard]

How would you "pick up posts that haven’t already been saved in fb" without the nextPageToken?

[brainysmurf]

Testing shows that the fast forward approach is just too slow; I’ll put nextPageToken in fb instead. By avoiding the use of script properties, user can just make copy of project to finish off the downloading.

[brainysmurf]

I've refactored Code.gs entirely with improvements from this branch; closing in preference for refactor branch.