search

tehhowch / apps-script

A collection of files used by Google Apps Script to develop and serve data via Google Sheets
0 stars 0 forks source link

FusionTables.replaceRows returns empty response #7

Closed tehhowch closed 6 years ago

tehhowch commented 7 years ago

When the database grows to a certain size (hypothesis), calls to .replaceRows() fail with an empty response. This situation has arisen twice with MHCC so far (May 18th & April 22-26ish) since the implementation of the FusionTables interface.

This empty response occurs when running the permissive maintenance function, resulting in a halt to scoreboard updates.

tehhowch commented 7 years ago

Current workaround:

  1. Run restrictive maintenance function (doBookending)

Eventually this too will fail, or will need to be done so often that the permissive maintenance function will be worthless to use.

Possible fixes:

  1. Actually diagnose the issue and get it fixed

or

  1. Detect the error
  2. Split the upload into multiple parts
  3. Run DeleteRows() to delete all rows
  4. Upload the individual small parts.
tehhowch commented 7 years ago

Still happening regularly - is definitely related to the size of the upload. hypothesis: It's possible the hang is because the call is not async - the script waits around to receive the status, and times out before it gets the status, and thus (whether or not the task was successful), fails to update the rest of the system about the successful clean.

tehhowch commented 6 years ago

Making the doReplace return void isn't helpful. It appears the empty response is due to using the innate Google Apps Script FusionTables API and having a large CSV to upload (large being >= 10MB). The Google Apps Script FusionTables API naturally does not provide any documentation at all, beyond its very limited, example-free "intellisense" (instead referring to the FusionTables v2 API). The FusionTables API does describe multipart & resumable uploads, but provides only non-apps-script examples. Either the Google Apps Script FusionTables API does not support multipart/resumable, or it does but no one at Google has cared to document it (equally likely IMO).

My solution going forward is to develop a standalone prune tool (likely a python or JS browser script) that can be executed on my own resources and can utilize the latest available language features (and the full FusionTables v2 API).

tehhowch commented 6 years ago

With https://github.com/tehhowch/apps-script/commit/a1c267e3c651d78586db3c4358b72b08e34d6931 I am now able to perform maintenance actions (mass deletions & then importrow) for a csv that is ~14000 kB on disk without issue, even without using a resumeable request. At some point that'll need to be done, but that point isn't now.