Properly handle when list of feed entry records is more than 1MB in size

#
I 07-11 07:03PM 07.423

Retrieved 179 feed entries, 0 of which have been seen before

#
I 07-11 07:03PM 07.489

Saving 179 new/updated entries

#
E 07-11 07:03PM 07.817

The request to API call datastore_v3.Put() was too large.
Traceback (most recent call last):
  File
"/base/python_lib/versions/1/google/appengine/ext/webapp/__init__.py", line
503, in __call__
    handler.post(*groups)
  File
"/base/data/home/apps/pubsubhubbub/hacky-fetch.334811843024886524/main.py",
line 212, in decorated
    return func(myself, *args, **kwargs)
  File
"/base/data/home/apps/pubsubhubbub/hacky-fetch.334811843024886524/main.py",
line 1525, in post
    db.run_in_transaction(lambda: db.put(entities_to_save))
  File "/base/python_lib/versions/1/google/appengine/api/datastore.py",
line 1718, in RunInTransaction
    DEFAULT_TRANSACTION_RETRIES, function, *args, **kwargs)
  File "/base/python_lib/versions/1/google/appengine/api/datastore.py",
line 1809, in RunInTransactionCustomRetries
    result = function(*args, **kwargs)
  File
"/base/data/home/apps/pubsubhubbub/hacky-fetch.334811843024886524/main.py",
line 1525, in <lambda>
    db.run_in_transaction(lambda: db.put(entities_to_save))
  File "/base/python_lib/versions/1/google/appengine/ext/db/__init__.py",
line 1098, in put
    keys = datastore.Put(entities)
  File "/base/python_lib/versions/1/google/appengine/api/datastore.py",
line 164, in Put
    apiproxy_stub_map.MakeSyncCall('datastore_v3', 'Put', req, resp)
  File
"/base/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py",
line 72, in MakeSyncCall
    apiproxy.MakeSyncCall(service, call, request, response)
  File
"/base/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py",
line 244, in MakeSyncCall
    stub.MakeSyncCall(service, call, request, response)
  File "/base/python_lib/versions/1/google/appengine/runtime/apiproxy.py",
line 183, in MakeSyncCall
    rpc.CheckSuccess()
  File "/base/python_lib/versions/1/google/appengine/api/apiproxy_rpc.py",
line 110, in CheckSuccess
    raise self.exception
RequestTooLargeError: The request to API call datastore_v3.Put() was too large.

Original issue reported on code.google.com by bslatkin on 12 Jul 2009 at 2:34

Tallefer / pubsubhubbub

Properly handle when list of feed entry records is more than 1MB in size #20