This commit adds an optional callback argument to the ChunkedInsert
object. This callback is a callable object which gets called before the
chunked insert happens. This is useful for clearing any local caches
that may be in place to deal with the eventual consistency resulting
from the delayed nature of the chunked inserts.
For example,
cache = set()
chunked_table = ChunkedInsert(table, callback=lambda queue: cache.clear())
while True:
data = get_data_id()
key = data['key']
if key in cache or table.find_one(key=key)
continue
cache.add(key)
chunked_table.insert(data)
This commit adds an optional
callback
argument to theChunkedInsert
object. This callback is a callable object which gets called before the chunked insert happens. This is useful for clearing any local caches that may be in place to deal with the eventual consistency resulting from the delayed nature of the chunked inserts.For example,