I am trying to perform large bulk operations, but I am constantly running into the 40,000 mutation limit.
For instance, instead of db.session.query(table).delete() I must now do
while db.session.query(table).count() > 0:
subquery = db.session.query(table.id).limit(40000).subquery()
db.session.query(table).filter(table.id.in_(subquery)).delete()
which is incredibly slow for large tables (especially when the tables have indexes, and so I must limit by an even smaller chunk size).
https://cloud.google.com/spanner/docs/dml-partitioned
I am trying to perform large bulk operations, but I am constantly running into the 40,000 mutation limit. For instance, instead of
db.session.query(table).delete()
I must now dowhich is incredibly slow for large tables (especially when the tables have indexes, and so I must limit by an even smaller chunk size).