googleapis / python-spanner-sqlalchemy

Apache License 2.0
38 stars 28 forks source link

Do you support partitioned DML? #304

Closed trevor-pope closed 1 year ago

trevor-pope commented 1 year ago

https://cloud.google.com/spanner/docs/dml-partitioned

I am trying to perform large bulk operations, but I am constantly running into the 40,000 mutation limit. For instance, instead of db.session.query(table).delete() I must now do

while db.session.query(table).count() > 0:
    subquery = db.session.query(table.id).limit(40000).subquery()
    db.session.query(table).filter(table.id.in_(subquery)).delete()

which is incredibly slow for large tables (especially when the tables have indexes, and so I must limit by an even smaller chunk size).

asthamohta commented 1 year ago

Hey @trevor-pope, we currently don't support partitioned DML for SqlAlchemy