googleapis / python-spanner-sqlalchemy

Apache License 2.0
38 stars 28 forks source link

Do you support batched insert operations? #305

Closed trevor-pope closed 1 year ago

trevor-pope commented 1 year ago

https://cloud.google.com/spanner/docs/modify-mutation-api

Since Partitioned DML doesn't support insertions, the best we can do is batching inserts. I am trying to avoid the 40000 mutation limit, and doing the batching on my end is not very performant. For instance:

for batch in range(n_batches):
    new_users = [User(str(uuid.uuid4())) for _ in range(40000)]
    db.session.bulk_save_objects(packages)
    db.session.commit()
IlyaFaer commented 1 year ago

@trevor-pope, batches are not supported by sqlalchemy-spanner dialect, but it allows to go to the underlying API, where Spanner batches methods are available. See the comment: https://github.com/googleapis/python-spanner-sqlalchemy/issues/301#issuecomment-1410136206

I'm investigating how we can make Spanner batches work with SQLAlchemy, but no ETAs for now.