tortoise / tortoise-orm

Familiar asyncio ORM for python, built with relations in mind
https://tortoise.github.io
Apache License 2.0
4.37k stars 355 forks source link

Bulk Create without returning objects #1614

Open Abdeldjalil-H opened 1 month ago

Abdeldjalil-H commented 1 month ago

The main problem here is that bulk_create converts objects to a list before insertion which loads all the models at the same time. The only reason for this is to return the inserted objects after the query is done. However, those objects are not being updated (adding pk for example), i.e the mothod will return what you sent. I think we can enhance this and make bulk_create benefits from #1613 .

Describe the solution you'd like There are several approaches for this:

  1. Do not return anything from bulk_create or return the number of inserted rows. This maybe a breaking change.
  2. Add a keyword argument called return_objects or something like that to control the behaviour.
  3. Do not return anything when batch_size is set.

If none of the above seems logical, I think at least we should mention what is going behind the scenes in the docs, so the user can handle chunking on his own. And maybe use tuple(objects) instead of list(objects).

Describe alternatives you've considered The most natural alternative is to chunk data on your side and leave batch_size as None. But this makes batch_size useless.

abondar commented 1 month ago

Hi!

I think ideally would be to support id population for backends that allow it to do in same query - as with RETURNING in postgres But not sure I will be able to do it in any near time

As right now returning objects doesn't bring any benefits, as we return what we got in - I don't see much value in this return and we can change it

I think most explicit way to do that - would be for now just removing that return object, allowing to iterate through models without loading them all in memory

Next step could be adding return_objects param and allowing it only for supported databases, where we would return updated objects from db in same query

Abdeldjalil-H commented 1 month ago

Hi @abondar . I can make a PR on that, removing the returned object.

abondar commented 1 month ago

If you could - I would gladly help you with reviewing and merging it