pgsql-io / multicorn2

http://multicorn2.org
Other
73 stars 16 forks source link

Add support for batch insert #45

Closed mfenniak closed 1 month ago

mfenniak commented 2 months ago

Adds support for two new APIs on FDWs in order to support bulk inserts:

Property modify_batch_size, which defaults to 1 and disables bulk inserts, and the bulk_insert method:

    def bulk_insert(self, all_values):
        """
        Insert multiple tuples defined by ''values'' in the foreign table.
        Will only be invoked if modify_batch_size is a value greater than 1.

        Args:
            all_values (list of dicts): a dictionary mapping column names to
                column values
        Returns:
            A list of dictionaries containing the new values. These values can
            differ from the ``values`` argument if any one of them was changed
            or inserted by the foreign side -- For example, if a key is auto
            generated.

            Note that in PG14-PG16, the return value is only used if the INSERT
            statement involves a view WITH CHECK OPTIONS, or if the foreign
            table has an AFTER ROW trigger; notably it is not used a RETURNING
            clause is present in the INSERT statement.  If the FDW wants to
            support RETURNING on inserts, it must set modify_batch_size to 1
            and avoid using the bulk insert api.
        """
luss commented 1 month ago

This is looking good Mathieu. Let me know when you want me to merge (or u can when u are comfortable)