The subsetter spends a significant chunk of overall running time on
single-row inserts into the destination tables. This patch accumulates
rows in the pending attribute on each table, then periodically flushes
updates to the database when the number of accumulated rows exceeds the
buffer argument. Depending on the value of buffer, this change
reduced running time by 20-30% on a sample database.
This change assumes that rdbms-subset is run against an empty destination database--if this isn't the case, the existence check on line 242 can be incorrect, since it isn't actually making a call to the database (which also saves a lot of time!). I can revise the patch to load existing primary keys from the destination database before inserting, which I think would take care of this potential issue.
The subsetter spends a significant chunk of overall running time on single-row inserts into the destination tables. This patch accumulates rows in the
pending
attribute on each table, then periodically flushes updates to the database when the number of accumulated rows exceeds thebuffer
argument. Depending on the value ofbuffer
, this change reduced running time by 20-30% on a sample database.This change assumes that
rdbms-subset
is run against an empty destination database--if this isn't the case, the existence check on line 242 can be incorrect, since it isn't actually making a call to the database (which also saves a lot of time!). I can revise the patch to load existing primary keys from the destination database before inserting, which I think would take care of this potential issue.