tbayart / sql-server-data-producer

Automatically exported from code.google.com/p/sql-server-data-producer
0 stars 0 forks source link

Performance is horrible when inserting lots of rows for a single Execution Item #32

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago
The current implementation to insert multiple rows for one execution item is  
having horrible performance problems.

the UNION ALL method is not good enough.

It should at least be possible to insert one million rows within reasonable 
time.

Test right now showed that it took more than one hour to insert 100 000 rows

Original issue reported on code.google.com by a03pe...@gmail.com on 9 Jun 2012 at 8:10

GoogleCodeExporter commented 9 years ago
Trying to use XML instead of union all

Original comment by a03pe...@gmail.com on 9 Jun 2012 at 8:11

GoogleCodeExporter commented 9 years ago
Using VALUES instead. It is limited to 1000 rows per insert so I limited the 
max RepeatCount on ExecutionItem to be 1000.
If need to insert more than 1000 for one item then the Execution count based 
execution type should be used instead. 

XML had problems of values being hardcoded (ie no use of the SQL query 
generator), and it was still not possible to insert as many rows in one go as I 
wanted. The script file simply became to big.
It might be possible to do this if we choose to use Parameters instead and then 
stream the XML data to the sql server.

Original comment by a03pe...@gmail.com on 9 Jun 2012 at 10:41