search

chop-dbhi / dataexpress

[NOT MAINTAINED] DataExpress is a simple, Scala-based cross database ETL toolkit supporting Postgres, MySql, Oracle, SQLServer, and Sqlite
http://dataexpress.research.chop.edu
BSD 2-Clause "Simplified" License
72 stars 9 forks source link

Use insert_rows To.class #12

Closed masinoa closed 11 years ago

masinoa commented 12 years ago

In the To.class in the create method, the /: op inserts data one row at a time to the target using the insert_row method in DataWriter. This should (1) Check to see of the transform functions are empty, in which case the insert_table method should be used. (2) If the transforms are not empty, there is probably a way to use insert_rows instead of insert_row by collecting up some fo the rows before applying the transform.

masinoa commented 11 years ago

If transform functions are empty, create and append methods use insert_rows(table) method in API. If transform functions are not empty: Added insertRows(rows: Iterable[DataRow]) method to API, which is now used by DSL in To.class.

Both situations lead to use of batch inserts at 50 rows at time.