Closed josephramsay closed 11 years ago
Using a Memory datasource addresses this but makes the conversion operation memory dependent and negates some of the benefits of pagination. File datasources have other issues including edit-ability and data truncation
Results of testing. 1.Driver copy using memory datasource; fastest but gives out-of-memory exceptions on large datasets. This is probably only an issue on memory constrained systems.
This selection is left to the user who can modify the MISC/temptable parameter in the main/user config file
NB. 772 = NZ Primary Parcels
Size/Speed test comparison v:x783 (5648 rows) : Memory DS - 2328kB/5.0s, Direct DS - 2504kB/6.9s v:x794 (1040424 rows) : Memory DS - 313MB/11m52s, Direct DS - 344MB/11m47s
Since DB column deletes leave columns intact but hidden (until vacuum) we should avoid performing these operation on client destination tables.