Open hannes opened 8 years ago
@ajdamico , 👍!
could use dataflow for this
just to clarify: data import parallelization seems most useful as monetdb/mserver improvement and not a monetdblite-specific enhancement? thanks
no in this case its lite-specific because we need to convert sexp's to monetdb columns
way to go here is to add column conversion for query results / appends to MAL plan. Need to come up with reasonable C interface for this.
can't really do this for query results, most time is spent in string columns and those need to be single-threaded because of R's global string hash table...
I like the non-blocking dbSendQuery
idea described here: https://github.com/r-dbi/DBI/issues/69
A typical interaction could look like this:
q1 <- dbSendQuery(c1, "COPY INTO t1 ...")
q2 <- dbSendQuery(c2, "COPY INTO t2 ...")
r1 <- dbFetch(q1)
r2 <- dbFetch(q2)
The two queries would run at the same time, but dbFetch
would only return if the result of the corresponding query (in this case an "OK" is available. Comments?
We could probably parallelise dbWriteTable, in particular value conversions. Also on the query side. Because every column is essentially independent. Actually thats a great idea, will probably add this at some point.
importing ten large tables on ten cores is 7x as fast with a monetdb session that allows multiple connections. until monetdblite can compete with multicore imports, monetdb server control code should not be deprecated.
reproducible script below
Results: