search

bfatemi / sqlsauce

Utility functions built on top of the package RODBC to facilitate an efficient SQL + R workflow
Other
0 stars 0 forks source link

Implement "rows-at-a-time" for very large queries #5

Open bfatemi opened 8 years ago

bfatemi commented 8 years ago

Using rodbc:

res <- sqlFetch(ch, "table name", max = m) res <- sqlFetchMore(ch, "table name", max = m)

bfatemi commented 8 years ago

Requires a locked connection handle.

Can also spawn new R processes to parallelize the work flow: query -> process -> save results - process next -> save results... (implement a map-reduce algorithm for SQL + R data processing and analysis).