search

beanumber / airlines

An R package providing access to medium airline flight delay data
21 stars 36 forks source link

bypass R entirely and send LOAD DATA statement through DBI #15

Closed beanumber closed 8 years ago

beanumber commented 9 years ago

This would improve performance quite a bit, but will everyone have access?

beanumber commented 8 years ago

So maybe this will get done in DBI, but maybe we'll just have to 

dbSendQuery("LOAD DATA ...")
beanumber commented 8 years ago

Duplicate of (https://github.com/beanumber/etl/issues/12)

beanumber commented 8 years ago

The problem is that we do manipulations on the original data. If we move those transformations to the transform phase, then the load phase will be much faster, but the transform phase will be slower. But logically, this is probably the right thing to do. It also means that subsequent loading off the processed data would be much faster.

beanumber commented 8 years ago

Nevermind. This is implemented in 3847c05f