search

WinVector / rqdatatable

Implement the rquery piped query algebra in R using data.table. Distributed under choice of GPL-2 or GPL-3 license.
https://winvector.github.io/rqdatatable/
Other
37 stars 3 forks source link

I don't know how the ex_data_table works for a remote database relop operations tree #2

Closed konny0201 closed 6 years ago

konny0201 commented 6 years ago

vv_cid_day_table=db_td(con_dm_result,'vv_cid_day') vv_bid_day_info_tree <- vv_cid_day_table %.>% select_rows_nse(.,date>=startday_parameter & date<=endday_parameter ) %>.% project_nse(.,groupby = c('date','bid'),vv:=sum(vv)) %.>% extend_nse(.,group_bid:=ifelse(bid==2|bid==102,2,ifelse(bid==4|bid==104,4,ifelse(bid==5|bid==105,5,bid)))) %.>% project_nse(.,groupby=c('date','group_bid'),vv_commercial:=max(vv),vv_nocommercial:=min(vv) ) e=ex_data_table(vv_bid_day_info_tree,tables=list(vv_cid_day_table=vv_cid_day_table) )[]

JohnMount commented 6 years ago

ex_data_table() is only for local data. I'll improve the error msgs.

konny0201 commented 6 years ago

Thank you~In this case can I use the wrapr::execute_parallel() function?

JohnMount commented 6 years ago

Combining wrapr::execute_parallel() with remote tables is not recommended and not a scenario we are supporting. The issue is each worker would need a different connection the remote database (such connections should not be compared over sockets) and likely the remote system would not handle well many connections from the same R to the same tables anyway.