Open talegari opened 1 month ago
Thanks. This is a tight balance -- how much SQL generation should DBI support. I see how it would be useful to have this "just work", at the same time, I don't want to take too much responsibility regarding SQL generation.
The specific cases where compute()
doesn't work -- are there corresponding issues in the dbplyr repo?
@krlmlr
dbplyr
does. Simply put, DBI
needs to accept a query like it accepts a dataframe.compute()
is supposed to fail (gracefully) in some cases. AFAIK, Hadley/dbplyr visualizes compute()
more like a exploratory tool rather than a serious "write to database" tool. So, not ideal for production usecases.One example is spark sql where we get a meaningful error.
IMHO, DBI
has worked like a hard button (DBI::dbExecute
) and it is the last resort for what a R-user can do from within R. Whatever did not work with compute()
was accomplished this way successfully:
# this works
tile_details_tbl %>%
sql_render() %>%
paste("CREATE TABLE sandbox.srikanthks.tile_data AS", .) %>%
DBI::dbExecute(con, .)
This brings us exactly to:
Now that
DBI
already does it, should it support the functionality viadbWriteTable
(most intuitive place a user would for).
As an individual, I can wrap the previous chunk within a function and make my workflow clean, but I think, getting this as a part of dbWriteTable
makes me feel empowered!
The "CREATE TABLE sandbox.srikanthks.tile_data AS"
part is what I refer to as SQL generation.
The dbplyr message seems to originate here: https://github.com/tidyverse/dbplyr/pull/1379/files#diff-f099d8ec45544adf5c40196344d55c458564434dbaa5d7ad090510f33d5cf63bR115 .
@hadley: do you remember why you added the early exit for compute(temporary = FALSE)
in Spark?
@krlmlr I think that original hack was due to a misunderstanding. I wouldn't replicate it.
@hadley: Thanks. Does this mean that you'd accept a PR that fixes compute(temporary = FALSE)
on Spark?
Yes.
This is one of my typical workflow:
con (say odbc) --> tbl --> some_dplyr_ops --> save (as a table)
. Note that data does NOT come into the R's memory. But the code does look clumpsy:This could be (intuitively) replaced by:
Right now,
value
is understood to be adata.frame
in R session's memory. In some cases, some database backends allowvalue
to be a filename too.Q1. Would overloading
DBI::dbWriteTable
to cover this case add value without breaking the semantics? Q2. Having a new generic sayDBI::dbWriteQuery
make sense?PS:
dbplyr::compute
does this job is most cases. In practice, it has hit roadblocks like "only temporary tables are supported" (based on DB backend) whereas rawDBI::dbExecute
gets the job done.value = <sql>
should be atDBI
level, notdbplyr
.