Open nviraj opened 12 months ago
I did some digging. The warning is referring entirely to an issue in dbplyr itself. It essentially saying "the developers of dbplyr haven't yet written a backend for JDBC that is up to their own current standards".
dbplyr_fallback
.dbplyr_edition
is called here, ideally there's supposed to be an edition 2 backend implemented. If an edition 2 backend is implemented, for example in MySQL, the S3 method for dbplyr_edition
returns 2, and dbplyr_fallback
doesn't generate a warning. dbplyr_edition
dispatches to the default method which returns 1 and triggers the else in this if-else and gets you the warning.Currently, dbplyr doesn't support JDBC but only ODBC. @shearerpmm @nviraj does any of you maybe want to create a PR that adds JDBC support?
I actually would really like to. I'm not sure how big of a lift it is for me, my experience in developing R packages is pretty limited.
Despite not having a specific backend, I've found that dbplyr already works fantastically well over JDBC. So far the only issue I've found is due to a problem in RJDBC, not dbplyr.
There are some places where the generated SQL doesn't work, but I'm not sure that can be solved with a JDBC backend, because JDBC is just a way to connect to a variety of databases with different SQL dialects. In my recent experience, I find SQL generation glitches can be easily routed around by using sql()
and manually coding anything dialect-specific.
From this perspective, I'm not sure there a JDBC backend has anything of value to do other than silence this warning. And maybe warn the user that compute()
currently doesn't work due to the RJDBC issue mentioned above.
@shearerpmm I think those are still valuable contributions 😄
The little I'm able to do is worth my time. IMHO dbplyr is the best tool available today for building complex data pipelines in SQL databases. I know about the alternatives and the competition is not close.
Big +1 on this request and dbplyr being invaluable
I've had to jump over to RJDBC to handle testing an enterprise db migration from an old DB2 to a new SQL Server. i would have used odbc connections, but the recent issues with odbc and db2 meant we were stuck using R 4.1.1 and thus hit memory limits etc (dealing with many tables that have tens of millions of entries and a dozen rows)
there are definitely still a lot of db2 databases out there and I imagine that other folks have had to do similar transitions if they wanted to update R itself, so would expect that there's likely a pick-up in need compared to a few years ago when you could use any old ODBC to connect to a db2 so wouldn't hit this problem.
so far i've been able to generally just hot-swap the source between the db2 and sqlserver but i've just run into this issue of not having a "proper" backend as i've been trying to push more compute towards the database. in particular, i'm trying to do things like inlining tables to avoid collecting the massive dataset just to do a join (rather, i'd like to inline the table and then join it before running a collect statement), and this is what isn't working without the backend. (getting the same warning as OP posted).
i'm not really adept at the backend coding, although i would be up for learning to support this, but i definitely have multiple use cases and would be happy to test things in the wild or provide reprexes if either would be helpful
@juniperlsimonis what are the problems with odbc and DB2? Could you file an issue in https://github.com/r-dbi/odbc?
@hadley see ibm's support page there have been multiple issues in odbc that were filed like this one . the solutions folks have been able to implement haven't worked on my end given enterprise limitations.
To get rJava working on my arm64 mac:
brew install openjdk
# follow sudo ln advice
sudo R CMD javareconf
I don't think there's much dbplyr can do with RJDBC since it doesn't appear to expose a way to figure out the underlying database type from the driver/connection:
library(RJDBC)
#> Loading required package: DBI
#> Loading required package: rJava
# Download from https://github.com/xerial/sqlite-jdbc
class_path <- c(
"~/Downloads/sqlite-jdbc-3.45.1.0.jar",
"~/Downloads/slf4j-api-1.7.36.jar"
)
con <- DBI::dbConnect(
JDBC("org.sqlite.JDBC", class_path),
url = "jdbc:sqlite::memory:"
)
dbGetInfo(con)
#> list()
Created on 2024-02-22 with reprex v2.1.0
Since the user generally knows, could they have the option to tell dbplyr what the database is when the connection is JDBC? I thought all it really affects is the SQL dialect used?
@shearerpmm I think we could in principle do that, but the idea that the connection defines the database type is deeply intwined in the guts of dbplyr, so fixing that would be quite a lot of work.
Yes, it only affects the SQL dialect used but given that dbplyr is all about converting R code to SQL, that's really important!
Hi,
I am using “pool” with “RJDBC” and “dbplyr” packages to connect to Athena tables. However I see the following warning.
This has been slightly tricky to pinpoint. Initially I thought it had something to do with RJDBC but it does not seem to be the case, as seen here. I subsequently reported the issue to Pool, but it does not seem to be originating from there as well as I could replicate the issue using a DBI connection as well as Pool.
It was easier to generate a reprex using PostgreSQL Docker and JDBC, and the issue does not seem to be database specific. Please let me know if I am doing something incorrectly or need to report this elsewhere.
Steps to reproduce:
1. Create docker image to connect to using the official image:
docker run --network=host --name postgresdb_reprex -e POSTGRES_PASSWORD=mysecretpassword -e POSTGRES_USER=postgres -d postgres
2. Download JAR file from here Alternatively you can use the attached Project which contains the script and the JAR file for Windows. pool_issue.zip
3. Run reprex code reprex.R contains the actual code. Running through
reprex::reprex(input = "reprex.R", venue = "gh", session_info = TRUE)
Reprex Content:
Created on 2023-09-04 with reprex v2.0.2
Session info
``` r sessioninfo::session_info() #> ─ Session info ─────────────────────────────────────────────────────────────── #> setting value #> version R version 4.3.0 (2023-04-21 ucrt) #> os Windows 11 x64 (build 22621) #> system x86_64, mingw32 #> ui RTerm #> language (EN) #> collate English_India.utf8 #> ctype English_India.utf8 #> tz Asia/Calcutta #> date 2023-09-04 #> pandoc 3.1.1 @ C:/Program Files/RStudio/resources/app/bin/quarto/bin/tools/ (via rmarkdown) #> #> ─ Packages ─────────────────────────────────────────────────────────────────── #> ! package * version date (UTC) lib source #> blob 1.2.4 2023-03-17 [1] CRAN (R 4.3.1) #> cli 3.6.1 2023-03-23 [1] CRAN (R 4.3.1) #> DBI * 1.1.3 2022-06-18 [1] CRAN (R 4.3.1) #> dbplyr 2.3.3 2023-07-07 [1] CRAN (R 4.3.1) #> digest 0.6.33 2023-07-07 [1] CRAN (R 4.3.1) #> dplyr * 1.1.2 2023-04-20 [1] CRAN (R 4.3.1) #> evaluate 0.21 2023-05-05 [1] CRAN (R 4.3.1) #> fansi 1.0.4 2023-01-22 [1] CRAN (R 4.3.1) #> fastmap 1.1.1 2023-02-24 [1] CRAN (R 4.3.1) #> fs 1.6.3 2023-07-20 [1] CRAN (R 4.3.1) #> generics 0.1.3 2022-07-05 [1] CRAN (R 4.3.1) #> glue 1.6.2 2022-02-24 [1] CRAN (R 4.3.1) #> htmltools 0.5.6 2023-08-10 [1] CRAN (R 4.3.1) #> knitr 1.43 2023-05-25 [1] CRAN (R 4.3.1) #> later 1.3.1 2023-05-02 [1] CRAN (R 4.3.1) #> lifecycle 1.0.3 2022-10-07 [1] CRAN (R 4.3.1) #> magrittr 2.0.3 2022-03-30 [1] CRAN (R 4.3.1) #> pillar 1.9.0 2023-03-22 [1] CRAN (R 4.3.1) #> pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.3.1) #> pool * 1.0.1 2023-02-21 [1] CRAN (R 4.3.1) #> purrr 1.0.2 2023-08-10 [1] CRAN (R 4.3.1) #> R.cache 0.16.0 2022-07-21 [1] CRAN (R 4.3.1) #> R.methodsS3 1.8.2 2022-06-13 [1] CRAN (R 4.3.0) #> R.oo 1.25.0 2022-06-12 [1] CRAN (R 4.3.0) #> R.utils 2.12.2 2022-11-11 [1] CRAN (R 4.3.1) #> R6 2.5.1 2021-08-19 [1] CRAN (R 4.3.1) #> Rcpp 1.0.11 2023-07-06 [1] CRAN (R 4.3.1) #> reprex 2.0.2 2022-08-17 [1] CRAN (R 4.3.1) #> D rJava * 1.0-6 2021-12-10 [1] CRAN (R 4.3.0) #> RJDBC * 0.2-10 2022-03-24 [1] CRAN (R 4.3.1) #> rlang 1.1.1 2023-04-28 [1] CRAN (R 4.3.1) #> rmarkdown 2.24 2023-08-14 [1] CRAN (R 4.3.1) #> rstudioapi 0.15.0 2023-07-07 [1] CRAN (R 4.3.1) #> sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.3.1) #> styler 1.10.2 2023-08-29 [1] CRAN (R 4.3.1) #> tibble 3.2.1 2023-03-20 [1] CRAN (R 4.3.1) #> tidyselect 1.2.0 2022-10-10 [1] CRAN (R 4.3.1) #> utf8 1.2.3 2023-01-31 [1] CRAN (R 4.3.1) #> vctrs 0.6.3 2023-06-14 [1] CRAN (R 4.3.1) #> withr 2.5.0 2022-03-03 [1] CRAN (R 4.3.1) #> xfun 0.40 2023-08-09 [1] CRAN (R 4.3.1) #> yaml 2.3.7 2023-01-23 [1] CRAN (R 4.3.0) #> #> [1] C:/Users/viraj/AppData/Local/R/win-library/4.3 #> [2] C:/Program Files/R/R-4.3.0/library #> #> D ── DLL MD5 mismatch, broken installation. #> #> ────────────────────────────────────────────────────────────────────────────── ```4. Stop and remove Docker image
Please let me know if you need any further details. Thanks!