DyfanJones / noctua

Connect R to Athena using paws SDK (DBI Interface)
https://dyfanjones.github.io/noctua/
Other
45 stars 5 forks source link

Method to set unload at a package level #174

Closed DyfanJones closed 2 years ago

DyfanJones commented 2 years ago

Currently noctua supports AWS Athena unload method through adhoc setting of parameter unload in dbGetQuery , dbSendQuery and dbExecute.

Unfortunetly this means that dplyrfunctionality doesn't benefit from this feature. By having a method to set unload = TRUE on a package level then dplyr can take advantage of this feature with little implementation on dplyr side of things.

DyfanJones commented 2 years ago

Implementation seems to be working nicely :D

library(DBI)
library(noctua)
library(dplyr, warn.conflict = FALSE)

con <- dbConnect(athena())

noctua_options(cache = 10, unload = TRUE)

df = dbGetQuery(con, "select * from awswrangler_test.noaa")

noaa = tbl(con, dbplyr::in_schema("awswrangler_test", "noaa"))

noaa %>% filter(!is.na(m_flag))

noctua:::athena_option_env$cache_dt

#>                                 QueryId                                                                                 Query     State StatementType WorkGroup                            UnloadDir
#> 1: 774b6a83-cf53-4a5b-849c-d4c144bcf672 SELECT *\nFROM "awswrangler_test"."noaa"\nWHERE (NOT((("m_flag") IS NULL)))\nLIMIT 11 SUCCEEDED           DML   primary 08313f36-6da9-490f-8ba0-947f14d9a9f8
#> 2: 9a94ae61-8b85-4620-9440-9033990dc83f                                                   select * from awswrangler_test.noaa SUCCEEDED           DML   primary 72c55e13-6770-4b15-8e98-bc77be1e5c8b