search

nfultz / AWR.Athena

Short R Wrapper for Athena JDBC connections
23 stars 13 forks source link

Athena can only pull 999 rows at a time #8

Closed daroczig closed 6 years ago

nfultz commented 6 years ago

Yeck, RJDBC needs to get updated to use new DBI -

I think it makes more sense to implement dbFetch here than dbGetQuery, otherwise things might work inconsistently across the package.

In the dbFetch implementation, you should be able to calculate block size a la: with block = if(n < 0) 999 else pmin(n, 999)

and then delegate the request back to RJDBC::fetch with the smaller block size.

See also [1] and [2]

[1] https://github.com/s-u/RJDBC/blob/master/java/JDBCResultPull.java#L69 [2] https://github.com/s-u/RJDBC/blob/master/R/class.R#L328

nfultz commented 6 years ago

Also is this 999 limitation documented anywhere?

nfultz commented 6 years ago

Ah https://github.com/s-u/RJDBC/issues/44

daroczig commented 6 years ago

I'd like to get dbGetQuery working on Athena, and as fetch has a default 2K set for the blocksize -- that is called by dbGetQuery without passing block, I don't really see any other easy way for this. This PR required more commits than I originally hoped :) But seems to work fine now.

daroczig commented 6 years ago

@nfultz pls feel free to tweak this further, I don't really want to start hacking dbSendQuery and fetch instead

daroczig commented 6 years ago

Closing in favour of #9