ropensci / elastic

R client for the Elasticsearch HTTP API
https://docs.ropensci.org/elastic
Other
245 stars 58 forks source link

scroll with asdf = TRUE ? #163

Closed joelgombin closed 7 years ago

joelgombin commented 7 years ago

If my initial Search uses the asdf = TRUE argument, is it possible to then make sure that scroll returns objects consistent with the one returned by the initial Search?

Alternatively, can I easily bind the lists and then make sure to get a dataframe?

sckott commented 7 years ago

thanks @joelgombin

this is somewhat related to #160 in which experimenting with jsonlite::stream_in which allows a custom handler to determine how each JSON page is parsed.

pushing something up soon.

we maybe can just add the asdf param to scroll, as we call jsonlite::fromJSON(res, FALSE) inside scroll and if we have asdf in scroll then we can do jsonlite::fromJSON(res, asdf) to let use control output

joelgombin commented 7 years ago

I think an asdf param to scroll would make a lot of sense. Alternatively, I realised that by passing a query to the body argument, I can go around the 10000 observations limit of Search:

query <- '{
  "from": 0,
"size": 1000000,
....
}'

Search(body = query, asdf = TRUE, scroll = "5m")
sckott commented 7 years ago

@joelgombin reinstall and try again, added asdf parameter to scroll

joelgombin commented 7 years ago

Thanks, that works!