Pulling >25 fields at once with bdh

[Ddfulton]

Would people like a pull request that lets you pull more than 25 fields at once with bdh? I have made some workarounds (for example two pulls + a merge, as suggested by @mtkerbeR the other day) but they fail when the dates don't line up.

For example

merge(bdh(securities, first_25_fields, ...),
           bdh(securities, next_25_fields, ...),
           by="date", all=TRUE)

Is there a better way that works every time? I have made a more elaborate workaround and am considering opening a pull request, but wanted to check with the folks here before I proceed.

[mtkerbeR]

I though about this as well, but I don't see an "easy" solution (at least not in the R-Code).

Apart from "dates don't line up" , things might get complicated as the order of securities returned might be different for each call to bdh (as mentioned in the function documentation):

library(Rblpapi)
blpConnect()

sec <- paste(bds("DAX Index", "INDX_MEMBERS")[[1]], "Equity")

df1 <- bdh(sec, "PX_LAST", Sys.Date()-5)
df2 <- bdh(sec, "PX_LAST", Sys.Date()-5)

names(df1) == names(df2)
#>  [1]  TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
#> [13] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
#> [25] FALSE FALSE FALSE FALSE FALSE FALSE

Hence I think a function as proposed would probably require (a parameter) to return the results of bdh sorted along vector securities (if securities is of length > 1) - which was discussed several times in the past, e.g. #139 #101

[eddelbuettel]

I think I would be moderately against a patch. To me this is best used in user-space -- plus maybe a write up in a vignette (or, easier, a demo/ script or wiki entry) -- as this strikes me as inherently fragile and hard to test / maintain.

We could warn though when the length of fields vector is greater than twenty-five (or another given cutoff).

[johnlaing]

I did some digging to convince myself that this is in fact a BB constraint rather than something we've introduced. And so it is:

HistoricalDataResponse = {
    responseError = {
        source = "bbdbh6"
        code = 19
        category = "BAD_ARGS"
        message = "Number of fields exceeds max of 25 [nid:809] "
        subcategory = "TOO_MANY_FIELDS"
    }
}

In that case I agree with Dirk, any sort of merge belongs in user-space.

But, it is clear that we could do a much better job with error handling. In particular we could capture the message and show it to the user. I'll open a separate issue for that.

[klin333]

user code like this may be of use, for the tidyverse users

# assumes library(Rblpapi); library(tidyverse)
robust_bdh <- function(securities, fields, ..., split_large_req = FALSE) {

  # bdh only works for up to 25 fields at once
  field_chunks <- split(fields, ceiling(seq_along(fields)/25))

  # bbg has server timeout of about 10-20 mins, so may need to split large queries
  if (split_large_req) {
    security_chunks <- split(securities, ceiling(seq_along(id)/100))
  } else {
    security_chunks <- list(securities)
  }

  map_dfr(security_chunks, function(security_chunk) {
    map(field_chunks, function(field_chunk) {
      dat <- bdh(security_chunk, field_chunk, ...)
      if(length(security_chunk) == 1) {
        dat <- as_tibble(dat) %>% mutate(id = security_chunk)
      } else {
        dat <- map(dat, as_tibble) %>% bind_rows(.id = 'id')
      }
      dat %>% select(id, date, everything())
    }) %>%
      reduce(full_join, by = c('id', 'date'))
  }) %>%
    arrange(id, date) # user argument `securities` ordering not respected
}