server side request timeout causes infinite loop

[klin333]

Hi,

If we make a big bdh request, Rblpapi::bdh can hang in an infinite loop, without ever returning data or throwing an error. Indicative reproducable example below. From experimenting, it appears if a single request takes more than 10 mins server side, there is a time out that Rblpapi does not catch.

library(Rblpapi)
blpConnect()

fields <- c( # use more fields to slow down request and raise chance of server timeout
  "px_mid", "px_ask", "px_bid",
  "asset_swap_spd_mid", "asset_swap_spd_ask", "asset_swap_spd_bid"
)

ids <- rep("ZR653898 Corp", 2000)   # use more ids to slow down request and raise chance of server timeout
opts <- c("pricingOption" = "PRICING_OPTION_PRICE") # make request even slower to raise chance of server timeout

tictoc::tic()
res <- bdh(ids, fields, start.date = as.Date('2020-01-01'), end.date = as.Date('2021-11-30'), options = opts, verbose = TRUE)
tictoc::toc()

The Bloomberg developer guide https://data.bloomberglp.com/professional/sites/10/2017/03/BLPAPI-Core-Developer-Guide.pdf page 73 says

REQUEST_STATUS
- RequestFailure — Occurs when the request times out on the Bloomberg backend

In bdh message processing loop, if server side crashed and refuses to ever return a RESPONSE event, then rblpapi will be stuck in an infinite loop. Currently there is no server timeout error handling in this loop. After experimenting with debug prints, I am certain that in the event of a long running query, about more than 10 mins, server returns an event of type 4 ie Event::REQUEST_STATUS. Then, server never returns Event::RESPONSE, hence our loop hangs infinitely. Full error message below.

https://github.com/Rblp/Rblpapi/blob/e5ca48a0089ac53919d8cba6758ae13732b55b09/src/bdh.cpp#L160-L176

03DEC2021_05:20:37.915 32820:9632 ERROR blpapi_platformconnection.cpp:569 blpapi.session.platformconnection.{1}.<localhost:8196> ErrorComm received for request id: 8388631 on ConnectionCtx: [0/1]. Full prolog: 
MessageProlog = [ Version = 1.6 messageType = 956(ERRORCOMM) serviceCode = 9(STATUS) packetNumBytes = 40 prologNumWords = 8 headerNumWords = 5 payloadNumBytes = 8 contextId = 385908736 flags = none leadingFlags = none] 

from my own debug prints: REQUEST_STATUS bdh received unexpected event of type 4

I suggest we add a case to the event loop in bdh.cpp, eg

while (true) {
        Event event = session->nextEvent();
        switch (event.eventType()) {
        case Event::RESPONSE:
        case Event::PARTIAL_RESPONSE:
            ans_names.push_back(getSecurityName(event));
            ans[i++] = HistoricalDataResponseToDF(event, fields, rtypes, verbose);
            break;
        case Event::REQUEST_STATUS:
            Rcpp::stop("Bloomberg request timed out on server side\n");
            break;
        default:
            MessageIterator msgIter(event);
            while (msgIter.next()) {
                Message msg = msgIter.message();
                // FIXME:: capture msg here for logging
            }
        }
        if (event.eventType() == Event::RESPONSE) { break; }
    }

It's probably safer to also check for the response_type of "RequestFailure" as specified by the Bloomberg developer guide. However, I couldn't figure out the c++ code to check for that.

Arguably we can add this check in other event processing loops such as in bds.cpp. However, I can't see those requests ever taking 10 mins each. It appears only bdh has the potential to take so long because we jam pack all securities and all fields into 1 single bdh request.

[eddelbuettel]

This seems well reasoned. It would probably help if @johnlaing or @armstrtw could confirm the behavior.

[johnlaing]

I'll try to look into this.

[johnlaing]

Behavior confirmed. The problem with debugging timeout errors is it takes forever to trigger them.