Missing field error for bulkref

[jxg852]

Hi, I have a similar issue to #13, except with con.bulkref - when a field does not exist for a subset of securities I'm trying to query, it throws an error for the entire thing.

test = ['594918BD5 Corp','594918BE3 Corp','88428LAA0 Corp']
bbg = con.bulkref(test, ['INDEX_LIST'])

The query works for the first two securities but not the last, so the whole query fails instead of just returning NaN where appropriate (error reproduced below).

securityData = {
  security = "88428LAA0 Corp"
  eidData[] = {
  }
  fieldExceptions[] = {
  }
  sequenceNumber = 2
  fieldData = {
  }
}

Traceback (most recent call last):

  File "<ipython-input-142-61c895cb719c>", line 1, in <module>
    bbg = con.bulkref(test, ['INDEX_LIST'])

  File "C:\Temp\Jay\Anaconda\envs\py35\lib\site-packages\pdblp\pdblp.py", line 378, in bulkref
    data = self._parse_bulkref(flds)

  File "C:\Temp\Jay\Anaconda\envs\py35\lib\site-packages\pdblp\pdblp.py", line 404, in _parse_bulkref
    if not fieldData.getElement(fld).isArray():

  File "C:\Temp\Jay\Anaconda\envs\py35\lib\site-packages\blpapi\element.py", line 347, in getElement
    _ExceptionUtil.raiseOnError(res[0])

  File "C:\Temp\Jay\Anaconda\envs\py35\lib\site-packages\blpapi\exception.py", line 145, in raiseOnError
    _ExceptionUtil.raiseException(errorCode, description)

  File "C:\Temp\Jay\Anaconda\envs\py35\lib\site-packages\blpapi\exception.py", line 137, in raiseException
    raise errorClass(description, errorCode)

NotFoundException: Attempt to access unavailable sub-element 'INDEX_LIST' of element 'fieldData'. (0x0006000d)

[matthewgilbert]

This looks like a bug. I agree the behaviour should be similar to ref in this scenario.

[matthewgilbert]

Similar to ref, there can be empty fields as well as NOT_APPLICABLE_TO_REF_DATA. For example

con.bulkref('CL1 Comdty', ['FUT_DLVRBLE_BNDS_ISINS'])

DEBUG:root:Sending Request:
 ReferenceDataRequest = {
    securities[] = {
        "CL1 Comdty"
    }
    fields[] = {
        "FUT_DLVRBLE_BNDS_ISINS"
    }
    overrides[] = {
    }
}

DEBUG:root:Message Received:
 ReferenceDataResponse = {
    securityData[] = {
        securityData = {
            security = "CL1 Comdty"
            eidData[] = {
            }
            fieldExceptions[] = {
                fieldExceptions = {
                    fieldId = "FUT_DLVRBLE_BNDS_ISINS"
                    errorInfo = {
                        source = "908::bbdbd10"
                        code = 9
                        category = "BAD_FLD"
                        message = "Field not applicable to security"
                        subcategory = "NOT_APPLICABLE_TO_REF_DATA"
                    }
                }
            }
            sequenceNumber = 0
            fieldData = {
            }
        }
    }
}

As well as

con.bulkref(['88428LAA0 Corp'], ['INDEX_LIST'])

DEBUG:root:Sending Request:
 ReferenceDataRequest = {
    securities[] = {
        "88428LAA0 Corp"
    }
    fields[] = {
        "INDEX_LIST"
    }
    overrides[] = {
    }
}

DEBUG:root:Message Received:
 ReferenceDataResponse = {
    securityData[] = {
        securityData = {
            security = "88428LAA0 Corp"
            eidData[] = {
            }
            fieldExceptions[] = {
            }
            sequenceNumber = 0
            fieldData = {
            }
        }
    }
}

However unlike ref(), the fieldData returned can have multiple sub values. For example for DVD_HIST, from con.bulkref("101 HK EQUITY", "DVD_HIST")

fieldData = {
    DVD_HIST[] = {
        DVD_HIST = {
            Declared Date = 1986-09-26
            Ex-Date = 1986-10-31
            Record Date = 1986-11-01
            Payable Date = 1986-11-17
            Dividend Amount = 0.720000
            Dividend Frequency = "Semi-Anl"
            Dividend Type = "Final"
        }
        DVD_HIST = {
            Declared Date = 1986-01-01
            Ex-Date = 1986-04-11
            Record Date = 1986-04-14
            Payable Date = 
            Dividend Amount = 0.280000
            Dividend Frequency = "Semi-Anl"
            Dividend Type = "Interim"
        }
    [ommited]
}

This makes assigning values NaN which are not applicable, similar to how it is treated in ref() discussed in #6, problematic since the actual value names not just the values are unknown. Omitting the rows entirely is one option, however this is somewhat akin to failing silently which I am not an advocate of. I think the best solution here would be to NaN out the name, value and position columns. So for example both con.bulkref('CL1 Comdty', ['FUT_DLVRBLE_BNDS_ISINS']) and con.bulkref(['88428LAA0 Corp'], ['INDEX_LIST']) would return DataFrames like

       ticker                   field   name value  position
0  CL1 Comdty  FUT_DLVRBLE_BNDS_ISINS    NaN   NaN       NaN

and

           ticker       field   name value  position
0  88428LAA0 Corp  INDEX_LIST    NaN   NaN       NaN

[jxg852]

Ok that makes sense, but what's the best way to handle the error such that the query just returns results for securities with valid values for that field, rather than breaks completely?

This works, but is there a smarter way to do it for very large lists of securities?

test = ['594918BD5 Corp','594918BE3 Corp','88428LAA0 Corp']

bbg = pd.DataFrame(columns=['ticker','field','name','value','position'])
for i in range(0, len(test)):
    try:
        temp = con.bulkref(test[i], ['INDEX_LIST'])
        bbg = bbg.append(temp)
    except:
        print('Error - ' + test[i])

[matthewgilbert]

I will implement the fix I discussed above in the next day or two. Until then I would do something like you are doing above.

[jxg852]

Awesome thanks!

[matthewgilbert]

This is not yet released but if you install the latest version of master the issue should be resolved.