json_normalize(prices) results in empty DataFrame if NaN occurs

[starsnpixel]

Hi, I encountered the following problem: ig_service.fetch_historical_prices_by_epic_and_num_points can lead to cells which are NaN (which for me is the case for the "last" values, for some reason). In this case, json_normalize(prices) results in an empty DataFrame and the code crashes. I've seen several explanations on stackoverflow on how to handle this (NaN needs to be replaced by {}) but I haven't been able to fix it yet - if I come around with a solution, I'll let you know.

[bug-or-feature]

@starsnpixel See the FAQ item on reporting errors, item 3

[starsnpixel]

Sorry for missing that:

from trading_ig import IGService
from trading_ig.config import config

# login ...

selected_market='IX.D.DAX.IFMM.IP'
data=ig_service.fetch_historical_prices_by_epic_and_num_points(selected_market, '15Min', 2)
# at this point, data looks like this:
#{'prices':                          bid                             ... last                 
#                        Open     High      Low    Close  ... High Low Close Volume
#DateTime                                                 ...                      
#2022-12-27 19:45:00  13997.8  13998.8  13981.8  13985.8  ...  NaN NaN   NaN    186
#2022-12-27 20:00:00  13986.3  13995.8  13983.3  13991.8  ...  NaN NaN   NaN    116
#
#[10 rows x 13 columns], 'instrumentType': 'INDICES', 'allowance': {'remainingAllowance': 8466, 'totalAllowance': 10000, 'allowanceExpiry': 507561}}

avg=ig_service.mid_prices(data['prices'],'2')

After the line "df = json_normalize(prices)", debugging reports "Empty DataFrame [Columns: []] [Index: [0, 1, 2, 3, 4, 5, 6, 7]]" and in the end the following error is returned: "KeyError: 'None of ['snapshotTimeUTC'] are in the columns'"

[bug-or-feature]

There's a couple of issues here. The first one is that mid_prices() is not supposed to be used like that. Should be:

data = ig_service.fetch_historical_prices_by_epic_and_num_points(
    selected_market,
    '15Min',
    2,
    format=ig_service.mid_prices)

You pass in the function that you want to do the dataframe manipulation, not call it directly. NaNs get dealt with if you do it like that. Having said that, there is a bug in the mid_prices() function if called using older methods, so the above would fail anyway. I'll create an issue to improve the docs about how to use those functions, and to fix the bug.

In the meantime, you would be better off using the newer fetch_historical_prices_by_epic() method. Always use the newer functions where you can. Newer ones have higher version number, 3 in this case

[starsnpixel]

Thanks for your help, switching to fetch_historical_prices_by_epic got it running for me! :)