search

ig-python / trading-ig

A lightweight Python wrapper for the IG Markets API
https://trading-ig.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
320 stars 199 forks source link

fetch_historical_prices_by_epic_and_date_range raise KeyError(key) from err KeyError: 'last' #305

Closed gfx-prog closed 1 year ago

gfx-prog commented 1 year ago

Hi, I do get the error in the title 'raise KeyError(key) from err KeyError: 'last'' depending on the time set to download price data.

When start_date=str(dt_to.strftime("%Y-%m-%d")) + " 02:00:00" then I get raise KeyError(key) from err KeyError: 'last' When start_date=str(dt_to.strftime("%Y-%m-%d")) + " 03:00:00" then data is downloaded without any error.

`def price_dlw_ig(lbl_log, lbl_progress_status, tg_table, tickers, dt_fr, dt_to):

''' Download prices from IG API and save output as csv file in datafeed directory '''

start_date=str(dt_to.strftime("%Y-%m-%d"))  + " 02:00:00"
end_date = str(dt_to.strftime("%Y-%m-%d"))  + " 24:00:00"

for i in range(0, len(tickers)):

    response = ig_service.fetch_historical_prices_by_epic_and_date_range(epic=tickers[i], resolution= tg_table  + 'Min', start_date=start_date, end_date=end_date)

    df_bid = response['prices']['bid']        
    df_bid['Volume'] = response['prices']['last']['Volume']`

Also, if I add to the code print(response['prices']) when start_date=str(dt_to.strftime("%Y-%m-%d")) + " 02:00:00" then I do see that the downloading did work partially. 

                     bid                                 ask           \
                    Open     High      Low    Close     Open     High

DateTime 2023-10-02 02:00:00 15412.6 15416.3 15409.1 15414.5 15419.6 15423.3 2023-10-02 02:05:00 15415.7 15419.6 15414.4 15416.7 15422.7 15426.6 2023-10-02 02:10:00 15417.6 15426.5 15417.6 15420.5 15424.6 15433.5 2023-10-02 02:15:00 15419.6 15464.0 15418.7 15462.0 15426.6 15469.0 2023-10-02 02:20:00 15462.5 15462.5 15455.5 15456.0 15467.5 15467.5 ... ... ... ... ... ... ... 2023-10-02 23:40:00 15197.5 15197.5 15197.5 15197.5 15204.5 15204.5 2023-10-02 23:45:00 15197.6 15197.9 15197.6 15197.6 15204.6 15204.9 2023-10-02 23:50:00 15198.4 15209.7 15198.4 15209.1 15205.4 15216.7 2023-10-02 23:55:00 15208.6 15210.9 15208.6 15210.0 15215.6 15217.9 2023-10-03 00:00:00 15211.7 15213.5 15207.2 15210.4 15218.7 15220.5

                     Low    Close

DateTime 2023-10-02 02:00:00 15416.1 15421.5 2023-10-02 02:05:00 15421.4 15423.7 2023-10-02 02:10:00 15424.6 15427.5

The difference is that response['prices']['last'] is missing in the downloaded data set.

Any clue what's going on ?

Thks.

gfx

gfx-prog commented 1 year ago

no help ?

bug-or-feature commented 1 year ago

hi @gfx-prog please read the new section in the README about support. I can't understand your problem currently. Please fix the formatting too

gfx-prog commented 1 year ago

Hi, thank you for your email. Sorry if confusing. I'll try to shorten the issue. Below my code:

response = ig_service.fetch_historical_prices_by_epic_and_date_range(epic=tickers[i], resolution= '15Min', start_date=start_date, end_date=end_date)

epic can be : IX.D.DAX.IDF.IP

This time range works : start_date=str(dt_to.strftime("%Y-%m-%d")) + " 03:00:00" end_date = str(dt_to.strftime("%Y-%m-%d")) + " 24:00:00"

This one doesn't :

start_date=str(dt_to.strftime("%Y-%m-%d")) + " 02:00:00" end_date = str(dt_to.strftime("%Y-%m-%d")) + " 24:00:00"

Question is why ['prices']['last'] section IS NOT RETURNED BY THE REQUEST WHEN TIME IS SET TO '02:00:00'

I guess you can test it as such

galburn commented 1 year ago

I've just tested and cannot reproduce your issue:-

    epic = 'IX.D.DAX.IDF.IP'
    start_date = '2023-10-02 02:00:00'
    end_date = '2023-10-03 24:00:00'
    response = ig_service.fetch_historical_prices_by_epic_and_date_range(epic=epic, resolution= '15Min', start_date=start_date, end_date=end_date)    
    print(f"TEST response={response}")

TEST response={'prices': bid ask last Open High Low Close Open High Low Close Open High Low Close Volume DateTime 2023-10-02 02:00:00 15447.0 15459.0 15445.5 15455.0 15452.0 15464.0 15450.5 15460.0 NaN NaN NaN NaN 70 2023-10-02 02:15:00 15454.5 15462.0 15451.0 15458.0 15459.5 15467.0 15456.0 15463.0 NaN NaN NaN NaN 77 2023-10-02 02:30:00 15457.5 15461.0 15452.5 15457.0 15462.5 15466.0 15457.5 15462.0 NaN NaN NaN NaN 43 2023-10-02 02:45:00 15457.5 15460.5 15453.0 15455.0 15462.5 15465.5 15458.0 15460.0 NaN NaN NaN NaN 26 2023-10-02 03:00:00 15454.5 15457.0 15450.0 15452.0 15459.5 15462.0 15455.0 15457.0 NaN NaN NaN NaN 44 ... ... ... ... ... ... ... ... ... ... ... .. ... ... 2023-10-03 23:00:00 15057.6 15063.7 15053.4 15056.3 15064.6 15070.7 15060.4 15063.3 NaN NaN NaN NaN 0 2023-10-03 23:15:00 15055.9 15063.2 15055.4 15062.4 15062.9 15070.2 15062.4 15069.4 NaN NaN NaN NaN 0 2023-10-03 23:30:00 15062.8 15068.9 15061.5 15068.9 15069.8 15075.9 15068.5 15075.9 NaN NaN NaN NaN 0 2023-10-03 23:45:00 15069.3 15072.8 15068.0 15069.9 15076.3 15079.8 15075.0 15076.9 NaN NaN NaN NaN 0 2023-10-04 00:00:00 15069.0 15069.0 15058.6 15061.6 15076.0 15076.0 15065.6 15068.6 NaN NaN NaN NaN 0

[185 rows x 13 columns], 'instrumentType': 'INDICES', 'allowance': {'remainingAllowance': 9537, 'totalAllowance': 10000, 'allowanceExpiry': 604717}}

gfx-prog commented 1 year ago

Thks for following up. I did replace my code with yours.

` epic = 'IX.D.DAX.IDF.IP' start_date = '2023-10-02 02:00:00' end_date = '2023-10-03 24:00:00' response = ig_service.fetch_historical_prices_by_epic_and_date_range(epic=epic, resolution= '15Min', start_date=start_date, end_date=end_date)

response = ig_service.fetch_historical_prices_by_epic_and_date_range(epic=tickers[i], resolution= tg_table + 'Min', start_date=start_date, end_date=end_date)

    print(f"TEST response={response}")  `

TEST response={'prices': bid ask \ Open High Low Close Open High
DateTime 2023-10-02 02:00:00 15402.9 15416.8 15399.4 15410.8 15409.9 15423.8
2023-10-02 02:15:00 15409.9 15468.0 15409.0 15462.0 15416.9 15473.0 2023-10-02 02:30:00 15461.5 15466.0 15438.0 15448.0 15466.5 15471.0 2023-10-02 02:45:00 15447.5 15453.5 15445.0 15447.5 15452.5 15458.5 2023-10-02 03:00:00 15447.0 15459.0 15445.5 15455.0 15452.0 15464.0 .... 2023-10-03 12:45:00 15165.0 15176.0 15162.5 15166.5 15166.4 15177.4 2023-10-03 13:00:00 15164.5 15171.5 15144.5 15149.5 15165.9 15172.9 2023

` epic = 'IX.D.DAX.IDF.IP' start_date = '2023-10-02 03:00:00' end_date = '2023-10-03 24:00:00' response = ig_service.fetch_historical_prices_by_epic_and_date_range(epic=epic, resolution= '15Min', start_date=start_date, end_date=end_date)

response = ig_service.fetch_historical_prices_by_epic_and_date_range(epic=tickers[i], resolution= tg_table + 'Min', start_date=start_date, end_date=end_date)

    print(f"TEST response={response}")       `

TEST response={'prices': bid ask ... last \ Open High Low Close Open ... Open
DateTime ...
2023-10-02 03:00:00 15447.0 15459.0 15445.5 15455.0 15452.0 ... NaN
2023-10-02 03:15:00 15454.5 15462.0 15451.0 15458.0 15459.5 ... NaN
2023-10-02 03:30:00 15457.5 15461.0 15452.5 15457.0 15462.5 ... NaN
2023-10-02 03:45:00 15457.5 15460.5 15453.0 15455.0 15462.5 ... NaN
2023-10-02 04:00:00 15454.5 15457.0 15450.0 15452.0 15459.5 ... NaN
2023-10-02 04:15:00 15451.5 15458.0 15448.0 15452.5 15456.5 ... NaN
...

2023-10-03 22:30:00 NaN NaN NaN 0 2023-10-03 22:45:00 NaN NaN NaN 0 2023-10-03 23:00:00 NaN NaN NaN 0 2023-10-03 23:15:00 NaN NaN NaN 0 2023-10-03 23:30:00 NaN NaN NaN 0 2023-10-03 23:45:00 NaN NaN NaN 0 2023-10-04 00:00:00 NaN NaN NaN 0

[181 rows x 13 columns], 'instrumentType': 'INDICES', 'allowance': {'remainingAllowance': 1273, 'totalAllowance': 10000, 'allowanceExpiry': 603199}}

gfx-prog commented 1 year ago

Would you mind testing

start_date = '2023-10-02 00:30:00'

gfx-prog commented 1 year ago

Also, if you are using a demo account, would you mind sharing credentials ? To check if the issue is account related.

bug-or-feature commented 1 year ago

I'm closing this. As mentioned here, issues here are for problems with this library. Your issue is due to the data that comes from IG. Please contact them about why you are not seeing the data you expect. I suggest you change your code so that it doesn't make any assumptions about which columns/values are present in the response data. Another thing you could do is to pass a format parameter into the fetch_historical_prices_by_epic_and_date_range() function. This defines the function that converts the raw JSON from IG into a Pandas dataframe.

gfx-prog commented 1 year ago

Please contact them ... do you have any email address of people caring about the API ?

9gix commented 9 months ago

I just encounter a possibly similar issue. I traces to the code and trying to understand why the last.volume column is randomly missing. And it seems that IG would return the some data point with 0 volume. And in the code, i noted that if the first data has lastTradedVolume == 0, then that volume column will be dropped. You gotta be lucky enough to have that as your first row. https://github.com/ig-python/trading-ig/blob/master/trading_ig/rest.py#L1574

this only happen if you use the default format_prices, for flatten_prices i don't think this will happen.

For example, here's what i get from the raw data: Harvesting IX.D.NASDAQ.IFG.IP with 1H resolution from 2024-02-06 06:00:00+08:00 to 2024-02-06 11:00:00+08:00 (Timezone: UTC+8)

           snapshotTime  lastTradedVolume
0   2024/02/06 06:00:00                 0
1   2024/02/06 07:00:00              3281
2   2024/02/06 08:00:00              2745
3   2024/02/06 09:00:00              3562
4   2024/02/06 10:00:00              3084
5   2024/02/06 11:00:00              3679