overcome expected 404 error in call to historical_marketdata_beta (/hmds/history)

[Harald-F]

in my first call to historical_marketdata_beta, I get the 404 error. This is expected as documented by IB:

The first time a user makes a request to the /hmds/history endpoints will result in a 404 error. This initial request instantiates the historical market data services allowing future requests to return data. Subsequent requests will return data as expected.

As a result, my script stops with an HTTPError by the exception handler. How can I overcome this? I'm fine to make a dummy pre-call somewhere, but this does not eliminate the problem. There is potentially a way to detect the specific 404 for that function and ignore it, but I have no idea how to make it. To be clear, this is primarily a lack of knowledge on my side and not an issue of iBind. But if somebody can suggest a solution how to work around this problem, I would be very thankful.

[Voyz]

Hey @HaSiMiPa many thanks for trying out that endpoint and sharing your results. I just tried it and I get a 404 followed by a only 500s, hence I wouldn't be surprised if that endpoint isn't fully functional yet.

As for catching the initial 404, the ExternalBrokerError carries over the status code that you can use:

class ExternalBrokerError(Exception):
    """ Something unexpected happened externally """
    def __init__(self,*args, status_code:int=None, **kwargs):
        self.status_code = status_code
        super().__init__(*args)

Hence, you could get around it as follows:

def call_historical_marketdata_beta(max_attempts=3):
    for attempt in range(max_attempts):
        try:
            return ibkr_client.historical_marketdata_beta(265598, '1d', '1hrs', outside_rth=True)
        except ExternalBrokerError as e:
            if e.status_code == 404 and attempt < max_attempts - 1:
                print('Ignoring the initial 404 code')
                continue
            raise e

This function assumes the ibkr_client is available. It will attempt to call the endpoint several times, and ignore the 404 errors up to max_attempts times. An even simpler version of this would be:

try:
    return ibkr_client.historical_marketdata_beta(265598, '1d', '1hrs', outside_rth=True)
except ExternalBrokerError as e:
    if e.status_code == 404:
        return ibkr_client.historical_marketdata_beta(265598, '1d', '1hrs', outside_rth=True)
    else:
        raise e

[Harald-F]

Hey Voy, thanks a lot, once again! Much more help than expected. Coding is done, I need to check tomorrow, once I get the 404 again. I never got a 500, even after several hundred request, works very stable for me.

[Voyz]

Great, glad to be of help 👏 Interesting to hear that this endpoint doesn't return 500 for you, it never seems to work for me. Could you share what parameters you run it with?

[Harald-F]

for me, even your example code from above works client.historical_marketdata_beta(265598, '1d', '1hrs', outside_rth=True) returns Result(data={'startTime': '20240528-10:00:00', 'startTimeVal': 1716883200000, 'endTime': '20240528-11:32:50', 'endTimeVal': 1716888770000, 'data': [{'t': 1716883200000, 'o': 190.6, 'c': 194.22, 'h': 194.22, 'l': 190.6, 'v': 44481.0}, {'t': 1716886800000, 'o': 194.04, 'c': 194.23, 'h': 194.23, 'l': 194.04, 'v': 534.0}], 'points': 2, 'mktDataDelay': 0}, request={'url': 'https://xxxxxxxxxxxxxx/v1/api/hmds/history', 'params': {'conid': 265598, 'period': '1d', 'bar': '1hrs', 'outsideRth': True}})

[Harald-F]

Today, I had indeed a 500, but only once, directly after the 404. After the intialization phase, never seen it again. So I included the 500 to the exception ignore list. I still would get a print if it happens. So far, it did not.

[Voyz]

Interesting! That's great to hear it works for you though, I wonder what must be the cause. Are you calling it within the trading hours, or outside?

[Harald-F]

have it running within and outside official exchange opening hours, also used with both parameter outside_rth options True and False. No visible difference in behaviour. For testing purposes, I added a 1 sec wait after a 404 response, since then I have not ever seen a 500. This may indicate that after the very first call and the expected 404, the backend is not instantly ready and the next call, that happens within microseconds, gives a 500. But waiting for a 1sec the backend is meanwhile ready to properly respond.

[Voyz]

Interesting. For me it just happened shortly after 404 and just wouldn't work at all. No idea why. In either case, I'm glad to have helped with catching the 404 👍

[Harald-F]

today, I noticed another strange behaviour. I had 2 different accounts in operation. For one, the endpoint was working stable, for the other one it did not respond at all. Well, it is marked beta. Anyways, the original objective was to be able to handle the 404 and that was acomplished. Hence, closing the issue.

[Voyz]

Yeah, I guess it being beta makes sense. How about the normal historical_marketdata? Is it not suffice for you? Any reason to try the beta?

[Harald-F]

honestly, I do not recall. I think I had a reason but revisiting both functions now, it doesn't make sense anymore. The beta function is instable and provides less data. I had meanwhile changed back to the regular one.

[Voyz]

understood, thanks!