Historical / Bar Data Reuse

[01tsos]

Hello there!

I started using this api yesterday and everything worked fine but starting today, the historical data started acting weird. It doesn't bring the latest bar and the bar range of the output is more than requested. No matter what timeframe or range of dates requested, it brings this same constant output of M1 bars requested over a day. It was the last request I made yesterday:

historic_data: EURUSD M1 1434 bars

{'EURUSD_M1': {'2023.03.08 10:53': {'open': 1.05407, 'high': 1.05413, 'low': 1.05398, 'close': 1.05407, 'tick_volume': 73.0}, '2023.03.08 10:54': {'open': 1.05408, 'high': 1.05413, 'low': 1.05394, 'close': 1.05413, 'tick_volume': 76.0}, '2023.03.08 10:55': {'open': 1.05413, 'high': 1.05414, 'low': 1.05397, 'close': 1.05412, 'tick_volume': 62.0}, '2023.03.08 10:56': {'open': 1.05412, 'high': 1.05428, 'low': 1.05385, 'close': 1.05385, 'tick_volume': 79.0}, '2023.03.08 10:57': {'open': 1.05386, 'high': 1.05386, 'low': 1.05367, 'close': 1.05372, 'tick_volume': 43.0},

...                   ...      ...      ...      ...          ...

'2023.03.09 10:50': {'open': 1.0557, 'high': 1.0557, 'low': 1.05548, 'close': 1.05554, 'tick_volume': 55.0}, '2023.03.09 10:51': {'open': 1.05555, 'high': 1.05588, 'low': 1.05555, 'close': 1.05588, 'tick_volume': 32.0}, '2023.03.09 10:52': {'open': 1.0559, 'high': 1.05595, 'low': 1.0559, 'close': 1.05595, 'tick_volume': 12.0}}}

Time : 2023-03-10 13:43:49.481380

Here is a snap of the code:

class tick_processor():

    def __init__(self, MT4_directory_path, 
                 sleep_delay=0.005,             # 5 ms for time.sleep()
                 max_retry_command_seconds=10,  # retry to send the commend for 10 seconds if not successful. 
                 verbose=True
                 ):

        self.dwx = dwx_client(self, MT4_directory_path, sleep_delay, 
                              max_retry_command_seconds, verbose=verbose)
        sleep(1)

        self.dwx.start()

        # request historic data:
        end = datetime.utcnow()
        start = end - timedelta(hours=2)  
        self.dwx.get_historic_data('EURUSD', 'M15', start.timestamp(), end.timestamp())

    def on_historic_data(self, symbol, time_frame, data):
        data_ = self.dwx.historic_data
        print('historic_data:', symbol, time_frame, f'{len(data)} bars')
        print(data_)
        print(f"Time : {datetime.utcnow()}")

What am I doing wrong?

[01tsos]

A simple restart of my computer fixed everything. But I would like to know what went wrong that made that happen because it can happen again.

[01tsos]

Also. Is there a way I can request for historical data in real time like bar data? That would be useful if I want to make calculations for indicators

[elvinex]

Hi,

regarding your last question, there is a function to subscribe to bar data. For example, like this:

symbols = [
    ['EURUSD', 'M1'],
    ['GBPUSD', 'M5']
]
self.dwx.subscribe_symbols_bar_data(symbols)

But you would then have to handle the history yourself, for example by appending the new data in the on_bar_data method to an existing list.

About the first issue, self.dwx.historic_data is a dictionary containing the last request for each symbol+timeframe combination. The output you posted shows the "EURUSD_M1" key which contains the last EURUSD M1 request. However, in your __init__ method you request M15 data. For the latest M15 data, you would have to check the "EURUSD_M15" part of the dictionary.

[01tsos]

Thanks for the reply!

I have another issue.

Sometimes, my orders get executed multiple times instead of once. All orders can be executed at the same time (down to millisecond) or at most a second apart from each other. I don't know why this happens. Maybe my local machine is too slow to read the DWX_Orders_Stored file but I ran it on a VPS and the same thing happend. On some occasions, orders get executed up to five times!

Have you stumbled upon this issue before, maybe there's something I have to fix on my end?

class tick_processor():

    def __init__(self, MT4_directory_path, 
                 sleep_delay=0.005,           
                 max_retry_command_seconds=10,
                 verbose=True
                 ):
        self.open_time = datetime.utcnow()
        self.dwx = dwx_client(self, MT4_directory_path, sleep_delay, 
                              max_retry_command_seconds, verbose=verbose)
        sleep(1)
        self.dwx.start()
        self.dwx.subscribe_symbols(['EURUSD'])
        self.dwx.subscribe_symbols_bar_data([['EURUSD', 'M1']])

    def on_tick(self, symbol, bid, ask):
        now = datetime.utcnow()
        if len(self.dwx.open_orders) == 0:
            self.dwx.open_order(symbol=symbol, order_type='buy', 
                                        price=ask, lots=0.01)  
            self.open_time = now
        else:
            if now > self.open_time + timedelta(seconds=15):
                for ticket in self.dwx.open_orders.keys():
                    self.dwx.close_order(ticket)

[elvinex]

Did you use the latest version,where the command ID is passed along? MT4 should only allow one ID to be executed once and print an error if the same one is used multiple times.

[01tsos]

I believe I am using the latest version because I copied this repo on my computer about two months ago with Git Bash. I use it with MT5, not MT4 and I do not think a command ID is passed along.

Is this command ID feature only for MT4?

[elvinex]

The MT5 version also checks the command ID. Maybe there is a bug with how it works. You could open the mq5 file in MetaEditor and search for "Not executing command because ID already exists."

https://github.com/darwinex/dwxconnect/blob/b9405e2abdf9d27073f31e517eaf4cfb6d83ff03/mql/DWX_Server_MT5.mq5#L243

[01tsos]

Okay, I just did the search and found the code. What may be the bug there?

[elvinex]

That is a good question.

Are you 100% sure that you are only sending one open_order() command? One scenario I could image is that a trade triggers on one tick, but it takes a few ticks until it is executed and MT5 sends the info back to python. So if you don't have some extra conditions (like not allowing trades for 10s after one triggered), it might happen that a trade triggers on 3 ticks in a row until python has the info that the first one was filled.

I double checked the code on both python and mql5 side and don't see how a command with the same ID could get executed multiple times. The MT5 side stores all IDs in an array and goes through the array when there is a new ID.

The ID array is reset either when you call reset_command_ids() from python (do you?). Or when the MT5 EA re-initialized, which should only happen if you restart MT5 or the EA manually.

[01tsos]

As you can see from my code above, the open_order() command is only sent once. That is exactly how my code is, nothing more nothing less except from the rest of the methods after on_tick() which all contains the pass statement e.g

def on_bar_data():
    pass

def on_order_event():
    pass

This is the first time I'm learning about the reset_command_ids() function so I do not call it in my code.

When I checked Deals history on MT5 (not to be confused with Positions history), I observed that those multiple orders do not have the same deal number. Each one has a unique deal number so I do not think an order with the same deal number is executed multiple times but rather different orders with different deal numbers and yet the open_order() is only called once. Does this make sense to you?

Anyways I tried your suggestion and put sleep(10) after the open_order() method and I am no more experiencing multiple orders. I'd like to put this under more observation and see if it doesn't persist. I'd feed you back on that.

Thank you very much for you time. Much appreciated!

[elvinex]

Ah right, I actually did't look at the code. 😄 But it is the exact case that I described. In on_tick the only condition you have is:

if len(self.dwx.open_orders) == 0

This will be true as long as MT5 has not sent the info about a filled position back to python. Filling an order can take >100ms plus the latency. So you could have quite a few ticks in that time and the condition would again be true.

If you ever use this in a real live strategy, I would not sleep(10) because it prevents all other stuff you code might do. Better define a last_open_time and add a condition for that to allow only trades once every 10s or so.

from datetime import datetime, timedelta

class tick_processor():

    def __init__(self, MT4_directory_path, 
                 sleep_delay=0.005,           
                 max_retry_command_seconds=10,
                 verbose=True
                 ):
        # set a past date as start value:
        self.last_open_time = datetime.utcnow() - timedelta(days=1)
        self.dwx = dwx_client(self, MT4_directory_path, sleep_delay, 
                              max_retry_command_seconds, verbose=verbose)
        sleep(1)
        self.dwx.start()
        self.dwx.subscribe_symbols(['EURUSD'])
        self.dwx.subscribe_symbols_bar_data([['EURUSD', 'M1']])

    def on_tick(self, symbol, bid, ask):
        now = datetime.utcnow()
        if len(self.dwx.open_orders) == 0 and now >= self.last_open_time + timedelta(seconds=10):
            self.dwx.open_order(symbol=symbol, order_type='buy', 
                                        price=ask, lots=0.01)  
            self.last_open_time = now
        else:
            if now > self.last_open_time + timedelta(seconds=15):
                for ticket in self.dwx.open_orders.keys():
                    self.dwx.close_order(ticket)

[01tsos]

Thank you very much, Elvinex!