Q: Multiple publishers for live market data streaming using ZMQ Black Box Pattern

[ppseverin]

Hi,

I'm trying to implement a live data streaming from a MT4 terminal to a Postgres database. To accomplish this, I'm using dwx-zeromq-connector and I had success when doing this for 2 currency pairs.

The way the connector works is by setting a pub/sub ports and stream data from MT4 using a python script. Now I want to stream 28 currencies, so the number of charts in the MT4 terminal is 28 and the problems of slow subscribers feels very heavy. The messages queue up and no matter what I try, it does not seems to process them any faster.

Probably one question to answer is "why not use the subscribe function provided in dwx-zeromq-connector to subscribe to more currency pairs". Well, the answer to that is that if I do that, I'm not getting the real tick of other currencies, but the quote of the currency when the tick of the chart where the EA is running arrives. For example, if I run the EA in the EURUSD and subscribe to EURUSD and EURGBP, whenever a tick arrives to the EURUSD I get the quote for EURUSD and EURGBP, but I'm not getting the real EURGBP tick.

What I have tried so far is:

• 28 different sub sockets for all pub ports using multiprocessing and daemon threads (not at the same time)
• 1 socket for all 28 pub ports using multiprocessing and daemon threads (not at the same time)
• 28 different classes running 1 daemon thread each, one per pub
• 1 class having 7 different threads and each thread has 1 socket that manages 4 pub ports

None of them improves the performance. Now, looking at this zmq guide, the Black Box Pattern seems to be suitted for what I need. However, I was wondering if I should reduce the amount of pub ports by making that the publishers publish information to only 1 pub socket.

So, according to the zmq guide Black Box Pattern, I guess that I need to use only 1 publisher and then push the messages to worker, right? Based on all this, is it possible that many publishers publish messages to only one port using tcp protocols? And is the zmq guide what I should really use or I'm missing something? The specs of the VPS I'm using is 2 vCPU and 12 GB RAM.

Thank you!

[elvinex]

Hi, We will (hopefully soon) publish an alternative easier solution just based communication via files, which will be more performant when it comes to subscribing to a lot of symbols. It will check prices regularly in a set interval (e.g. every 25ms).

Btw. you could also use the default subscribe function of zeromq and reduce the MILLISECOND_TIMER_PRICES variable. MT4 only checks for new prices in regular intervals anyway, so if you set it low enough (<100ms), there is not much additional delay.