Q: Multiple publishers for live market data streaming using ZMQ Black Box Pattern

[ppseverin]

Hi,

I'm trying to implement a live data streaming from a MT4 terminal to a Postgres database. To accomplish this, I'm using dwx-zeromq-connector and I had success when doing this for 2 currency pairs.

The way the connector works is by setting a pub/sub ports and stream data from MT4 using a python script. Now I want to stream 28 currencies, so the number of charts in the MT4 terminal is 28 and the problems of slow subscribers feels very heavy. The messages queue up and no matter what I try, it does not seems to process them any faster.

What I have tried so far is:

• 28 different sub sockets for all pub ports using multiprocessing and daemon threads (not at the same time)
• 1 socket for all 28 pub ports using multiprocessing and daemon threads (not at the same time)
• 28 different classes running 1 daemon thread each, one per pub
• 1 class having 7 different threads and each thread has 1 socket that manages 4 pub ports

None of them improves the performance. Now, looking at this zmq guide, the Black Box Pattern seems to be suitted for what I need. However, I was wondering if I should reduce the amount of pub ports by making that the publishers publish information to only 1 pub socket.

So, can many publishers publish messages to only one port using tcp protocols? How can is the zmq guide what I should really use? The specs of the VPS I'm using is 2 vCPU and 12 GB RAM.

Thank you!

[minrk]

Closing some stale issues. I hope you found what you were looking for! I can't really help out with application design on this repo, that's probably more of a task for stack overflow.