search

ledatelescope / bifrost

A stream processing framework for high-throughput applications.
BSD 3-Clause "New" or "Revised" License
66 stars 29 forks source link

inter-node communication block and multi-node pipeline control #21

Open telegraphic opened 8 years ago

telegraphic commented 8 years ago

A thought, and part of a longer-term goal: can we implement an inter-node communication block? That is, could we implement an ethernet-based "virtual ring" that packetizes data up and sends it over from one computer to another? Or more explicitly have a "to another node" packetizer block, and a matching "from another node" depacketizer block?

Relatedly, is there an elegant way to launch and monitor pipelines on multiple systems? For example, when we run the LEDA pipeline we spin up 22 pipelines across 11 computers; if bifrost could facilitate pipeline bringup across nodes it would be great.

While these functions would be pretty darn neat, it would also be a massive time sink. But some thought now might make it easier in the future?

jaycedowell commented 2 years ago

I've been thinking about the inter-node communication aspect with the Verbs/RDMA work. I've started on a rdma.py module that should enable sending the contents of a ring, including when a sequence changes, in an "easy" way. The implementation leaves all of the packetization/depacketization details up to RDMA.