What if we have some code in C, Perl, Ruby or another language and want to have it as a worker inside a pipeline? Currently we can't run do this unless we create a Python worker that executes a process, calling the external code.
A possible implementation to solve this problem is to create a "worker socket" (or other kind of inter-process communication, maybe using ZeroMQ) on Broker so any process that connects there (and authenticates?) can register itself as a worker, independent of how (which language) it's implemented.
What if we have some code in C, Perl, Ruby or another language and want to have it as a worker inside a pipeline? Currently we can't run do this unless we create a Python worker that executes a process, calling the external code. A possible implementation to solve this problem is to create a "worker socket" (or other kind of inter-process communication, maybe using ZeroMQ) on
Brokerso any process that connects there (and authenticates?) can register itself as a worker, independent of how (which language) it's implemented.