search

runabol / piper

piper - a distributed workflow engine
Apache License 2.0
481 stars 86 forks source link

Distributed computing issue when deployed on multiple nodes with kubernetes #42

Open OmarSastec opened 4 years ago

OmarSastec commented 4 years ago

I have deployed piper in kubernetes using 5 piper workers/pods .

After proceeding with the tests, I have noticed that all workers are responsible for a single job which is a problem in case I want to make some POST-TRANSCODING actions with the video.

Here's an explained scenario :

  1. worker X gets the job and started transcoding the file and saving the output to a temporary file
  2. worker X finished the transcoding
  3. worker Y tries to make some POST-TRANSCODING actions with the video => Here come the problem, file not found in the worker's tmp directory.

I did a hack using a mounted NFS volume to sync the workers processing but this is very Network IO consuming.

So my question: Is it possible to bind a unique worker to a unique job ?

cranky-coder commented 4 years ago

just curious if you have your deployment files. I am moving this direction and would like to be able to assist