The idea here is to improve the way we currently handle the deployment of jobs.
Initially the scope for this issue should be very modest: A binary application based on a server/client model which would be run like a daemon in the master and different workers.
This master would be able to assign a driver amongst the available workers for job execution, deploy the compiled executable which contains the parallel work DAG, monitorize status and assign resources for the job in the different workers.
In the future the clean up of the different jobs can be probably translated to this binary too.
The idea here is to improve the way we currently handle the deployment of jobs.
Initially the scope for this issue should be very modest: A binary application based on a server/client model which would be run like a daemon in the master and different workers.
This master would be able to assign a driver amongst the available workers for job execution, deploy the compiled executable which contains the parallel work DAG, monitorize status and assign resources for the job in the different workers.
In the future the clean up of the different jobs can be probably translated to this binary too.