Refactor scheduler to remove duplicity/unused code

[iduartgomez]

We got right now:

• local_scheduler.rs
• distributed_scheduler.rs
• base_scheduler.rs
• dag_scheduler.rs

local and distributed have some duplicate code still (basically the event loop and run job) which could be factored into a common trait (or pull it inside impl_common_scheduler_funcs macro too).

Then we have currently the base_scheduler (or NativeScheduler trait) which should be merged with dag_scheduler and made clear. Initially NativeScheduler was created to hide implementationfrom the public API (DAGScheduler can in theory be implemented by the user, but first a clear API should be found), NativeScheduler should implement DAGScheduler if we decide to go along this path. Then DAHScheduler should be required to context which would be generic over it (I guess).

Nothign to pressing but must do some clean up around all this eventually.

[rajasekarv]

I am refactoring the networking layer and scheduler heavily. It is coming very nicely. Removed the dependency of capnp. The deployment process is extremely easy now. I feel that this along with block manager is the most important part for a strong stable foundation. It should be finished in a week or two(atleast all essentials - spark has too many configurable things).

Compile times are surprisingly low now than when I started. For a 300 line project using this library along with parquet dependency, cargo check is barely taking 0.30 to 0.40 secs in my laptop. Debug build is taking like 3-5 secs. Release build is like 18-22 secs. Not at all bad.

@iduartgomez One thing I noticed is that, after the job completes, it is taking some time to exit the process. It is probably due to the resource freeing logic. Can you check that if time permits?

[iduartgomez]

How long is "some time"? There is some waiting forced in order to allow for the executors to end up whatever they were doing in other threads upon receiving a "shutdown signal", could be removed but is not very long.

Cargo check wasn't too bad for me after the first incremental compilation but faster compilation is always welcome :D Good to hear about that, and removing cpnp will make life easier for certain things (harder for others maybe? we will see), are we rolling our own RCP system based on bincode then?

Anyway, if I can help with the block manager and/or deployment let me know!

[iduartgomez]

Re. the deployment process, do you think creating a binary app for running an independent master which handles all the deployment/submission of the different jobs, resource allocation etc. (like a standalone Spark cluster does) would be a good addition and something we should tackle now? If so I would like to get onto that if that's not what you are working on.

Remotely deploying each job by calling to 'scp' is an ok workaround now but in the long run we probably want an application that handles all this, can run a webserver with lists of jobs, metrics, manage jobs and resources etc. There is a lot of work to do on all that but IDK how prioritary do you think it is, should I wait to see your changes before working on any of that?

[rajasekarv]

For standalone this is fine. But people can always extend it to Yarn, Mesos, Kubernetes schedulers depending on their environment. In the future, maybe we can provide the alternatives ourselves. I would say Kubernetes scheduler might be the most sought after one nowadays

[rajasekarv]

We do need to think about developing UI for monitoring jobs and tasks which is very essential in production. Not a priority now. But something which we eventually need to.