Open kszucs opened 7 years ago
rq worker
or just a pickled python function (I'm not sure which one is better - preference?)I wanted to create a framework for Jupyter Notebooks as well,
For dask there was an example already create by Mrocklin I think no? We can reuse that I guess.
For RQ, do you mean we also spin up a redis instance and things or we completely remove redis from the picture and do what the current satyr implementation did for running arbitrary python functions somewhere?
There is also this: https://github.com/tobilg/mesos-framework/blob/master/examples/
For RQ without redis. Mrockling had an example for running distributed on top of marathon which I've ported to daskathon. Can You implement these two as examples in mentos? Then we'll see what needs to be done in the pending PR.
I think yes, I was a bit busy lately, but I will try to work on it a bit in the next few days
I have another use case. Right now we're moving to https://clickhouse.yandex/ There are no mesos frameworks for it yet. How about implementing one (as a database example)?
We should define a couple of frameworks which would implemented via mentor. A good starting point can be already implemented ones from here and here.
Write them in pure mentos (the core concepts are enough), then refactor to generalize the common parts.
There are a couple of domains we should cover:
I think each of these have different needs.