Closed spirali closed 6 years ago
Based on the feedback from the mentioned Reddit discussion, our long-term goals and internal discussion, this is the list of issues to work on, their [priority], (asignee) and their sub-tasks.
Requested by several people in the discussion, seems like a good idea anyway. For now with Capnp.
DataStore
API with direct calls. (@spirali) [high]Multiple options, priorities may vary. (@spirali)
Pythonize the client API.
rain start
and running on OpenStack, Exoscale, AWS. Does not have to be a part of CI (even for running locally). Depends on / part of #37.Replace capnp RPC and the current monitoring dashboard HTTP API with common protocol. Part of #11 (more discussion there) but specific to the public API.
Lower priority, best based on real use-cases. Ideas: numpy subtasks, C++/Rust subworkers
utils/bench/simple_task_scaling.py
. The results as of 0.2 are here.First, I start with my todo list as looked like before the reddit post:
The list of items that was actually in our long term goals, but we should reconsider its priority.
Implement additional subworkers. It should be relatively easy to implement a simple library for e.g. Rust and C++ that provide basic subworker interface and allows to use simply use Rust/C++ code in Rain. There are some open questions how should API exactly look like, but prepaparing some initial prototype should be no problem. I think that only a real question is if we should wait for decesion about new RPC protocol. The good thing is that worker<->subworker is quite separated from other communication (because it referes to local objects - e.g. it does not even use data store API, hence we do not have wait for datastore revamp).
Implement additional clients. It seems that having non-Python clients is more popular than we expected. As far I know, we did not discuss this option too much; however create some working prototype should be relatively easy. However, the qestion about waiting for new RPC is more serious here.
Is "Python subworker as a library" necessary? I have the feeling that for each environment where we can transfer a function to a subworker from a client in reasonable (and portable) way then we should do it that way. The overhead of transferring a function is minimal (it is done only once) and flexibility is huge. I consider building a "fixed" subworker as a kind of side-step where there is no such option (C++/Rust [?*])
I can imagine some scenarios where a python worker could be useful:
Also, the built-in pytask subworker can be trivially implemented as one such subworker task (with a bit of unpacking logic) and so it is not much more work.
I think that even you have a stable pipeline then sending it to subworkers costs nothing (especially compared to the fact that you use Python) and setting up the infrastructure with own subworkers is always more painful (admin work, changes & updates).
Pypy actually works right now - there is no problem to cloudpickle an object in CPython and unpickle it in Pypy. Capnp works in pypy.
Cloudpickle transports Python bytecode not binary code, so if you have a library installed on both ends, there is no ABI problem.
However, I see now that it can be useful to define tasks that may be called e.g. from Java client where cloudpickle is not easily accessible.
PR #40 implements replacement of DataStore API with direct calls
PR #52 implements Exoscale deployment scripts.
These are what I think useful in future:
Transitioned to #64 after v0.3 release
We have received many feedbacks from our reddit post (https://www.reddit.com/r/rust/comments/89yppv/rain_rust_based_computational_framework/). I think that now is time to recap our plans and maybe reconsider priorities to reflect real needs of users. This is meant as a kind of brainstorming white board; each individal task should have own issue at the end. Also, I would like to focus on a relatively short term road plan (let us say things that could be finished within 3 months), maybe we can create a similar post for our long term plans.
EDIT (gavento): Moved @spirali's TODO to a post below.