search

rainwoodman / vast

vala and scientific numerical computation
11 stars 1 forks source link

WIP: prototyping Graph/computing network. #18

Open rainwoodman opened 7 years ago

rainwoodman commented 7 years ago

SubGraph : Graph as a node.

Named arguments?

arteymix commented 7 years ago

I rebased this branch upon master so that you can have access to the function introspetion API.

arteymix commented 7 years ago

I'll probably update this PR if you don't mind to use the new function introspection API.

I'll be in vacation for the coming week though, so don't expect anything before the 22 of december.

By the way, I think we should use a breadth-first evaluator with an async queue and some threads evaluating the graph in parallel.

The idea is to push the variables to evaluate in the queue. Then, each thread pop the head and attempt to realize the node, if so, nice, you evaluate and store the result in the cache, otherwise push all the node dependencies in the queue then the node itself and iterate.

We also need a reuse strategy for items from the cache that become useless. This is why I think the slice allocator could be interesting there. We could use built-in refcounting to clean old variables and reallocate them using Slice.alloc.

I'll also write a chi-squared example if it works well. Do you have some sample data for that test case?

rainwoodman commented 7 years ago

Sorry I've been trapped at work most of the week.

For test data, I'd just fill the field with constant for a starter.

We probably still need a non-threaded walker.

arteymix commented 7 years ago

Sure, in the end we'll probably have various flavours of executors anyway, so let's start with the simplest.

arteymix commented 7 years ago

There's a really nice project called GOCL https://github.com/elima/gocl that provide a GObject wrapper around OpenCL.

It would be interesting to have a graph evaluator that would compile down the graph into a OpenCL kernel and evaluate it on a GPU or any compatible device.

arteymix commented 7 years ago

This https://github.com/elima/gocl/pull/6 should allow us to map dense arrays directly into OpenCL devices.

If we follow a memory mapping approach, we can have a single implementation with different mapping strategy. I wonder if it's possible to have a custom mmap protocol to do stuff like implicit byteswapping.

arteymix commented 7 years ago

I need to fix the current evaluation algorithm because it loops on inout variables. I'm pretty sure we'll need directions on node as well.