fonsp / Pluto.jl

🎈 Simple reactive notebooks for Julia
https://plutojl.org/
MIT License
4.94k stars 285 forks source link

Parallelize evalutation of independant cells #1079

Closed MasonProtter closed 3 years ago

MasonProtter commented 3 years ago

One advantage of Pluto's reactive dependency analysis and general philosophy that I think is currently under-utilized is that when you can prove that two cells are independent of each-other, it shouldn't matter which one runs first, and hence, it also shouldn't matter if they run simultaneously

To be concrete, suppose I have the notebook

# ╔═╡ cell 1
x = 1

# ╔═╡ cell 2
y = 2

# ╔═╡cell 3
z = some_function(x) 

# ╔═╡ cell 4
w = another_function(y)

# ╔═╡ cell 5
yet_another_function(z, w)

which would then have a dependency graph that looks like

  cell1   cell2
    |       |
  cell3   cell4
      \   /
      cell5

Hence, assuming the user behaved themselves, we could have a flow that looks something like

@eval workspace begin
    @sync begin
       @spawn begin
            $cell1
            $cell3
        end
        @spawn begin
            $cell2
            $cell4
        end
    end
    $cell5
end

This could potentially save a lot of time for certain larger, slower notebooks.

Of course, there are also lots of potential pitfalls, like if both cell3 and cell4 were doing dense, multithreaded matrix multiplication this might end up slowing down both of them, hence we'd at the very least want a way to stop Pluto from using threads as a startup argument, but there's also potential for a more fine grained interface.

fonsp commented 3 years ago

Hey Mason! See #4, the problem is that we need to assume that users do not write state-mutating code.