Open rafaqz opened 3 years ago
A valid point. We had a comparison before, but removed it once the only cellular automata we had was the game of life. Now that forest fire is back to a static system again, I can re-run some numbers to that end.
@rafaqz have you seen this: https://juliadynamics.github.io/InteractiveChaos.jl/dev/agents/ ? I don't see something of similar complexity or interactivity in the docs of DynamicGrids.jl. In your readme I did read
[DynamicGrids...] provides better visualization tools.
however. What does this mean, exactly?
Also live parameter manipulation.
Is there a video of this in the docs? I just went through them but I didn't find one.
DynamicGrids.jl was around two orders of magnitude faster than Agents.jl
Yeap, that is true, I also remember it. Notice however that in version 4.0 we have re-written our GridSpace
from scarttch, and it now has a design more similar to DynamicGrids.jl. It is still very much clear to me that DynamicGrids.jl will be faster, but by how much we have to check again.
I only mention the visualisation because you mention it as a positive for mesa in the docs. It's good to see you have some live interfaces working too!
But there's a simple REPLOutput built in, and DynamicGridsGtk.jl, DynamicGridsInteract.jl are image-based live interfaces. You can use ColorSchemes.jl to color the simulation, do multi-grid layouts, and set fonts etc for labels/time counter if you need it. A GifOutput is built-in too. See the gifs in the readme https://github.com/cesaraustralia/DynamicGrids.jl for examples of the output. The interfaces just show the same thing, but as the simulation runs. It can do 100fps pretty easily on a simple simulation.
The InteractOutput has control widgets and start/stop etc like your example. There is a video of it in my JuliaCon talk about half way through, but yeah it should be in the DynamicGridsInteract.jl docs. It's in the DynamicGridsInteract.jl readme now, and you can find it here:
https://www.youtube.com/watch?v=cXzYGHw_DaA&feature=youtu.be
One difference is that params are auto-generated for arbitrary custom model structs - the parameters can be nested anywhere in the model object. There is no setup code like in your example to make the output.
I'm currently abstracting out that auto interface generation here: https://github.com/rafaqz/ModelParameters.jl, in the internal InteractModels.jl package (in the process of registration still). I might borrow some of your code from that example for the planned MakieModels subpackage :)
[DynamicGrids...] provides better visualization tools.
This was written a long time ago when agents didn't really have much for visualisation, Ill edit.
Finally got time to run the GOL benchmarks. These are the life rules in the docs of both packages. DG comes out 2-3 and maybe even 4 orders of magnitude faster for running game of life on varying grid sizes. The last one was looking like taking a whole day for Agents.jl so I killed it.
The comparison is better where there are less agents, and where GPU is less competitive - on small grids.
using Agents, CUDA, DynamicGrids, BenchmarkTools, KernelAbstractions, StaticArrays
rules = (2, 3, 3, 3)
mutable struct Cell <: AbstractAgent
id::Int
pos::Dims{2}
status::Bool
end
function build_model(; rules::Tuple, dims = (100, 100), metric = :chebyshev)
space = GridSpace(dims; metric)
properties = Dict(:rules => rules)
model = ABM(Cell, space; properties)
idx = 1
for x in 1:dims[1]
for y in 1:dims[2]
add_agent_pos!(Cell(idx, (x, y), false), model)
idx += 1
end
end
return model
end
function ca_step!(model)
new_status = fill(false, nagents(model))
for agent in allagents(model)
nlive = nlive_neighbors(agent, model)
if agent.status == true && (nlive ≤ model.rules[4] && nlive ≥ model.rules[1])
new_status[agent.id] = true
elseif agent.status == false && (nlive ≥ model.rules[3] && nlive ≤ model.rules[4])
new_status[agent.id] = true
end
end
for k in keys(model.agents)
model.agents[k].status = new_status[k]
end
end
function nlive_neighbors(agent, model)
neighbor_positions = nearby_positions(agent, model)
all_neighbors = Iterators.flatten(ids_in_position(np,model) for np in neighbor_positions)
sum(model[i].status == true for i in all_neighbors)
end
function setup_agents(s)
model = build_model(rules = rules, dims = (s, s))
for i in 1:nagents(model)
if rand() < 0.5
model.agents[i].status = true
end
end
model
end
# DynamicGrids Life rule states - setting up from scratch for transparancy
const lifestates = (false, false, false, true, false, false, false, false, false),
(false, false, true, true, false, false, false, false, false)
function setup_dynamicgrids(s, nsteps)
output = ResultOutput(rand(Bool, s, s); tspan=1:nsteps+1)
rule = Neighbors(Moore(1)) do hood, state
@inbounds lifestates[state + 1][sum(hood) + 1]
end
output, rule
end
function run_benchmarks(; sizes=(100, 200, 500, 1000, 2000), nsteps=1000)
for s in sizes
println("\n\n###################### $s * $s, $nsteps steps #######################")
output, rule = setup_dynamicgrids(s, nsteps)
println("\nDynamicGrids SingleCPU: $s * $s")
@btime sim!($output, $rule, opt=NoOpt(), proc=SingleCPU());
println("\nDynamicGrids SingleCPU SparseOpt: $s * $s")
@btime sim!($output, $rule, opt=SparseOpt(), proc=SingleCPU());
println("\nDynamicGrids Threaded (5): $s * $s")
@btime sim!($output, $rule, opt=NoOpt(), proc=ThreadedCPU());
println( "\nDynamicGrids Threaded SparseOpt (5): $s * $s")
@btime sim!($output, $rule, opt=SparseOpt(), proc=ThreadedCPU());
println( "\nDynamicGrids CUDA GPU (5): $s * $s")
@btime CUDA.@sync sim!($output, $rule, opt=SparseOpt(), proc=CuGPU());
println("\nAgents: $s * $s")
model = setup_agents(s)
@btime for i in 1:$nsteps step!($model, $dummystep, $ca_step!, 1) end
end
end
julia> run_benchmarks()
###################### 100 * 100, 1000 steps #######################
DynamicGrids SingleCPU: 100 * 100
44.933 ms (5591 allocations: 548.94 KiB)
DynamicGrids SingleCPU SparseOpt: 100 * 100
37.041 ms (6596 allocations: 1.09 MiB)
DynamicGrids Threaded (5): 100 * 100
22.456 ms (75627 allocations: 10.46 MiB)
DynamicGrids Threaded SparseOpt (5): 100 * 100
25.964 ms (78354 allocations: 11.05 MiB)
DynamicGrids CUDA GPU (5): 100 * 100
82.766 ms (542711 allocations: 47.80 MiB)
Agents: 100 * 100
4.773 s (10001000 allocations: 1.80 GiB)
###################### 200 * 200, 1000 steps #######################
DynamicGrids SingleCPU: 200 * 200
185.801 ms (5593 allocations: 608.59 KiB)
DynamicGrids SingleCPU SparseOpt: 200 * 200
147.550 ms (6598 allocations: 1.16 MiB)
DynamicGrids Threaded (5): 200 * 200
81.241 ms (75705 allocations: 10.52 MiB)
DynamicGrids Threaded SparseOpt (5): 200 * 200
64.855 ms (77560 allocations: 11.10 MiB)
DynamicGrids CUDA GPU (5): 200 * 200
46.048 ms (183169 allocations: 11.71 MiB)
Agents: 200 * 200
24.371 s (40002000 allocations: 7.19 GiB)
###################### 500 * 500, 1000 steps #######################
DynamicGrids SingleCPU: 500 * 500
1.438 s (5593 allocations: 1022.34 KiB)
DynamicGrids SingleCPU SparseOpt: 500 * 500
1.022 s (6600 allocations: 1.67 MiB)
DynamicGrids Threaded (5): 500 * 500
425.345 ms (75671 allocations: 10.92 MiB)
DynamicGrids Threaded SparseOpt (5): 500 * 500
307.628 ms (78796 allocations: 11.64 MiB)
DynamicGrids CUDA GPU (5): 500 * 500
165.871 ms (544455 allocations: 17.93 MiB)
Agents: 500 * 500
322.102 s (250002000 allocations: 44.94 GiB)
###################### 1000 * 1000, 1000 steps #######################
DynamicGrids SingleCPU: 1000 * 1000
6.092 s (5597 allocations: 2.43 MiB)
DynamicGrids SingleCPU SparseOpt: 1000 * 1000
4.545 s (6604 allocations: 3.47 MiB)
DynamicGrids Threaded (5): 1000 * 1000
1.192 s (78236 allocations: 12.44 MiB)
DynamicGrids Threaded SparseOpt (5): 1000 * 1000
1.318 s (80120 allocations: 13.48 MiB)
DynamicGrids CUDA GPU (5): 1000 * 1000
567.078 ms (1769255 allocations: 39.14 MiB)
Agents: 1000 * 1000
1637.079 s (1000002000 allocations: 179.75 GiB)
###################### 2000 * 2000, 1000 steps #######################
DynamicGrids SingleCPU: 2000 * 2000
29.593 s (5597 allocations: 8.17 MiB)
DynamicGrids SingleCPU SparseOpt: 2000 * 2000
22.071 s (6604 allocations: 10.63 MiB)
DynamicGrids Threaded (5): 2000 * 2000
5.810 s (76190 allocations: 18.11 MiB)
DynamicGrids Threaded SparseOpt (5): 2000 * 2000
5.591 s (79195 allocations: 20.62 MiB)
DynamicGrids CUDA GPU (5): 2000 * 2000
2.193 s (6546754 allocations: 122.07 MiB)
Agents: 2000 * 2000
^CERROR: InterruptException:
Wow, nice!
Snowed under with other engagements at the moment, but will certainly take a look at these and add something about this in the docs soon.
@rafaqz sorry for the delay on this one. We're going to do some reshuffling of our comparisons - this dedicated repo is where we'll ultimately put your comparison code. Documentation of the benchmarks will stay in the Agents.jl docs once all this shuffle is complete.
Re-ran the GOL benchmarks with the latest version of agents and dynamicgrids, and now agents is only 2 times slower in the single-cpu case, which is a win in my opinion for agents given the previous benchmark :D
julia> @benchmark run_model_dynamicgrids()
BenchmarkTools.Trial: 73 samples with 1 evaluation.
Range (min … max): 67.655 ms … 83.744 ms ┊ GC (min … max): 0.00% … 0.00%
Time (median): 68.477 ms ┊ GC (median): 0.00%
Time (mean ± σ): 68.645 ms ± 1.846 ms ┊ GC (mean ± σ): 0.00% ± 0.00%
▁ ▁ ▃▃ █▃
▄▇▄█▇▄▄▁▇▁▁▁▁█▁▇▄▄▄▇██▄▁██▆▆▆▆▁▇▄▄▄▁▁▁▁▁▄▁▁▄▄▁▁▁▁▁▁▁▁▁▁▁▁▁▄ ▁
67.7 ms Histogram: frequency by time 69.9 ms <
Memory estimate: 269.67 KiB, allocs estimate: 2541.
julia> @benchmark run_model_agents()
BenchmarkTools.Trial: 36 samples with 1 evaluation.
Range (min … max): 137.426 ms … 158.551 ms ┊ GC (min … max): 0.00% … 0.00%
Time (median): 138.684 ms ┊ GC (median): 0.00%
Time (mean ± σ): 140.068 ms ± 4.688 ms ┊ GC (mean ± σ): 0.00% ± 0.00%
▂▄█▆ ▂
█████▆█▁▁▄▁▁▁▁▁▁▁▁▁▁▄▁▁▁▁▁▁▁▄▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▄▁▁▁▁▁▄ ▁
137 ms Histogram: frequency by time 159 ms <
Memory estimate: 1.31 MiB, allocs estimate: 34889.
But I'm not sure how to handle the matter here since we don't have this model in the benchmarks, probably something more generic about cellular automata efficiency of dynamicgrids should be put in the docs of Agents.jl itself
I am closing this issue, because now there are instructions online of how to contribute new frameworks here. DynamicsGrids.jl is not an agent based modelling framework which means that from the examples we have in this repo, it can simulate forest fire and Schelling. Or at least, I don't see how it could do wolf sheep grass. In any case, @rafaqz if you want to add your framework to our comparison for only the forest fire or also the Schelling feel free to follow the README instructions and contribute a PR.
Actually, I'll keep this open until you put this PR in. Just making it clear that we won't be doing this PR for you (and the same goes for any other framework; if someone wants to be part of the comparison they have to do the PR).
The code is in the readme, and I have personally written a lot of code at your request, @datseris ;)
But sure, I will put a PR together at some stage, thanks for keeping this open and reminding me. I honestly just have too many other packages to get time to do this.
One question: are GPUs allowed?
Sure, but our comparison repo doesn't run on GPU.
For fixed grid models like forest fire, DynamicGrids.jl was around two orders of magnitude faster than Agents.jl the last time I checked.
It also has a good range of visualisation tools - REPL, GTK and web outputs. Also live parameter manipulation. It might be nice to mention that in the docs where you are comparing the forest fire model with mesa.
It wont run a lot of the models you define here, but it's pretty specialised for high-performance gridded models.