Closed belledon closed 4 years ago
@eivinasbutkus can you describe here the BO workflow?
we might need to use a distributed task scheduler in order to deal with the massive compute. Alternatively we could optimize over random subsets of trials (ie for each iteration sample k from N trials) to reduce cost. Things will add up quickly, especially if we consider using multiple chains..
We should also look into profiling the inference query. I can get to that tonight
How many trials are there and how long does it take for one chain? BO is used for very expensive functions, but it's true that it could add up really quickly.
there are 210 and since we are searching for the number of particles time will vary
True. Have you done something like launching Slurm jobs from within Julia function and waiting for result?
that tool is called a distributed asynchronous task scheduler
i've played with dask-distributed
in the past on pythons end. I think I came across some for julia in the past. They are incredibly powerful but equally complex
still profiling but we could use something like this https://github.com/JuliaParallel/ClusterManagers.jl
its taking 500ms to call pybullet once which is huge, it should be around 5ms.. poking around in finer detail
ok i think whats happening is a weird interaction with the JIT. After the first pass, the forward step goes from 430ms down to 25ms which is much more reasonable. inference should be about 2.3s with one particle but right now its at 45s
julia> step = @time GalileoRamp.forward_step(step, scene);
0.433361 seconds (887.39 k allocations: 44.865 MiB, 5.91% gc time)
julia> step = @time GalileoRamp.forward_step(step, scene);
0.023223 seconds (581 allocations: 25.875 KiB)
i'm current playing with PackageCompiler
to see if we can get the 23ms performance https://julialang.github.io/PackageCompiler.jl/dev/examples/plots/
[x] integrate dataset interface
[x] create batch inference
[x] re-integrate posterior visualization