search

CNCLgithub / GalileoEvents

Galileo + Events
MIT License
2 stars 0 forks source link

model analysis todo #3

Closed belledon closed 4 years ago

belledon commented 4 years ago
belledon commented 4 years ago

@eivinasbutkus can you describe here the BO workflow?

we might need to use a distributed task scheduler in order to deal with the massive compute. Alternatively we could optimize over random subsets of trials (ie for each iteration sample k from N trials) to reduce cost. Things will add up quickly, especially if we consider using multiple chains..

We should also look into profiling the inference query. I can get to that tonight

eivinasbutkus commented 4 years ago

How many trials are there and how long does it take for one chain? BO is used for very expensive functions, but it's true that it could add up really quickly.

belledon commented 4 years ago

there are 210 and since we are searching for the number of particles time will vary

eivinasbutkus commented 4 years ago

True. Have you done something like launching Slurm jobs from within Julia function and waiting for result?

belledon commented 4 years ago

that tool is called a distributed asynchronous task scheduler i've played with dask-distributed in the past on pythons end. I think I came across some for julia in the past. They are incredibly powerful but equally complex

https://distributed.dask.org/en/latest/#

belledon commented 4 years ago

still profiling but we could use something like this https://github.com/JuliaParallel/ClusterManagers.jl

belledon commented 4 years ago

its taking 500ms to call pybullet once which is huge, it should be around 5ms.. poking around in finer detail

belledon commented 4 years ago

ok i think whats happening is a weird interaction with the JIT. After the first pass, the forward step goes from 430ms down to 25ms which is much more reasonable. inference should be about 2.3s with one particle but right now its at 45s 

julia> step = @time GalileoRamp.forward_step(step, scene);
  0.433361 seconds (887.39 k allocations: 44.865 MiB, 5.91% gc time)

julia> step = @time GalileoRamp.forward_step(step, scene);
  0.023223 seconds (581 allocations: 25.875 KiB)
belledon commented 4 years ago

i'm current playing with PackageCompiler to see if we can get the 23ms performance https://julialang.github.io/PackageCompiler.jl/dev/examples/plots/