search

dysonance / Strategems.jl

Quantitative systematic trading strategy development and backtesting in Julia
Other
163 stars 39 forks source link

Parallelism #2

Open dysonance opened 6 years ago

dysonance commented 6 years ago

Should be able to run multiple backtests in parallel when doing parameter optimization runs.

femtotrader commented 6 years ago

Running same backtest code with same input data but with several differents parameters could be a great thing!

I wonder if something like this shouldn't be done using git and a continuous integration platform.

A first idea could be to implement this on top of EC2 with Docker.

Do you have some DevOps skills?

dysonance commented 6 years ago

@femtotrader Stoked to hear you think this would be a good feature. This is probably a major milestone in my mind for this package.

Yeah I have a little bit of experience dealing with stuff like this, we all kinda handle the whole shebang at my job so I've gotten some exposure to this problem space.

Ideally, I would love for this package to be able to use Julia's built-in parallelism and distributed computing support without getting too into the weeds with git/docker/AWS. I was playing with this support a little bit last night. Still new to this feature set, so just learning the ropes of how macros like @parallel, @sync work together and how to get them to share resources. Wasn't having much luck but getting this in place would be a major win.

The git/docker/AWS toolsets are more infrastructure concerns and thus seem outside the scope of what this package aims to do. That being said, we should attempt to develop this functionality with that degree of generalizability and scalability in mind, so that users would ideally be able to set up their own big strategy research development environments and have the code scale well with more computing resources.

femtotrader commented 6 years ago

Do you know price of Julia Run https://juliacomputing.com/products/juliarun.html ? Maybe guys from Julia Computing could help an open source project like this one? Pinging @viralbshah

ViralBShah commented 6 years ago

Pinging @aviks who knows the finance bits and also can help with JuliaRun/JuliaBox.

aviks commented 6 years ago

So a version of JuliaRun is now available to run within the new Juliabox. If you go to https://next.juliabox.com and navigate to the tutorial folder, you'll see a notebook showing you how to scale parallel julia processes via Juliarun. By default, free juliabox users get enough CPUs to run 2 workers, but we can allocate more for either of you if you wanted to test the parallelism on there.

femtotrader commented 6 years ago

http://pachyderm.io/ could also be considered