search

degiere / optimization-studies

Trading system optimization algorithm studies
9 stars 5 forks source link

Distribute optimization work #1

Open femtotrader opened 10 years ago

femtotrader commented 10 years ago

Hello,

maybe Celery http://www.celeryproject.org/ + Kombu http://kombu.readthedocs.org/ could be used to distribute optimization to several computers.

Celery + HTTP Callback Tasks (Webhooks) http://docs.celeryproject.org/en/master/userguide/remote-tasks.html could be also an interesting solution.

Results of optimization could be stored in a database

Kind regards

degiere commented 10 years ago

That seems reasonable to add another level of parallelization. The GA framework has a partner project called scoop: https://code.google.com/p/scoop/ for distributed tasks and parallel processing. You might look at that and iPython features for this also.

This repo is just a study, but I'd be interesting in seeing and incorporating anything you come up with.

Thanks,

Chris

femtotrader commented 10 years ago

Hi,

Some other ideas to distribute optimization studies could be to use either:

MrJob http://pythonhosted.org/mrjob/ Disco http://discoproject.org/ dumbo http://klbostee.github.io/dumbo/ hadoopy http://www.hadoopy.com/ luigi https://github.com/spotify/luigi streamparse (Apache Storm) https://github.com/Parsely/streamparse

but I have no experience with Hadoop and MapReduce stuffs.

Femto