Create IOptimizer Interface in QCAlgorithm

[jaredbroad]

Allow users to set the optimization technique to use. Set the IOptimizer which will generate the parameter sets to backtest.

IOptimizer would take in the last set of results; and return a new parameter set to backtest over.

E.g. A "GridSearchOptimization" would be called once with no results at the start; then it would return linear steps across each of the parameter variations to make up all possible combinations to backtest. A GenericOptimizer would remember breed its parameter sets as to get the best results and generate new ones. Bisect optimizer could bisect and skip large empty parts

The optimizer would be stateful; so could record result sets to perform searches over all results before returning the new parameter set.

[fbertram]

I poked a bit around in the code and am planning to do some coding on Thursday and Friday. Here are my current thoughts:

1) I'd like to extend ParameterAttribute to also hold value ranges for parameters. This way, an algorithm not only publishes its parameters, but also their valid ranges. I feel strongly about this. In addition, I'd like to have 'parameter groups'. Only parameters in the same group will be optimized concurrently and effectively multiply the size of the parameter space. Different groups will optimize sequentially. Regardless of the optimizer type (grid, genetic, other), this has the potential to greatly reduce the parameter space.

2) I was thinking to derive the optimizer classes from IJobQueueHandler. This way, an optimizer can be instantiated just like an algorithm. Unlike the current job queues, the optimizer would return a new job (or null) for every new parameter set. This will force minor changes in the current job handlers, as these would need to return null for the second and following calls.

3) to make (2) work, the launcher would need to be modified. Mostly, it will contain a while (leanEngineSystemHandlers.JobQueue.NextJob(...)), instead of just going through things just once.

4) so far, it seems that the engine will create a new thread to run each job in. If not, there is a bit of work required to make sure the jobs are running in parallel to the launcher.

5) a mechanism to feed results back into the job queue is needed. This is required under all circumstances: all optimizers needs to know when jobs are finished to limit the number of concurrent threads to a reasonable number; this will happen by blocking the NextJob call until a new slot is available. In addition, genetic optimizers needs to know the results of previous populations in order to make choices for the next generation. I am not 100% sure how this will look like, yet.

6) the optimizer needs to dump it's results. This could happen in a Shutdown method added to the job queue interface. For symmetry reasons, this would probably also call for an Initialize method... not sure what we would use that for, but maybe this helps in setting up initial populations.

It seems your ideas were slightly different and some of this might be more clear, once I am able to share some code.

Any comments welcome!

[jameschch]

A few more notes: the GAF framework is more than adequate to handle generational relationships internally. It also has support for parallel executions using the TPL. I now have parallel backtests and a polished configuration model in my fork.

It may be you could just reuse this capability rather than implement a custom threading model, at least for organic optimization. With the flexibility of custom generational operators in GAF, I quite imagine it would also be possible to perform other optimization methods using this framework.

[worthy7]

@jaredbroad Me and James are both actively looking to further the LeanOptimizer project and I'm just wondering if you're staying quiet due to what you said on the forums, about this being perhaps a conflict of interest in terms of QuantConnects business model. If it is then we need to know what work is and isn't worth doing. If it isn't then can we get an update please?

[jameschch]

I personally do not see a great deal of commonality between the simple genetic console app that I've been working on (predominantly for my own amusement), and the cloud-based enterprise grade machine learning solution that will be offered through QC.

I'd be happy to lend support by supplying my findings with relation to pertinent frameworks, for instance, I've thoroughly road-tested GeneticSharp and would recommend its use.

[Sdoof]

Hey folks, Just to check. I see there has been little activity is this project dead/"on ice"?

[jaredbroad]

No not dead; putting the pieces together slowly. We recently upgraded the backend to support optimizer style backtesting; https://www.quantconnect.com/forum/discussion/3057/new-backtesting-cloud

Todo is code side as described in the optimizer project. As much as possible we do bugs before features though but will hopefully get to this soon. https://github.com/QuantConnect/Lean/projects/1

[Sdoof]

Thanks Folks, fantastic to see this is in the pipeline.

[crystalneth]

QuantConnect and LEAN seem really neat, and I'm considering converting over from backtrader because of the speed of QC and rich data sets available. However I've implemented a custom optimization solution on backtrader using AWS lambda and would like to be able to do similar optimizations here. I would be interested in paying for a hosted solution such as QC, however it's hard for me to imagine doing so w/o basic optimization capabilities.

I see that you've been talking about this feature for 4 years now. Can you share some information about the roadmap and direction of the product that we can count on?

[grantwittenberg]

In a similar place here as crystalneth. Is this feature still on the road map or likely possible in the near future?

[omidabdi1]

Any update on IOptimizer?