Is there any pytorch version of implementation for the control task?

[Yufei-Kuang]

Thanks for the authors to provide such an impressing library!

However, as is described in the title, Is there any pytorch version of implementation for the control task?

I believe that pytorch implementation could be easier to follow for some researchers like me, and I am not sure whether the pytorch version implemented by myself will cause decreased performance.

[TOP1RM]

Hello, I have the same question, but with a Tensorflow 2 implementation instead of 1.14.

Best regards

[liruiluo]

I have waited for the pytorch version for a long time!

[brendenpetersen]

A PyTorch re-implementation of DSO is coming... our team is full-time researchers, not software developers, so while we know this is important for people it's hard to be our priority. Realistically it's on the 6-12 month horizon.

[Yufei-Kuang]

A PyTorch re-implementation of DSO is coming... our team is full-time researchers, not software developers, so while we know this is important for people it's hard to be our priority. Realistically it's on the 6-12 month horizon.

Thanks a lot for your kindly replying! Currently, I am implementing this work via PyTorch by myself based on the detailed supplementary materials from your ICLR2021 and ICML2021 papers. I am looking forward to the official PyTorch implementation.

[brendenpetersen]

I’d recommend forking off this repo instead of re-implementing your own from scratch! There are lots of new tasks and features on the horizon, so a re-implementation will not be able to get those.

Plus a forked feature we can consider for merge!

[Yufei-Kuang]

I’d recommend forking off this repo instead of re-implementing your own from scratch! There are lots of new tasks and features on the horizon, so a re-implementation will not be able to get those.

Plus a forked feature we can consider for merge!

Thanks for your suggestion:). Indeed, I really love this repository, as there are abundant code comments to help me understand the implementation details.

However, we are applying symbolic optimization techniques to a extremely huge and complicated control system. Compared with SOTA performance, the consistency with other libraries and previous designed interfaces has higher priority, which restricts PyTorch to be the only choice.

Anyway, we are delighted to cite these insightful works in our upcoming work. Many thanks!

[brendenpetersen]

If your control task uses PyTorch, that’s totally fine as the task is separate from the compute graph of the DSO policy. So it could mix TF and PyTorch. But hard to say without knowing more about your system.

[TOP1RM]

Do you plan to migrate your code in TensorFlow 2.0 ?

[brendenpetersen]

Probably just PyTorch. It shouldn’t be that hard to support both TF2 and PyTorch at once now that our latest release separates the Policy object from the PolicyOptimizer (algorithms and loss functions), but I don’t see much of an advantage to that. We are developing DSO for our own use cases and publications, and unfortunately don’t have the resources to develop it for the purposes of a software product itself.

The current TF1 implementation of the autoregressive policy — computing priors and making reward function calls outside the compute graph — gets really hairy. When we try to do things like make observations multidimensional, we find ourselves fighting against TF1 limitations. So, I think either TF2 or PyTorch would be huge steps that would make it much easier to mess around with (e.g., adding a transformer policy, graph-based observations).

[brendenpetersen]

But there's lots of other cool stuff coming! For example, we will release support for multi-objective optimization (currently DSO only supports scalar objectives). Or possibly Policys that support hybrid discrete-continuous search spaces. That's why I hate for others to miss out on these by implementing things from scratch. (Though this seems to be the grad student "way"...)

Lastly, I think most people find DSO based on our papers, which are tied to particular methods and (usually) the symbolic regression problem. I'd guess 75% of users just care about symbolic regression (regression task) and don't know or care about DSO's generic capabilities of being a discrete optimization framework. Maybe 20% care about the control task, leaving maybe 5% who actually recognize/care that DSO supports arbitrary symbolic/combinatorial optimiztion tasks and the ability to write/customize your own tasks. Compare that to DEAP, for example, which everyone knows is a generic tool and for which symbolic regression is just one particular use case. DSO is no different--it's just that we happen to have a lot of symbolic regression publications (because we started from funding from a symbolic regression project) so most of our traffic is from people interested in that particular problem. Anyway, I'm rambling now but I'm happy to hear anyone else's perspective on this. We've created what we think is a really cool tool; we recognize people want to use it for particular problems or to advance their own research, and we do want to be supportive of that even as we pursue our own use cases.

[Yufei-Kuang]

But there's lots of other cool stuff coming! For example, we will release support for multi-objective optimization (currently DSO only supports scalar objectives). Or possibly Policys that support hybrid discrete-continuous search spaces. That's why I hate for others to miss out on these by implementing things from scratch. (Though this seems to be the grad student "way"...)

Lastly, I think most people find DSO based on our papers, which are tied to particular methods and (usually) the symbolic regression problem. I'd guess 75% of users just care about symbolic regression (regression task) and don't know or care about DSO's generic capabilities of being a discrete optimization framework. Maybe 20% care about the control task, leaving maybe 5% who actually recognize/care that DSO supports arbitrary symbolic/combinatorial optimiztion tasks and the ability to write/customize your own tasks. Compare that to DEAP, for example, which everyone knows is a generic tool and for which symbolic regression is just one particular use case. DSO is no different--it's just that we happen to have a lot of symbolic regression publications (because we started from funding from a symbolic regression project) so most of our traffic is from people interested in that particular problem. Anyway, I'm rambling now but I'm happy to hear anyone else's perspective on this. We've created what we think is a really cool tool; we recognize people want to use it for particular problems or to advance their own research, and we do want to be supportive of that even as we pursue our own use cases.

I am so impressed and appreciate your long reply. I have learnt much from your repository and your reply in this issue! In fact, I am applying DSO to some commercial software for research purpose. To align with the interfaces and packages used by other collaborators, and to make the code style more uniform for good team collaboration, all the codes we implemented have to be rewritten finally. This is why I have to re-implement this code. Thankfully, the comments in the codes is detailed to help me understand, even though I am not exceling at tensorflow 1.x. Anyway, I am always looking forward to the PyTorch implementation, if it is possible, and thanks for your kindly reply again.

By the way, Though LLM is very very booming, I know that there are many situations where small ML models on CPU only are better choices. Thus, I deeply believe that DSO is a very promising technique in the future, just like LLM today.

[YaqiXie]

@Yufei-Kuang Hi, I'm in search of a PyTorch version to integrate into my codebase as well. Could you possibly share the PyTorch version you've implemented? I would really appreciate it! I only need the basic functions.