search

ray-project / ray

Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
https://ray.io
Apache License 2.0
32.85k stars 5.57k forks source link

[RLLib] Want to update PolicyClient and PolicyServerInput to the new API stack #46430

Open ajlangley opened 1 month ago

ajlangley commented 1 month ago

Description

My understanding is that PolicyServerInput and PolicyClient don't work with the new API stack, which appears to be due to their use of the older RolloutWorker instead of EnvRunners.

The ability to hook in non-gym environments using this interface has been very valuable to me so far, and I'm interested in making an attempt to update these modules so that we can use the latest algorithms with them.

Currently, I have identified that mainly PolicyServer itself will need some updating, in addition to IOContext (it currently only takes a RolloutWorker in its __init__ method).

My main question about this is, could I get some general guidance on how to update the modules in question without making breaking changes to other parts of the project? My main concern is that IOContext could touch other code, although I'm relatively sure that PolicyServerInput and PolicyClient can be updated without causing issues elsewhere.

Thank you in advance for considering my request!

Use case

The PolicyServerInput feature is really useful for cases in which wrapping the environment in a gym interface is difficult. It would be nice to make sure it gets updated to the new API stack so everyone can continue using it.

simonsays1980 commented 1 month ago

@ajlangley Thanks for raising this issue. The PolicyClient and PolicyServer is definitely on our list to have, but before the Summit 2024 there is no chance to get to this nor guide you in it as the list is already quite full for us. We will come back to this after the Summit.