search

Unity-Technologies / ml-agents

The Unity Machine Learning Agents Toolkit (ML-Agents) is an open-source project that enables games and simulations to serve as environments for training intelligent agents using deep reinforcement learning and imitation learning.
https://unity.com/products/machine-learning-agents
Other
17.12k stars 4.15k forks source link

RenderTextureSensor runtime support #4496

Closed Godatplay closed 2 years ago

Godatplay commented 4 years ago

Is your feature request related to a problem? Please describe. Currently the RenderTextureSensor is limited to edit time instead of runtime, and I think this is too limiting. For use cases that are more computer vision related, ML Agents can still be really useful because we can rely on Unity's scene management and especially simulation capabilities in order to create a more advanced training setup in a faster time relative to other CV environments, as the examples demonstrate. However the examples only show a small subset of common use cases. In my case, I have a simulation that runs at runtime that prepares a RenderTexture, which happens in just a couple seconds, and I want to trigger training manually once that is finished.

In my opinion, the most common use case for a RenderTexture is to generate it at runtime for post effects or other custom render pipeline uses, so I find it confusing that this sensor doesn't support assigning a RT at runtime.

Describe the solution you'd like The ability to set the RenderTexture of a RenderTextureSensor a couple seconds into play mode before triggering training.

Describe alternatives you've considered The main alternative is reworking any runtime code so that it occurs at edit time instead, which feels needlessly cumbersome, since the reason I'm using ML Agents over competing environments is its ease of use since those other environments have more robust CV support.

sini commented 4 years ago

I've created a ticket to track this request internally as MLA-1396 and will raise it at our next meeting.

chriselion commented 4 years ago

Hi @Godatplay, Nice avatar :)

This should be a simple change to make if you're interested in submitting a pull request; you'd make RenderTextureSensorComponent.UpdateSensor() also set the sensor's m_RenderTexture, and call this from the RenderTexture set property (for an example, see how CameraSensorComponent handles setting the Camera property). One caveat is that the old and new RenderTextures need to have the same dimensions (if the sensor is non-null) because we can't change the observation size at runtime; we should either throw an exception, or warn and don't set the RenderTexture if there's a mismatch.

github-actions[bot] commented 1 year ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.