search

Unity-Technologies / ml-agents

The Unity Machine Learning Agents Toolkit (ML-Agents) is an open-source project that enables games and simulations to serve as environments for training intelligent agents using deep reinforcement learning and imitation learning.
https://unity.com/products/machine-learning-agents
Other
16.86k stars 4.12k forks source link

Visual Observation image size? #2454

Closed superjayman closed 4 years ago

superjayman commented 5 years ago

So, what image size are reasonable to use with ml-agents, no where in the docs you talk about this. Can I pass in say 1080p image for visual input and how is this related to training times?

xiaomaogy commented 5 years ago

Hi @superjayman, larger the image, longer the training would take. In our example environment, we use 84 * 84, and it is already good enough for most of the cases.

harperj commented 4 years ago

Thanks for the discussion. We are closing this issue due to inactivity. Feel free to reopen it if you’d like to continue the discussion.

ZijianHan-VolvoCars commented 3 years ago

Hello, I noticed the way how ml agent create the grpc service to send out the observations from Unity. I know the limit for each message in grpc is around 4MB, and it means that the observation of image size cannot be larger than that right? Actually I've been having some issue broadcasting large image with grpc service in Unity ( not exactly using your ml agent), but I saw how you publish image data so just might wanna ask here

chriselion commented 3 years ago

Hi @ZijianHan-VolvoCars, That's correct, we can't support images that take up more than 4MB for training. By default, we use GIF compression so that might help with some of the image size, but that also might not help much depending on your scene.

I tried this locally by changing the GridWorld example to use uncompressed observations at 750 x 750 (3 channels and 4 bytes/pixel results in ~6.75MB). This results in a "Resource Exhausted" GRPC exception, but we silently ignore the content of the exception and exit Play Mode without any error message, which is something we should handle better.

ZijianHan-VolvoCars commented 3 years ago

Thanks a lot! I fully understand it's unnecessary for ML training to have such large image size. But in case this will be handled later, grpc streaming service might be an update for your rpc service of output : )

github-actions[bot] commented 3 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.