search

Unity-Technologies / ml-agents

The Unity Machine Learning Agents Toolkit (ML-Agents) is an open-source project that enables games and simulations to serve as environments for training intelligent agents using deep reinforcement learning and imitation learning.
https://unity.com/products/machine-learning-agents
Other
16.91k stars 4.13k forks source link

Can the ML-agent do Image segmentation task? #590

Closed Xunzhaocunzi closed 6 years ago

Xunzhaocunzi commented 6 years ago

Hi all, I have been looking around for a while, hoping to find an example that do the image segmentation in Unity, but I failed to find one.

I have city model purchased from Unity Asset store, which has a variety of pedestrians, vehicles, and traffic signs. What I want to do is to train the ML-agent to recognize those objects using CNN or similar models. I know how to train the model with Tensorflow/Python alone, but I'd like to train the ML-agent within Unity too. Thank you very much!

mmattar commented 6 years ago

Hi @Xunzhaocunzi - what is the Agent in your scenario and presumably you'd like to just collect the data and pipe it to Python?

For the purpose of transferring your data set from Unity to Python you can use the a visual observation (Camera) to represent the input image and encode the ground-truth pixel labels as a numeric observation.

gyanshankar commented 6 years ago

Hi @mmattar - Can you please suggest any example environment for this? I am trying to use visual observation from camera for object detection using tensorflow. The ground truth is going to be a object name and a bounding box of the detected object.

xiaomaogy commented 6 years ago

Hi @gyanshankar , we don't have any example environment right now for this. It seems that you are able to transfer your image data from Unity to Python through visual observation? What's blocking yo now?

mmattar commented 6 years ago

Hi @Xunzhaocunzi - reconnecting to see if you have any additional follow-ups. As @xiaomaogy mentioned, we do not have example environments for your scenario as the primary task (for now at least) that ML-Agents addresses is reinforcement learning.

mmattar commented 6 years ago

Closing this issue for now, but please re-open if you continue facing issues.

lock[bot] commented 4 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.