GPU is no faster for training, even with visual observations?

[lukemadera]

Is your feature request related to a problem? Please describe. I've read that MLAgents isn't set up to leverage the GPU, except if maybe there are "lots of agents" with "visual observations" and using "SAC", not PPO. Is that true? What are "lots" of agents - 4? 40? 400? With each having 1 camera, at what resolution (I think the default is 64x64)? And even so, PPO does not have a large enough network for the GPU to matter?

MLAgents is great and I started with vector (raycast) observations to train a simple self driving car in under 1 million runs. On my 2015 MacBook Pro it takes about 12 hours. Which is long, but not terrible. I figured I'd buy some nvidia RTX GPUs and get 5x or faster trainings, especially when I switched to visual observations.

I'm now training more complex characters to play a fairly simple puzzle - shoot some targets, move some blocks and I've shifted to visual observations at 128x128 resolution with 4 agent copies running at the same time (using more than that slowed my Macbook to a crawl, so running 30+ copies of the environment as in the examples seems infeasible). I'm using PPO as I want discrete actions and masking. It's taking 3 million or more runs to get to a decent level of training, which is taking 2 days. In reading, things like OpenAIs multiagent hide and seek took 40+ million iterations, which would take far too long to train. I tried on my friend's desktop gaming PC (RTX 2070 GPU) but things still seemed to be CPU bound (CPU was running at 112% and GPU didn't seem taxed much at all). Training was only 2x faster, which I think was just due to the CPU being 5 years newer and faster.

So my question is: can MLAgents be used to train agents for 10+ million iterations in a fast (hours or 1-2 days, not week(s)) way? What hardware or setup can we use to achieve that? Is something wrong with my setup?

Describe the solution you'd like

• In addition to the benchmark reward for the example environments, please list the number of training runs required as well as the hardware used and time to train. Do any of these take over 1 million runs? 10 million? Is MLAgents setup to train more complex models that would take tens of millions of runs?
• Hopefully the answer is yes, in which case, a crucial piece is understanding how to setup MLAgents and the corresponding hardware to train in a reasonable amount of time. Obviously there's a huge difference between something taking 1 day to train and 1 week and it would be immensely helpful to know up front what is realistically possible and what it would take and cost to achieve it.

Describe alternatives you've considered As noted above I've tried visual observations with both CPU & GPU as the inference mode, with tensorflow 1.14, 1.15, 2.0.0b, on both my Macbook Pro and a gaming PC with an RTX 2070 GPU. Training is slow (30k runs takes 478 seconds on the gaming PC and 1079 seconds on the MacBook).

Additional context Here's a scene view of training in the editor with the 4 arenas.

[xiaomaogy]

Hi @lukemadera, you are right, mlagents isn't setup to leverage GPU, so unless you have a scene that heavily uses visual and doing lots of model updates (such as SAC) and has a really large neural net, in which case you will be bound by model update computation, you won't benefit much from using better GPU.

To make training faster, you could use our multi environment argument to launch multiple unity instances to make the speed of generating steps faster. In this case more and better CPU definitely helps a lot. To go beyond that, you would need some kind of distributed training setting, which becomes a lot more complicated. Internally we are still working on such a solution and hoping to integrate into our mlagents cloud service in the future.

[lukemadera]

Thanks for the prompt reply @xiaomaogy though that's too bad to hear. Do you have an idea of how many steps the example environments took to train? And what a realistic upper limit is? It sounds like without being able to leverage the GPU, that means MLAgents is limited to just 1 to 5 million steps to reach training proficiency, meaning we can't train anything other than simple agents and behaviors? Definitely no generalization; would need to make agent actions very specific and then change brains, similar to Wall Jump, since training for even small levels of generalizations (e.g. in my case I can get the agent to learn to shoot OR to carry a box to a switch, but due to catastrophic forgetting, it can't learn both without running them in tandem and doing many millions of runs).

Are there plans to setup MLAgents to leverage GPUs and start to work toward being able to train more complex behaviors? One of the reasons I started using MLAgents in the first place was to work on generalization. Several articles I read said an issue is having realistic simulation environments beyond simple Atari games and OpenAI gym and Unity with MLAgents sounded like the perfect solution! But if we're basically performance capped to only a few million runs, that seems to take all the wind out of the sails unfortunately.

Simply put: my goal is to be able to start to replicate things like OpenAIs multi agent hide and seek (or even better OpenAI Five, but obviously that's much harder). I knew it would be costly to train and take tens or hundreds of millions of runs, but it needs to be able to be done in a reasonable amount of time. More specifically, we're building an AI speech therapy game that will have several mini games that leverage AI, so we need to train a lot of agents with a lot of different behaviors. It sounds like at present I'll have to use the multi-brain approach and be limited to simple behaviors for now. It would be awesome to be able to do more!

For now though, this seems like a huge limitation and it would be nice to have it noted on the main readme page, so others know up front what to expect and don't have to figure it out the hard way after weeks of failed training as I went through.

[lukemadera]

It also would be great to provide suggested alternatives in the meantime for how to best train non trivial behaviors. The multi-brain swapping (break down the task into a set of simple behaviors that each can be trained in 2 million steps or less) is the best way I could think of, but do you recommend something else? 2 examples of behaviors I've been stuck on for weeks:

1. walking up a ramp. I can train the humanoid character to move to a switch, to move to a box, lift it, and carry it to a switch, but if the switch is on a ramp, it will just move directly toward the switch, no matter what side it is on, and then just get stuck running into the wall, below the switch, because it does not know to run away from the target to get to the ramp to run up to it. I've tried many curriculums - first starting on the ramp, then at the bottom of the ramp (it can do these - just needs to run straight) but as soon as I start it slightly to the side of the ramp, it still runs straight to the target, missing the ramp. It seemed to get stuck and after 1.25 million steps of no apparent learning (2 days of training) I gave up. My thought based on reading was that it needs to randomly move onto the ramp and succeed, and then over time it slowly learns a path to success (go to the bottom of the ramp). It does this sometimes, and I think it may have been learning, but as with other examples in literature, it needed a few million more runs to master it. But I need to be able to do those few million runs in a day or less, rather than 3+ days. Especially since it seems tricky to figure out with AI training if it just won't learn with the current system OR it just needs to run (a lot) more to figure it out. Much of the literature, including the MLAgents guide, recommends keeping things (rewards) simple, which makes sense. But that requires lots of runs and exploration to figure it out. It seems I should be expecting at least 10 million runs to learn some of these behaviors, yet because of how long it takes to train, the most I've ever done is 4.5 million. The AI may just need more time to learn.

2. managing 4 characters. 1 to shoot (targets), 1 to hook, 1 to lift and carry boxes, 1 to jump. I originally tried 1 brain to manage all characters (as it is for the human player in the game) and for very simple tasks I had moderate success in about 3 million runs, but again if there was any directionality required (e.g. shooting or hooking to a target on a wall, so you need to approach from a particular angle, or you must back up far enough to have line of sight to the target) it would get stuck and fail. It also had a hard time changing characters and I had to add code to force it (via action masking) to change to the appropriate character for the given task. If I were to switch to 1 brain per character it would be easier to train (less generalization) BUT what about common behaviors such as movement? It would essentially need a subset of multiple brains at once (e.g. one for movement they all have, then one for the current character). Does MLAgents support multiple brains? Or is there a better way to train an AI to do this? This is just the tip of the iceberg - the original goal was to be able to train an AI to play the game, just like a human would - being able to get in and drive a car or vehicle for example (though again I could switch to a different brain once in the vehicle) and lots of other nuances similar to the ramp issue with navigating terrain and solving more challenging tasks and puzzles. This is starting to feel impossible with MLAgents now. But hopefully there is, or soon will be, a solution I'm missing! Thanks again for all your work on MLAgents!

[harperj]

Hi @lukemadera -- I think some clarification might be needed here with regard to ML-Agents performance. ML-Agents training includes Python/Tensorflow trainers as well as the Unity environment.

For the trainers, we are generally using relatively small models for our examples. Tensorflow supports GPU acceleration out of the box -- but the improvement provided is going to be small when the network and batch size are small. That said, we support both larger fully-connected networks and two different visual encoders (CNN and ResNet). You may see some improvement in performance by using GPU acceleration.

The environments are built using Unity, which naturally is able to take advantage of GPUs for rendering. If you're using visual observations, Unity will need to render them and you should see a benefit from having a powerful GPU. Even better, with a powerful GPU you'll be able to run more parallel environments without performance degrading. If the environment uses vector observations, the GPU won't generally provide any benefit (and you can turn off the rendering with the "no graphics" option).

In our experience the limiting factor in most training scenarios is the environment / simulation time. This can be due to the physics simulation, rendering, or other simulation logic. If you need steps quickly, one of the best first steps is to increase the parallel environments.

Another thing to consider is that it is more difficult to learn from visual observations than from vector observations. If you're looking to reduce the number of steps necessary, consider whether you can provide better/simpler observations or a better reward function.

[lukemadera]

Yes, clarifications would be very helpful @harperj It would be super helpful to have some non trivial example environments, ideally easy (which are what I consider the tasks I'm currently trying to train for but have been struggling for over a month now), medium and hard. And / or a statement that MLAgents currently only works (well) for trivial (small model, simple behaviors) agents currently, but more complex behaviors are being worked on.

Re: GPU acceleration - as I noted I did NOT see improvement in performance - do I need to change a setting? Also as mentioned, my CPU was at 112% so it still seemed CPU bound, and that was with just 1 environment. Are there any working examples of GPU with visual observations and / or parallel environments? I did notice that most of the example environments that had visual observations said they "do not train" - so it seems that even though visual observations are an option, practically speaking they should not be used?

Yes, I read about visual observations being more difficult but also that to capture "arbitrary complexity" you may need them. Where is the "break even" point for when there's so many vector observations that visual would be better (e.g. if there are 1000+ vector observations?). Again all the example environments seemed very simple. In my (still pretty simple) case I have a target to shoot vs a hook switch vs a switch to carry a box onto vs a switch to jump on AND each of those have an "on" and "off" state - so that's already at least 8 states just for simple switches. In the examples I saw a general strategy of storing the distance to the raycast hit AND a key for EACH object type that is 1 or 0 to tell what is hit. So for 250 raycasts, that means each would need to store at least 9 observations right? So already 2500 vector observations. Or is there a more compact way to store observations? Such as assigning a float number to each state, so it's always only 2 observations per raycast but can encode infinite states? This is likely required since you can't change the number of observations later so if you add in a new element to the game, you'd have to retrain from scratch. So we need a way for an agent to differentiate between all game objects (road vs grass vs switch A on vs switch A off vs switch B on vs ..). For the example environments and the self driving car I used vector observations for it was very simple - the car is either on the road or not, and that still required over 500 vector observations to drive well, and that was just on a single track. Are thousands or tens of thousands of vector observations expected and what we should be using?

[harperj]

it seems that even though visual observations are an option, practically speaking they should not be used?

Generally speaking, visual observations are simpler to set up but result in slower training (though there may be exceptions). This blog post we shared last year is an example of an environment in which we found visual observations to be the best approach.

Re: GPU acceleration - as I noted I did NOT see improvement in performance - do I need to change a setting?

See the Tensorflow GPU support page for the trainer side of this. The GPU should be used by the Unity environment without additional configuration.

Are thousands or tens of thousands of vector observations expected and what we should be using?

It's really hard to say, unfortunately. We don't yet have good answers as to the best observation space setup for any given environment. I would be concerned with that large of an observation space and my intuition is that there is a better way.

Unfortunately, though, we really don't have the bandwidth to try to troubleshoot how to get custom environments training. You might consider asking on the forum, as other folks in the community may have practical advice to share.

[lukemadera]

Thanks for the blog post link, though again that's a much simpler agent than what I've been trying though it did confirm the timing of taking 11 hours to train and about 1 million runs being a realistic max for MLAgents. That said, as per this comment, it seems just having one Unity instance with multiple arenas is better? I'm already doing that. As per the comment question, it's unclear to me why num-envs would ever be used, unless in that game there wasn't a good way to create multiple arenas within the same Unity instance. Especially since using num-envs requires pre-building an executable each time, so if you change anything you have to rebuild, which is time consuming. Or am I missing something here? https://github.com/Unity-Technologies/ml-agents/issues/2298#issuecomment-513895001

Yep, GPU config seemed to be working for me, but as things were CPU bound, it didn't matter.

Yes, my intuition was the same - too many vector observations. I totally understand you can't troubleshoot my particular use case; I didn't expect you to :) What I would ask for is to add one or both of:

1. a limitation note mentioning all the above (that trying to train more complex agents than the example environments likely will not work, that visual observations are unlikely to work, that 1 million runs can take about 12 hours to complete and that GPUs don't really help). That information would have saved me a month of time :)

2. more complex example environments (and how many steps were required to train each one). Maybe this can be crowdsourced? I'd be happy to share my experiences and I'd love to hear the experiences of other people as well. Last I checked the forum it didn't seem very active and searching for "gpu" gives an error, but I'm very curious how far others have been able to push MLAgents and what AIs have been created, and how they were trained!

[harperj]

That said, as per this comment, it seems just having one Unity instance with multiple arenas is better?

Actually, in our experience multiple Unity instances with multiple arenas often results in the best performance. In practice most environments have work that needs to be synchronized in a main thread which becomes a bottleneck at some point. We found improvements with parallel environments for almost all of our examples environments.

Also thanks for the suggestions. I think (1) makes a lot of sense and I'll track it as a potential addition to the docs. (2) feels more like something that would be better maintained by the community.

[lukemadera]

Thanks! Good to know a balance of multiple arenas and multiple Unity instances may work best. Just to clarify - to use multiple Unity instances we must first build an executable right? Once the model and rewards are all worked out that can be fine, but in the beginning I often change things and watch in the Editor to see where the AI is struggling and how to adjust.

Secondly, could you share what hardware you used? As I mentioned, even with a <1 year old gaming PC I was at 112% CPU usage and the GPU was barely being used so I'm not sure I could have run more Unity instances since I already seemed maxed out CPU wise.

Thanks! I'd love to help save future people the time with those limitation notes, including the hardware it was trained on, since that can make a big difference. I think it's crucial people have correctly set expectations of what MLAgents can do, what they can do on their current hardware before they invest the time and money to build a machine learning rig just to find out it doesn't help! And I opened a forum post here as per your suggestion. If you know any other places or people who could point me to more complex example environments and agents I'd appreciate it! https://forum.unity.com/threads/what-is-the-most-complex-ai-agent-you-have-trained-and-how-long-did-it-take.913745/

Thank you for the prompt replies and all your work on MLAgents! I'd love to see it make it to the next level of being able to train more complex agents, but I do appreciate all the hard work you have done thus far!

[harperj]

Secondly, could you share what hardware you used? As I mentioned, even with a <1 year old gaming PC I was at 112% CPU usage and the GPU was barely being used so I'm not sure I could have run more Unity instances since I already seemed maxed out CPU wise.

I'm not sure of the system specifics, but generally 112% CPU is referring to 1.12 CPU cores. Modern gaming PCs frequently have 8+ cores. I have specifically tested multiple environments in parallel on a MacBook Pro 13" (2018 I believe) and seen the benefit of parallel environments. Since each environment is an independent application with different compute requirements, the specific number of environments / areas within an environment is something you'll need to tune.

Thank you for the prompt replies and all your work on MLAgents! I'd love to see it make it to the next level of being able to train more complex agents, but I do appreciate all the hard work you have done thus far!

Thanks, :-) -- we've already seen success training fairly complex agents, but since different games / environments can require fairly different observations/actions/rewards it can be difficult to provide general advice to someone working on a new environment. We are working on more complex examples, so stay tuned.

[lukemadera]

Great to know! Again just to make sure - to use multiple parallel environments, you need to do a build and have an executable right, rather than using the Unity editor?

What are some examples of the "fairly complex agents"? Very eager to see the examples!

[lukemadera]

Also @harperj while I understand you don't have time to troubleshoot my specific agent, do you have a general answer to the question of how to best encode arbitrary complexity in vector observations? Specifically would this work: For each raycast store two observations:

1. distance to hit (or -1 if no hit)
2. a float value that is a map to a specific object and state, e.g. 1 if switch A on 2 if switch A off 3 if switch B on 4 if switch B off 5 if enemy A 6 if ally A 7 if road 8 if obstacle ...

That way infinite states can be stored in just 2 vector observations per raycast, which is the only way I see to use vector observations without having 1000+ vector observations. If you instead store 1 or 0 for each state, then with just 10 different states and 100 raycasts, you already are at 1000 vector observations, which obviously does not scale. This is how the example environments were set up, but because they were so simple you could get away with just 3 or so states per ray cast (e.g. block or wall) but for any non trivial environment there's tens or hundreds of states the agent needs to distinguish between.

If there's not a simple answer or if there's a better place to ask this question, feel free to point me there. Thanks!

[AdhamAlHarazi]

@lukemadera Thank you for raising this issue. For me, I spend more than a year, since I was new to Unity as well, building a complex environment such as OpenAI hide and seek. In which, I though I could achieve it. But unfortunately, I needed more computing power to train it. I learned that the hard way as you.

I do believe Unity is shaping the future of DRL with its simulation platform and MLAgents as a DRL framework. However, the computations requirement is beyond the individual's budget. I am wondering if the upcoming unity cloud training would have reasonable price for us.

Also, I am thinking of using visual observation for my next, simpler, project, and I just read in this thread that it may not work!

With that said, I hope to see ML-Agents training going to the cloud for distributed training with reasonable prices.

[lukemadera]

Hi @AdhamAlHarazi just to be clear, compute power and price are NOT the issue. The issue is that MLAgents currently does not work well with visual observations and GPU at all. Until that is addressed and MLAgents can leverage that, the CPU is the limiting factor and that can only be improved so much; the big ML training benefits come from GPUs, not CPUs.

As it stands now, only simple models that can train in around 1 to 2 million steps, using vector observations only, can be trained well. At least that is my understanding and I have never been able to train anything more complex than this (even with a Machine Learning rig with a great GPU) nor have I seen any examples that can either.

In my opinion the "gpu" and "visual observation" options should be removed until they actually work well, otherwise they mislead people like they did for me, that they can be used and will actually work. Or at least add a really big "limitations" / "disclaimer" note that they are there but don't really work yet.

[AdhamAlHarazi]

@lukemadera Interesting information. I am glad to hear that before going to my next project that is solely based on visual observations.

My first project relied only on Rays, vector observations only, and I could not train it on my MBP 2018 despite running it till 10 million steps!

[lukemadera]

Yeah, I wouldn't even bother trying visual observations, or anything that requires more than 2 to 3 million steps to train. Instead I think (for now at least) you need to train multiple brains for specific actions and then switch to the appropriate brain as needed. Not ideal to not be able to generalize at all, but I think that's the only option currently. If you can't train with vector observations only, that likely means your model / behavior is too complex and you need to simplify it. Again not ideal, but with how slow training is and no GPU / way to speed it up, anything over 2 million or so steps seems unmanageable to train.

My new rule of thumb is if I'm not seeing consistent improvement in scores by 1 million steps, I quit and try something else (simplify the behavior even more, tweak rewards and observations, tweak hyperparameters). In much of the research you see examples where it doesn't learn much at all (no clear learning) for 10 million runs or more, and then it learns and jumps up in score. But I think that needs to take 1-2 days of training or less to reach, meaning we need much faster training; otherwise how do you know if you need to keep running it and be patient, or if it will never learn? You can't afford to wait 3 to 5 days or more to get to 10 million steps, still have it not be any better, and then quit then, losing an entire week of time. It either starts learning within a day, or I quit, make a change (simplify), and restart.

[AdhamAlHarazi]

Exactly, machine learning is all about how fast you iterate your workflow. I am already doing what you suggest regarding reward engineering and hyperparameter tuning, but it is frustrating if you could not iterate that fast. GPU is the de facto when it comes to deep learning, and DRL needs that as well. Matrix multiplication is a work best suited for GPUs. I hope such full integration would be soon available to us along with the distributed training in the cloud.

[AdhamAlHarazi]

Dears at Unity @xiaomaogy @harperj @awjuliani @dannylange

It would be nice to know if ML-Agents can replicate OpenAI Five, AlphaGo, or Hide and Seek if the hardware requirements are met. And what is the current ML-Agents can achieve? This benchmark would help practitioners and researches set their expectations correctly when developing their projects in Unity. I do believe developing in a 3D environment such as Unity is the future of deep reinforcement learning, However, benchmarking against those demanding projects is key.

[awjuliani]

Hi @AdhamAlHarazi

This question depends on how you interpret what ML-Agents is. From an environment creation aspect, it is possible to create environments with all of the properties of Dota2, Go, or the Hide and Seek game. As for training such an environment, in all the cases you listed the researchers created very specific custom tailored algorithms for their problems. They also put hundreds of thousands of dollars of compute into solving their respective problems.

We don't think that the OpenAI/DeepMind approach is a useable solution for game developers or independent researchers using ML-Agents. As such, we are focused on providing robust algorithms that work on a broad set of environments, and plan to continue to improve them to work better for more complex tasks.

[lukemadera]

Thanks for the reply @awjuliani I don't expect MLAgents to be able to reach state of the art results such as Dota2 or Hide & Seek, for all the reasons you noted (custom tailored algorithms, expensive compute). However, it would be nice to understand what the current "limit" is for MLAgents, so people know ahead of time what's possible and what is currently out of reach. E.g. assuming we DID have budget for compute, what is the best that can be achieved?

Specifically, what is the most complex training that has been completed with MLAgents thus far (is there anything beyond the demo environments or are those currently the "max" limit we should expect)? And how long it takes to train (number of steps, hours to train, what hardware was used)? Just having that expectation would be hugely useful. And then in the future obviously the closer MLAgents can move toward reaching more complex generalization and models, the better.

Thanks!

[AdhamAlHarazi]

@awjuliani Thank you for your reply. I interpret ML-Agents as a deep reinforcement learning framework that is connected to Unity. Yes, Unity is capable of building such complex environments and even more complex; it is a game engine platform after all capable of simulations as well. However, is ML-Agents capable of solving such complex problems in general, taking out the tailored algorithms for now, giving that the hardware for computing is available?

[awjuliani]

Game developers have used ML-Agents to train agents with behaviors more complicated than those in the example environments we provide. The issue is that in most cases these games and the solutions used are proprietary property of the game developers, and not shared publicly.

We are actually working right now to put together documentation designed to help users of ML-Agents understand the current capabilities, and get a sense of whether a given game or environment will be trainable using the current state of the toolkit. We hope to share this in the coming months externally.

[lukemadera]

Great to hear about the capability documentation, I look forward to it!

When you say proprietary property - any idea of how much of that is "custom algorithms" vs using the giving PPO or SAC algorithms but with custom rewards, curriculum, hyperparameters? It would be helpful to know what the bottleneck is. It has been my understanding that PPO & SAC are still relatively state of the art and can achieve a lot, it just requires tens of millions of steps. To me the bottleneck is compute power, and being able to use GPU(s) so 10 million steps or more can be run in a day. Currently it takes me nearly 5 days to get to 5 million steps, so that's been my current blocker - no real GPU support and too slow to train, not the algorithms or anything else. Is / can that being addressed soon, or are you saying even if we could run 50 million steps easily (within days), they still wouldn't train?

[lukemadera]

Following up on this - have there been any updates to the roadmap for if / when MLAgents will be able to leverage GPU and train 50+ million steps within days and thus be used on more complex and generalized AI? In the meantime has the capability documentation been released? Thanks!

[AdhamAlHarazi]

Is there any update regarding the capability documentation?

[srlowe]

@lukemadera Thank you for your detailed insights here. This has helped me a lot with expectations. Since it's been almost a year since this discussion, please could you tell me - has the situation changed much since then in your opinion?

[lukemadera]

I gave up on this and moved on (both in terms of machine learning approaches and projects - I'm still working in Unity but not using MLAgents), so I'm mostly out of date on this now, but from what I'm aware, I haven't seen any notable improvements unfortunately @srlowe

[playztag]

@lukemadera I am building a rocket-league like a drone simulator using MLagents (drones flying and tagging each other by proximity, one of the drones have possession a virtual ball, if this drone is tagged from behind, the possession of the ball transfers to the tagger... whoever with possession flying to the opposite goal will score a point) It seems like the combination of the recent Unity Hummingbird ML agents project with the mixed with soccer example. While there are several strategies to the game, based on this thread, I'm setting my worst-case expectations to be multiple brains for each specific behavior if I can't get a general brain to work. Thanks!

[lukemadera]

Sounds like a fun simulator @playztag - best of luck and if you are able to train a more advanced model, let me know!

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.