Unity hangs indefinitely when Academy.Instance.EnvironmentStep() called on turn-based game

Unity-Technologies / ml-agents

The Unity Machine Learning Agents Toolkit (ML-Agents) is an open-source project that enables games and simulations to serve as environments for training intelligent agents using deep reinforcement learning and imitation learning.

https://unity.com/products/machine-learning-agents

Other

16.93k stars 4.14k forks source link

Unity hangs indefinitely when Academy.Instance.EnvironmentStep() called on turn-based game #4220

Closed applenicks closed 4 years ago

applenicks commented 4 years ago

I am trying to create the right environment to train ML agents on a simple two-player Noughts & Crosses/tic-tac-toe game, and am transitioning from the default auto-step environment to the manually-called Academy.Instance.EnvironmentStep(), as being more suitable for turn-based games. When I include the EnvironmentStep line and press play, unity simply goes into a hang state and needs a force quit (even if I wrap that code into a coroutine).

Reproducing:

behavior type set to Heuristic for both players (pre training), with random moves programmed
Academy.Instance.EnvironmentStep() is called every time the next player has a turn
if I apply it to the first player only (ie comment out if player 2) - it correctly runs and plays a move (before stopping on player 2 with no signal to prompt the next move)
as soon as I apply it to both players, and press play, the game hangs, not even running

Environment: Unity 2019.3.15f1 MLAgents 0.17.0 Tensorflow 1.5.0 Mac OS 10.12.3

andrewcoh commented 4 years ago

Hi @applenicks

Can you copy the entire function where you call EnvironmentStep?

applenicks commented 4 years ago

Here's the function NextPlayerTurn, which itself is triggered by an Action:

private void NextPlayerTurn(bool isP1Turn) { noMovesAllowed = false; if (isPlayer1 && isP1Turn) { this.RequestDecision(); Academy.Instance.EnvironmentStep(); } else if (!isPlayer1 && !isP1Turn)// this must be player 2 and it must be player 2's turn { this.RequestDecision(); Academy.Instance.EnvironmentStep(); //This will crash/hang unity! } }

andrewcoh commented 4 years ago

Is this function being manually called (either directly or indirectly) by CollectObservations or OnActionReceived? This can cause a loop that will prevent the main Update from being called.

applenicks commented 4 years ago

Solved - thank you.

Yes: the function is triggered by an Action that is ultimately called by OnActionReceived, which must lead to the infinite loop described in your branched conversation.

I've since prevented OnActionReceived calling EnvironmentStep() directly, and instead have it toggle the noMovesAllows boolean that's now checked on the independent FixedUpdate loop, which now calls the EnvironmentStep().

Thank you.

andrewcoh commented 4 years ago

Great! Additionally, we've added a catch for this today in #4227. Thank you for raising the issue. I am closing this issue as this has been resolved.

github-actions[bot] commented 3 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.