search

Unity-Technologies / ml-agents

The Unity Machine Learning Agents Toolkit (ML-Agents) is an open-source project that enables games and simulations to serve as environments for training intelligent agents using deep reinforcement learning and imitation learning.
https://unity.com/products/machine-learning-agents
Other
17.19k stars 4.16k forks source link

Grid sensor : Division by zero #5342

Closed TheTrope closed 3 years ago

TheTrope commented 3 years ago

Describe the bug Grid sensor isn't working: Leads to division by zero somewhere

To Reproduce Use the default grid sensor component with basic parameters, nothing more, I was able to create a minimalist env with the same issue. If the gridsensor gameobject is disabled, the training start normaly, when I enable the gridsensor, it's crashing. One wierd thing is the the warning: "[WARNING] Trainer has no policies, not saving anything." when the gridsensor is enabled.

EDIT : I've been able to reproduce the bug in the ml-agent examples, and to make my minimalist env working, by tweaking the grid size parameters. Changing the foodCollector agent grid size to 8,0,8 or 16,0,16 will give differents error. Moving grid size from 16,0,16 to 20,0,20 in my minimalist env makes the training run fine.

Console logs / stack traces

Version information:
  ml-agents: 0.26.0,
  ml-agents-envs: 0.26.0,
  Communicator API: 1.5.0,
  PyTorch: 1.7.1+cu110
[INFO] Listening on port 5004. Start training by pressing the Play button in the Unity Editor.
[INFO] Connected to Unity environment with package version 2.0.0-exp.1 and communication version 1.5.0
[INFO] Connected new brain: HeroBehaviorAlexisMonoHero?team=0
[INFO] Hyperparameters for behavior name HeroBehaviorAlexisMonoHero:
        trainer_type:   ppo
        hyperparameters:
          batch_size:   256
          buffer_size:  2048
          learning_rate:        0.0003
          beta: 0.01
          epsilon:      0.2
          lambd:        0.95
          num_epoch:    3
          learning_rate_schedule:       linear
        network_settings:
          normalize:    False
          hidden_units: 256
          num_layers:   2
          vis_encode_type:      simple
          memory:       None
          goal_conditioning_type:       hyper
        reward_signals:
          extrinsic:
            gamma:      0.99
            strength:   1.0
            network_settings:
              normalize:        False
              hidden_units:     128
              num_layers:       2
              vis_encode_type:  simple
              memory:   None
              goal_conditioning_type:   hyper
        init_path:      None
        keep_checkpoints:       5
        checkpoint_interval:    500000
        max_steps:      20000000
        time_horizon:   64
        summary_freq:   10000
        threaded:       False
        self_play:      None
        behavioral_cloning:     None
[WARNING] Trainer has no policies, not saving anything.
Traceback (most recent call last):
  File "c:\users\alexi\anaconda3\envs\mlagentconda37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\alexi\anaconda3\envs\mlagentconda37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\alexi\anaconda3\envs\mlagentconda37\Scripts\mlagents-learn.exe\__main__.py", line 7, in <module>
  File "c:\users\alexi\anaconda3\envs\mlagentconda37\lib\site-packages\mlagents\trainers\learn.py", line 250, in main
    run_cli(parse_command_line())
  File "c:\users\alexi\anaconda3\envs\mlagentconda37\lib\site-packages\mlagents\trainers\learn.py", line 246, in run_cli
    run_training(run_seed, options)
  File "c:\users\alexi\anaconda3\envs\mlagentconda37\lib\site-packages\mlagents\trainers\learn.py", line 125, in run_training
    tc.start_learning(env_manager)
  File "c:\users\alexi\anaconda3\envs\mlagentconda37\lib\site-packages\mlagents_envs\timers.py", line 305, in wrapped
    return func(*args, **kwargs)
  File "c:\users\alexi\anaconda3\envs\mlagentconda37\lib\site-packages\mlagents\trainers\trainer_controller.py", line 173, in start_learning
    self._reset_env(env_manager)
  File "c:\users\alexi\anaconda3\envs\mlagentconda37\lib\site-packages\mlagents_envs\timers.py", line 305, in wrapped
    return func(*args, **kwargs)
  File "c:\users\alexi\anaconda3\envs\mlagentconda37\lib\site-packages\mlagents\trainers\trainer_controller.py", line 107, in _reset_env
    self._register_new_behaviors(env_manager, env_manager.first_step_infos)
  File "c:\users\alexi\anaconda3\envs\mlagentconda37\lib\site-packages\mlagents\trainers\trainer_controller.py", line 268, in _register_new_behaviors
    self._create_trainers_and_managers(env_manager, new_behavior_ids)
  File "c:\users\alexi\anaconda3\envs\mlagentconda37\lib\site-packages\mlagents\trainers\trainer_controller.py", line 166, in _create_trainers_and_managers
    self._create_trainer_and_manager(env_manager, behavior_id)
  File "c:\users\alexi\anaconda3\envs\mlagentconda37\lib\site-packages\mlagents\trainers\trainer_controller.py", line 140, in _create_trainer_and_manager
    create_graph=True,
  File "c:\users\alexi\anaconda3\envs\mlagentconda37\lib\site-packages\mlagents\trainers\trainer\rl_trainer.py", line 119, in create_policy
    return self.create_torch_policy(parsed_behavior_id, behavior_spec)
  File "c:\users\alexi\anaconda3\envs\mlagentconda37\lib\site-packages\mlagents\trainers\ppo\trainer.py", line 231, in create_torch_policy
    separate_critic=True,  # Match network architecture with TF
  File "c:\users\alexi\anaconda3\envs\mlagentconda37\lib\site-packages\mlagents\trainers\policy\torch_policy.py", line 70, in __init__
    tanh_squash=tanh_squash,
  File "c:\users\alexi\anaconda3\envs\mlagentconda37\lib\site-packages\mlagents\trainers\torch\networks.py", line 592, in __init__
    self.network_body = NetworkBody(observation_specs, network_settings)
  File "c:\users\alexi\anaconda3\envs\mlagentconda37\lib\site-packages\mlagents\trainers\torch\networks.py", line 194, in __init__
    self.normalize,
  File "c:\users\alexi\anaconda3\envs\mlagentconda37\lib\site-packages\mlagents\trainers\torch\networks.py", line 54, in __init__
    normalize=normalize,
  File "c:\users\alexi\anaconda3\envs\mlagentconda37\lib\site-packages\mlagents\trainers\torch\utils.py", line 207, in create_input_processors
    obs_spec, normalize, h_size, attention_embedding_size, vis_encode_type
  File "c:\users\alexi\anaconda3\envs\mlagentconda37\lib\site-packages\mlagents\trainers\torch\utils.py", line 162, in get_encoder_for_obs
    return (visual_encoder_class(shape[0], shape[1], shape[2], h_size), h_size)
  File "c:\users\alexi\anaconda3\envs\mlagentconda37\lib\site-packages\mlagents\trainers\torch\encoders.py", line 174, in __init__
    kernel_gain=1.41,  # Use ReLU gain
  File "c:\users\alexi\anaconda3\envs\mlagentconda37\lib\site-packages\mlagents\trainers\torch\layers.py", line 49, in linear_layer
    layer = torch.nn.Linear(input_size, output_size)
  File "c:\users\alexi\anaconda3\envs\mlagentconda37\lib\site-packages\torch\nn\modules\linear.py", line 83, in __init__
    self.reset_parameters()
  File "c:\users\alexi\anaconda3\envs\mlagentconda37\lib\site-packages\torch\nn\modules\linear.py", line 86, in reset_parameters
    init.kaiming_uniform_(self.weight, a=math.sqrt(5))
  File "c:\users\alexi\anaconda3\envs\mlagentconda37\lib\site-packages\torch\nn\init.py", line 381, in kaiming_uniform_
    std = gain / math.sqrt(fan)
ZeroDivisionError: float division by zero

Environment (please complete the following information):

TheTrope commented 3 years ago

Another follow up:

When importing (or selection ) an onnx trained with the grid sensor, I have this error (x8 times)

Unexpected error while evaluating model output version_number. System.ArgumentException: Cannot reshape array of size 32768 into shape with multiple of 768 elements
  at Unity.Barracuda.TensorExtensions.Reshape (Unity.Barracuda.TensorShape shape, System.Int32[] size4Dor8D) [0x0018c] in C:\Users\alexi\Dev\MyProject\Library\PackageCache\com.unity.barracuda@1.4.0-preview\Barracuda\Runtime\Core\TensorExtensions.cs:495 
  at Unity.Barracuda.ModelAnalyzer.ListTemporaryTensorShapes (Unity.Barracuda.Model model, System.Collections.Generic.IDictionary`2[TKey,TValue] inputShapes, System.Collections.Generic.IDictionary`2[System.String,System.Nullable`1[Unity.Barracuda.TensorShape]]& shapesByName) [0x00c2f] in C:\Users\alexi\Dev\MyProject\Library\PackageCache\com.unity.barracuda@1.4.0-preview\Barracuda\Runtime\Core\Backends\ModelAnalyzer.cs:410 
  at Unity.Barracuda.ModelAnalyzer.TryGetOutputTensorShape (Unity.Barracuda.Model model, System.Collections.Generic.IDictionary`2[TKey,TValue] inputShapes, System.String output, Unity.Barracuda.TensorShape& shape) [0x00007] in C:\Users\alexi\Dev\MyProject\Library\PackageCache\com.unity.barracuda@1.4.0-preview\Barracuda\Runtime\Core\Backends\ModelAnalyzer.cs:586 
  at Unity.Barracuda.Editor.NNModelEditor+<>c__DisplayClass33_0.<OnEnable>b__2 (System.String i) [0x00006] in C:\Users\alexi\Dev\MyProject\Library\PackageCache\com.unity.barracuda@1.4.0-preview\Barracuda\Editor\ONNXModelImporterEditor.cs:209 
UnityEngine.Debug:LogError (object)
Unity.Barracuda.D:LogError (object) (at Library/PackageCache/com.unity.barracuda@1.4.0-preview/Barracuda/Runtime/Core/Internals/Debug.cs:131)
Unity.Barracuda.Editor.NNModelEditor/<>c__DisplayClass33_0:<OnEnable>b__2 (string) (at Library/PackageCache/com.unity.barracuda@1.4.0-preview/Barracuda/Editor/ONNXModelImporterEditor.cs:214)
System.Linq.Enumerable:ToList<string> (System.Collections.Generic.IEnumerable`1<string>)
Unity.Barracuda.Editor.NNModelEditor:OnEnable () (at Library/PackageCache/com.unity.barracuda@1.4.0-preview/Barracuda/Editor/ONNXModelImporterEditor.cs:204)

Is it related? or is it a separate issue?

surfnerd commented 3 years ago

Another follow up:

When importing (or selection ) an onnx trained with the grid sensor, I have this error (x8 times)

Unexpected error while evaluating model output version_number. System.ArgumentException: Cannot reshape array of size 32768 into shape with multiple of 768 elements
  at Unity.Barracuda.TensorExtensions.Reshape (Unity.Barracuda.TensorShape shape, System.Int32[] size4Dor8D) [0x0018c] in C:\Users\alexi\Dev\MyProject\Library\PackageCache\com.unity.barracuda@1.4.0-preview\Barracuda\Runtime\Core\TensorExtensions.cs:495 
  at Unity.Barracuda.ModelAnalyzer.ListTemporaryTensorShapes (Unity.Barracuda.Model model, System.Collections.Generic.IDictionary`2[TKey,TValue] inputShapes, System.Collections.Generic.IDictionary`2[System.String,System.Nullable`1[Unity.Barracuda.TensorShape]]& shapesByName) [0x00c2f] in C:\Users\alexi\Dev\MyProject\Library\PackageCache\com.unity.barracuda@1.4.0-preview\Barracuda\Runtime\Core\Backends\ModelAnalyzer.cs:410 
  at Unity.Barracuda.ModelAnalyzer.TryGetOutputTensorShape (Unity.Barracuda.Model model, System.Collections.Generic.IDictionary`2[TKey,TValue] inputShapes, System.String output, Unity.Barracuda.TensorShape& shape) [0x00007] in C:\Users\alexi\Dev\MyProject\Library\PackageCache\com.unity.barracuda@1.4.0-preview\Barracuda\Runtime\Core\Backends\ModelAnalyzer.cs:586 
  at Unity.Barracuda.Editor.NNModelEditor+<>c__DisplayClass33_0.<OnEnable>b__2 (System.String i) [0x00006] in C:\Users\alexi\Dev\MyProject\Library\PackageCache\com.unity.barracuda@1.4.0-preview\Barracuda\Editor\ONNXModelImporterEditor.cs:209 
UnityEngine.Debug:LogError (object)
Unity.Barracuda.D:LogError (object) (at Library/PackageCache/com.unity.barracuda@1.4.0-preview/Barracuda/Runtime/Core/Internals/Debug.cs:131)
Unity.Barracuda.Editor.NNModelEditor/<>c__DisplayClass33_0:<OnEnable>b__2 (string) (at Library/PackageCache/com.unity.barracuda@1.4.0-preview/Barracuda/Editor/ONNXModelImporterEditor.cs:214)
System.Linq.Enumerable:ToList<string> (System.Collections.Generic.IEnumerable`1<string>)
Unity.Barracuda.Editor.NNModelEditor:OnEnable () (at Library/PackageCache/com.unity.barracuda@1.4.0-preview/Barracuda/Editor/ONNXModelImporterEditor.cs:204)

Is it related? or is it a separate issue?

This is a different issue, please create another issue for the above error.

surfnerd commented 3 years ago

~A fix for the original issue has been merged into main from #5345. Could you please give it a try to see if it fixes your original problem? Thanks.~

surfnerd commented 3 years ago

My apologies, the issue wasn't fixed, but you will not receive a better error message about the dimensions of your grid.

TheTrope commented 3 years ago

@surfnerd So what's fixed now is the division/zero error handling right ?

However the cause of this error (the grid dimensions < 20) is still there, what should we do about it, start a new issue? Thanks

surfnerd commented 3 years ago

Yes, please create a new issue.

github-actions[bot] commented 3 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.