mlagents_envs.exception.UnityEnvironmentException: Environment shut down with return code -6 (SIGABRT).

[Pimool]

Hi, I am trying to run Reinforcement Learning on a GPU runbox.

With your code, I could train the model on Colab, and Saturn Cloud which is similar to colab.

However, when I tried to run on my personal GPU runbox, it occured an error.

mlagents-learn -h showed the options, so I thought it is a problem with environment.

How can I handle this error?

~$ mlagents-learn config.yaml --run-id=test --env=ML-Agents-with-Google-Colab/headless_build/3DBall_example/3dball.x86_64

Version information: ml-agents: 0.31.0.dev0, ml-agents-envs: 0.31.0.dev0, Communicator API: 1.5.0, PyTorch: 1.11.0+cu102 [INFO] Learning was interrupted. Please wait while the graph is generated. Traceback (most recent call last): File "/home/desktop/venv/bin/mlagents-learn", line 33, in sys.exit(load_entry_point('mlagents', 'console_scripts', 'mlagents-learn')()) File "/home/desktop/ml-agents/ml-agents/mlagents/trainers/learn.py", line 264, in main run_cli(parse_command_line()) File "/home/desktop/ml-agents/ml-agents/mlagents/trainers/learn.py", line 260, in run_cli run_training(run_seed, options, num_areas) File "/home/desktop/ml-agents/ml-agents/mlagents/trainers/learn.py", line 136, in run_training tc.start_learning(env_manager) File "/home/desktop/ml-agents/ml-agents-envs/mlagents_envs/timers.py", line 305, in wrapped return func(*args, *kwargs) File "/home/desktop/ml-agents/ml-agents/mlagents/trainers/trainer_controller.py", line 197, in start_learning raise ex File "/home/desktop/ml-agents/ml-agents/mlagents/trainers/trainer_controller.py", line 172, in start_learning self._reset_env(env_manager) File "/home/desktop/ml-agents/ml-agents-envs/mlagents_envs/timers.py", line 305, in wrapped return func(args, **kwargs) File "/home/desktop/ml-agents/ml-agents/mlagents/trainers/trainer_controller.py", line 105, in _reset_env env_manager.reset(config=new_config) File "/home/desktop/ml-agents/ml-agents/mlagents/trainers/env_manager.py", line 68, in reset self.first_step_infos = self._reset_env(config) File "/home/desktop/ml-agents/ml-agents/mlagents/trainers/subprocess_env_manager.py", line 446, in _reset_env ew.previous_step = EnvironmentStep(ew.recv().payload, ew.worker_id, {}, {}) File "/home/desktop/ml-agents/ml-agents/mlagents/trainers/subprocess_env_manager.py", line 101, in recv raise env_exception mlagents_envs.exception.UnityEnvironmentException: Environment shut down with return code -6 (SIGABRT).

[dhyeythumar]

Hi @Pimool, Check the ml-agents version (the environment given in this repo was built for release_1). Also, it seems your training exited with a critical error suggested by the SIGABRT error code.

[Pimool]

Hi, @dhyeythumar I used ml-agents release 20(the recent one). However, it worked in colab and Saturn cloud with your environment on release 20.. So, I don't think it's a problem with release.. I don't know why the SIGABRT error occurs only on my personal GPU server.

[dhyeythumar]

Then most probably the environment is exiting with an error, it's possible that the Linux executable is not supported on GPU. If I remember correctly on colab this env works on the CPU instance itself haven't tried it on GPU (try this and see if the GPU instance on colab works or not).

[Pimool]

It works in colab on T4 GPU. Also, Saturn Cloud was on GPU, too. The below is the colab notebook. https://colab.research.google.com/drive/1sFY_V-uirL9pCPBlHkme8zBMfp3e1cJQ?usp=sharing

[Pimool]

Below is the Player-0.log file when I try start training with the code above. I have no idea about the errors, and why the handler cannot load such files. Any help is greatly appreciated

''' Mono path[0] = '/home/desktop/ML-Agents-with-Google-Colab/headless_build/3DBall_example/3dball_Data/Managed' Mono config path = '/home/desktop/ML-Agents-with-Google-Colab/headless_build/3DBall_example/3dball_Data/MonoBleedingEdge/etc' Preloaded 'lib_burst_generated.so' Preloaded 'libgrpc_csharp_ext.x64.so' Initialize engine version: 2019.3.15f1 (59ff3e03856d) [Subsystems] Discovering subsystems at path /home/desktop/ML-Agents-with-Google-Colab/headless_build/3DBall_example/3dball_Data/UnitySubsystems Forcing GfxDevice: Null GfxDevice: creating device client; threaded=0 NullGfxDevice: Version: NULL 1.0 [1.0] Renderer: Null Device Vendor: Unity Technologies Begin MonoManager ReloadAssembly Completed reload, in 0.142 seconds WARNING: Shader Unsupported: 'Autodesk Interactive' - All passes removed WARNING: Shader Did you use #pragma only_renderers and omit this platform? UnloadTime: 1.141076 ms Fallback handler could not load library /home/desktop/ML-Agents-with-Google-Colab/headless_build/3DBall_example/3dball_Data/Mono/libcoreclr.so Fallback handler could not load library /home/desktop/ML-Agents-with-Google-Colab/headless_build/3DBall_example/3dball_Data/Mono/libcoreclr.so Fallback handler could not load library /home/desktop/ML-Agents-with-Google-Colab/headless_build/3DBall_example/3dball_Data/Mono/libcoreclr.so Fallback handler could not load library /home/desktop/ML-Agents-with-Google-Colab/headless_build/3DBall_example/3dball_Data/Mono/libdl.so Fallback handler could not load library /home/desktop/ML-Agents-with-Google-Colab/headless_build/3DBall_example/3dball_Data/Mono/libdl.so Fallback handler could not load library /home/desktop/ML-Agents-with-Google-Colab/headless_build/3DBall_example/3dball_Data/Mono/libdl.so Fallback handler could not load library /home/desktop/ML-Agents-with-Google-Colab/headless_build/3DBall_example/3dball_Data/Mono/libSystem.dylib Fallback handler could not load library /home/desktop/ML-Agents-with-Google-Colab/headless_build/3DBall_example/3dball_Data/Mono/libSystem.dylib.so Fallback handler could not load library /home/desktop/ML-Agents-with-Google-Colab/headless_build/3DBall_example/3dball_Data/Mono/libSystem.dylib Fallback handler could not load library /home/desktop/ML-Agents-with-Google-Colab/headless_build/3DBall_example/3dball_Data/Mono/libcoreclr.so Fallback handler could not load library /home/desktop/ML-Agents-with-Google-Colab/headless_build/3DBall_example/3dball_Data/Mono/libcoreclr.so Fallback handler could not load library /home/desktop/ML-Agents-with-Google-Colab/headless_build/3DBall_example/3dball_Data/Mono/libcoreclr.so Fallback handler could not load library /home/desktop/ML-Agents-with-Google-Colab/headless_build/3DBall_example/3dball_Data/Mono/libdl.so Fallback handler could not load library /home/desktop/ML-Agents-with-Google-Colab/headless_build/3DBall_example/3dball_Data/Mono/libdl.so Fallback handler could not load library /home/desktop/ML-Agents-with-Google-Colab/headless_build/3DBall_example/3dball_Data/Mono/libdl.so Fallback handler could not load library /home/desktop/ML-Agents-with-Google-Colab/headless_build/3DBall_example/3dball_Data/Mono/libSystem.dylib Fallback handler could not load library /home/desktop/ML-Agents-with-Google-Colab/headless_build/3DBall_example/3dball_Data/Mono/libSystem.dylib.so Fallback handler could not load library /home/desktop/ML-Agents-with-Google-Colab/headless_build/3DBall_example/3dball_Data/Mono/libSystem.dylib Caught fatal signal - signo:11 code:1 errno:0 addr:0x561114101530 Obtained 4 stack frames. 0 0x007f8bd7a1a520 in __sigaction 1 0x007f8bd66696b5 in grpc_completion_queue_create_internal(grpc_cq_completion_type, grpc_cq_polling_type) 2 0x007f8bd666abf0 in grpc_completion_queue_create_for_next 3 0x000000405aa870 in (wrapper managed-to-native) object:wrapper_native_0x7f8bd6657df0 () '''

[dhyeythumar]

Hi @Pimool , try this command !mlagents-learn config.yaml --run-id=$run_id --env=$env_name --no-graphics

I guess on your server it's trying to render the environment.

[Pimool]

Hi, @dhyeythumar

Thanks for your advice, but It raises same error.

[OmarVector]

Yeah, we have the same problem and we couldnt find any solution