search

threedworld-mit / tdw

ThreeDWorld simulation environment
BSD 2-Clause "Simplified" License
501 stars 75 forks source link

Install TDW on a remote Linux server problem: cannot communicate between 2 shell #510

Open LuZeking opened 1 year ago

LuZeking commented 1 year ago

Hi,

Thanks for your information before, this time I test in a remote HPC with X11 Servers following This document explains how to run TDW on a virtual display, which still requires X.. Where I test the way "Install TDW on a remote Linux server".

But the command DISPLAY=:0.0 ./TDW.x86_64 -port=1071 just stuck there and doesn't have any response. So I command echo $DISPLAY to check my display number is 113.0 so I do DISPLAY=:113.0 ./TDW.x86_64 -port=1071 again, and then the TDW program launched on my windows laptop. But the problem is: it closed quickly and was not connected to another shell I run my_controller.py (the display number is 103.0 for this shell), which is just stuck there and prints nothing still.

Do you have any idea about this? Thanks:)

alters-mit commented 1 year ago

You need to set launch_build=False in the Controller constructor: https://github.com/threedworld-mit/tdw/blob/master/Documentation/lessons/core_concepts/launch_build.md

Let me know if that works.

LuZeking commented 1 year ago

You need to set launch_build=False in the Controller constructor: https://github.com/threedworld-mit/tdw/blob/master/Documentation/lessons/core_concepts/launch_build.md

Let me know if that works.

Thanks for your advice, I did set it this way, but it doesn’t work, unfortunately. For more details, I used a remote HPC with already installed X11 servers, the HPC manages jobs and resources with the SLURM system, so I have to request an interactive node to get 4 GPUs, otherwise, I only get a CPU.

So what I did is firstly run the my_controller.py in a shell only with CPU and wait for the communication with the following second shell, the second shell is an interactive node with 4 GPUs, so I launched the" DISPLAY=: $DISPLAY number ./TDW.x86_64 -port=1071" there to build the rendering.

The problem is that the first shell printed nothing and got stuck there after running my_controller.py, and in the second shell, the TDW window quickly launched and disappeared. BTW, if I run the TDW.x86_64 in the CPU node shell, and run my_controller.py in the GPU node shell, it is a similar situation.

These two shells have random distinct DISPLAY numbers generated by HPC every time, should I manually set them to the same?

And it seems that no communication is established between the controller and build in the first and second shells, may I ask if you have any idea?

alters-mit commented 1 year ago

@LuZeking You need to set the display value in the shell that is running TDW.x86_64

Are the shells on two different machines? If so, you need to set the expected IP address:

https://github.com/threedworld-mit/tdw/blob/master/Documentation/lessons/setup/install.md#install-tdw-on-a-personal-computer-and-a-remote-linux-server

LuZeking commented 1 year ago

Hi @alters-mit, thanks for your suggestions, but unfortunately it still not works by following the above instructions (e.g. "export DIPLAY=:0.0").

To better illustrate the problem, as shown in the following figure, I use a VNC server in remote HPC: Here we have three shells, the top left is the shell for the my_controller.py, the top right is the shell for build simulation, and the bottom left one is just the content of my_controller.py example. They all run on the remote HPC. And when we run the controller in the top left first, then run the build in the top right, we can find the controller and the build is not successfully connected.

image

alters-mit commented 1 year ago

@LuZeking Please send your xorg config file and the player log.

LuZeking commented 1 year ago

Thanks for your reply. The context of /etc/X11/xorg-4gpus.conf and player log are shown as below

xorg-4gpus.conf `# nvidia-xconfig: X configuration file generated by nvidia-xconfig

nvidia-xconfig: version 1.0 (buildmeister@builder58) Fri Apr 17 00:40:10 PDT 2009

Section "DRI" Mode 0660 Group "vglusers" EndSection

Section "InputDevice"

# generated from default
Identifier     "Mouse0"
Driver         "mouse"
Option         "Protocol" "auto"
Option         "Device" "/dev/input/mice"
Option         "Emulate3Buttons" "no"
Option         "ZAxisMapping" "4 5"

EndSection

Section "InputDevice" Identifier "Keyboard0" Driver "kbd" Option "XkbModel" "pc105" Option "XkbLayout" "gb" EndSection

Section "Monitor" Identifier "Monitor0" VendorName "Unknown" ModelName "Unknown" HorizSync 28.0 - 80.0 VertRefresh 43.0 - 75.0 Option "DPMS" EndSection

Section "Device" Identifier "NV Tesla 1" VendorName "nvidia" Driver "nvidia"

update this with the PCI id of your card. Consult the output

# of the 'lspci' command.  The BusID is usually optional when
# only using one graphics card.
BusID       "PCI:4:0:0"

Option "UseDisplayDevice" "none"

Option "HorizSync"   "CRT-0: 50-110;  DFP-0: 40-80"
Option "VertRefresh" "CRT-0: 60-120;  DFP-0: 75"

EndSection

Section "Device" Identifier "NV Tesla 2" VendorName "nvidia" Driver "nvidia"

update this with the PCI id of your card. Consult the output

# of the 'lspci' command.  The BusID is usually optional when
# only using one graphics card.
BusID       "PCI:5:0:0"

Option "UseDisplayDevice" "none"

Option "HorizSync"   "CRT-0: 50-110;  DFP-0: 40-80"
Option "VertRefresh" "CRT-0: 60-120;  DFP-0: 75"

EndSection

Section "Device" Identifier "NV Tesla 3" VendorName "nvidia" Driver "nvidia"

update this with the PCI id of your card. Consult the output

# of the 'lspci' command.  The BusID is usually optional when
# only using one graphics card.
BusID       "PCI:6:0:0"

Option "UseDisplayDevice" "none"

Option "HorizSync"   "CRT-0: 50-110;  DFP-0: 40-80"
Option "VertRefresh" "CRT-0: 60-120;  DFP-0: 75"

EndSection

Section "Device" Identifier "NV Tesla 4" VendorName "nvidia" Driver "nvidia"

update this with the PCI id of your card. Consult the output

# of the 'lspci' command.  The BusID is usually optional when
# only using one graphics card.
BusID       "PCI:7:0:0"

Option "UseDisplayDevice" "none"

Option "HorizSync"   "CRT-0: 50-110;  DFP-0: 40-80"
Option "VertRefresh" "CRT-0: 60-120;  DFP-0: 75"

EndSection

Section "Screen" Identifier "Screen0" Device "NV Tesla 1" DefaultDepth 24 SubSection "Display" Viewport 0 0 Depth 24 Modes "1280x1024" "1024x768" "800x600" "640x480" EndSubSection Option "UseDisplayDevice" "none" EndSection

Section "Screen" Identifier "Screen1" Device "NV Tesla 2" DefaultDepth 24 SubSection "Display" Viewport 0 0 Depth 24 Modes "1280x1024" "1024x768" "800x600" "640x480" EndSubSection Option "UseDisplayDevice" "none" EndSection

Section "Screen" Identifier "Screen2" Device "NV Tesla 3" DefaultDepth 24 SubSection "Display" Viewport 0 0 Depth 24 Modes "1280x1024" "1024x768" "800x600" "640x480" EndSubSection Option "UseDisplayDevice" "none" EndSection

Section "Screen" Identifier "Screen3" Device "NV Tesla 4" DefaultDepth 24 SubSection "Display" Viewport 0 0 Depth 24 Modes "1280x1024" "1024x768" "800x600" "640x480" EndSubSection Option "UseDisplayDevice" "none" EndSection

Section "ServerLayout" Identifier "Default Layout" Screen 0 "Screen0" 0 0 Screen 1 "Screen1" RightOf "Screen0" Screen 2 "Screen2" RightOf "Screen1" Screen 3 "Screen3" RightOf "Screen2" InputDevice "Mouse0" "CorePointer" InputDevice "Keyboard0" "CoreKeyboard" Option "Xinerama" "False" EndSection`

And player.log is shown as below: Mono path[0] = '/home/hpczeji1/tdw_build/TDW/TDW_Data/Managed' Mono config path = '/home/hpczeji1/tdw_build/TDW/TDW_Data/MonoBleedingEdge/etc' Preloaded '[libOni.so](http://liboni.so/)' Preloaded '[libaudiopluginresonanceaudio.so](http://libaudiopluginresonanceaudio.so/)' Unable to preload the following plugins: [libflexUtils.so](http://libflexutils.so/) Display 0 'default 42"': 5760x1728 (primary device). Unable to load player prefs Desktop is 5760 x 1728 @ 0 Hz Unable to find a supported OpenGL core profile Failed to create valid graphics context: please ensure you meet the minimum requirements E.g. OpenGL core profile 3.2 or later for OpenGL Core renderer No supported renderers found, exiting

LuZeking commented 1 year ago

Sorry, the Player.log show above was on CPU, if it turns to GPU, it would like this:

`Mono path[0] = '/home/hpczeji1/tdw_build/TDW/TDW_Data/Managed' Mono config path = '/home/hpczeji1/tdw_build/TDW/TDW_Data/MonoBleedingEdge/etc' Preloaded 'libOni.so' Preloaded 'libaudiopluginresonanceaudio.so' Unable to preload the following plugins: libflexUtils.so Display 0 'default 45"': 6080x2160 (primary device). Unable to load player prefs Desktop is 6080 x 2160 @ 0 Hz Initialize engine version: 2020.3.24f1 (79c78de19888) Plugins: Couldn't open OculusXRPlugin, error: OculusXRPlugin: cannot open shared object file: No such file or directory [Subsystems] Discovering subsystems at path /home/hpczeji1/tdw_build/TDW/TDW_Data/UnitySubsystems [Subsystems] No descriptors matched for examples in UnitySubsystems/OculusXRPlugin/UnitySubsystemsManifest.json. [Subsystems] 1 'inputs' descriptors matched in UnitySubsystems/OculusXRPlugin/UnitySubsystemsManifest.json [Subsystems] 1 'displays' descriptors matched in UnitySubsystems/OculusXRPlugin/UnitySubsystemsManifest.json [Subsystems] No descriptors matched for meshings in UnitySubsystems/OculusXRPlugin/UnitySubsystemsManifest.json. [Subsystems] No descriptors matched for examples in UnitySubsystems/WindowsMRXRSDK/UnitySubsystemsManifest.json. [Subsystems] 1 'inputs' descriptors matched in UnitySubsystems/WindowsMRXRSDK/UnitySubsystemsManifest.json [Subsystems] 1 'displays' descriptors matched in UnitySubsystems/WindowsMRXRSDK/UnitySubsystemsManifest.json [Subsystems] 1 'meshings' descriptors matched in UnitySubsystems/WindowsMRXRSDK/UnitySubsystemsManifest.json GfxDevice: creating device client; threaded=1 Renderer: llvmpipe (LLVM 13.0.1, 256 bits) Vendor: Mesa/X.org Version: 4.5 (Core Profile) Mesa 21.3.4 GLES: 0 GL_3DFX_texture_compression_FXT1 GL_AMD_conservative_depth GL_AMD_draw_buffers_blend GL_AMD_gpu_shader_int64 GL_AMD_multi_draw_indirect GL_AMD_pinned_memory GL_AMD_query_buffer_object GL_AMD_seamless_cubemap_per_texture GL_AMD_shader_stencil_export GL_AMD_shader_trinary_minmax GL_AMD_texture_texture4 GL_AMD_vertex_shader_layer GL_AMD_vertex_shader_viewport_index GL_ANGLE_texture_compression_dxt3 GL_ANGLE_texture_compression_dxt5 GL_ARB_ES2_compatibility GL_ARB_ES3_1_compatibility GL_ARB_ES3_2_compatibility GL_ARB_ES3_compatibility GL_ARB_arrays_of_arrays GL_ARB_base_instance GL_ARB_blend_func_extended GL_ARB_buffer_storage GL_ARB_clear_buffer_object GL_ARB_clear_texture GL_ARB_clip_control GL_ARB_compressed_texture_pixel_storage GL_ARB_compute_shader GL_ARB_conditional_render_inverted GL_ARB_conservative_depth GL_ARB_copy_buffer GL_ARB_copy_image GL_ARB_cull_distance GL_ARB_debug_output GL_ARB_depth_buffer_float GL_ARB_depth_clamp GL_ARB_derivative_control GL_ARB_direct_state_access GL_ARB_draw_buffers GL_A RB_draw_buffers_blend GL_ARB_draw_elements_base_vertex GL_ARB_draw_indirect GL_ARB_draw_instanced GL_ARB_enhanced_layouts GL_ARB_explicit_attrib_location GL_ARB_explicit_uniform_location GL_ARB_fragment_coord_conventions GL_ARB_fragment_layer_viewport GL_ARB_fragment_shader GL_ARB_framebuffer_no_attachments GL_ARB_framebuffer_object GL_ARB_framebuffer_sRGB GL_ARB_get_program_binary GL_ARB_get_texture_sub_image GL_ARB_gl_spirv GL_ARB_gpu_shader5 GL_ARB_gpu_shader_fp64 GL_ARB_gpu_shader_int64 GL_ARB_half_float_pixel GL_ARB_half_float_vertex GL_ARB_indirect_parameters GL_ARB_instanced_arrays GL_ARB_internalformat_query GL_ARB_internalformat_query2 GL_ARB_invalidate_subdata GL_ARB_map_buffer_alignment GL_ARB_map_buffer_range GL_ARB_multi_bind GL_ARB_multi_draw_indirect GL_ARB_occlusion_query2 GL_ARB_parallel_shader_compile GL_ARB_pipeline_statistics_query GL_ARB_pixel_buffer_object GL_ARB_point_sprite GL_ARB_polygon_offset_clamp GL_ARB_post_depth_coverage GL_ARB_program_interface_query GL_ARB_provoking_vertex GL _ARB_query_buffer_object GL_ARB_robust_buffer_access_behavior GL_ARB_robustness GL_ARB_sample_shading GL_ARB_sampler_objects GL_ARB_seamless_cube_map GL_ARB_seamless_cubemap_per_texture GL_ARB_separate_shader_objects GL_ARB_shader_atomic_counter_ops GL_ARB_shader_atomic_counters GL_ARB_shader_bit_encoding GL_ARB_shader_draw_parameters GL_ARB_shader_group_vote GL_ARB_shader_image_load_store GL_ARB_shader_image_size GL_ARB_shader_objects GL_ARB_shader_precision GL_ARB_shader_stencil_export GL_ARB_shader_storage_buffer_object GL_ARB_shader_subroutine GL_ARB_shader_texture_image_samples GL_ARB_shader_texture_lod GL_ARB_shader_viewport_layer_array GL_ARB_shading_language_420pack GL_ARB_shading_language_include GL_ARB_shading_language_packing GL_ARB_spirv_extensions GL_ARB_stencil_texturing GL_ARB_sync GL_ARB_tessellation_shader GL_ARB_texture_barrier GL_ARB_texture_buffer_object GL_ARB_texture_buffer_object_rgb32 GL_ARB_texture_buffer_range GL_ARB_texture_compression_bptc GL_ARB_texture_compression_rgtc GL_ARB_te xture_cube_map_array GL_ARB_texture_filter_anisotropic GL_ARB_texture_filter_minmax GL_ARB_texture_float GL_ARB_texture_gather GL_ARB_texture_mirror_clamp_to_edge GL_ARB_texture_multisample GL_ARB_texture_non_power_of_two GL_ARB_texture_query_levels GL_ARB_texture_query_lod GL_ARB_texture_rectangle GL_ARB_texture_rg GL_ARB_texture_rgb10_a2ui GL_ARB_texture_stencil8 GL_ARB_texture_storage GL_ARB_texture_storage_multisample GL_ARB_texture_swizzle GL_ARB_texture_view GL_ARB_timer_query GL_ARB_transform_feedback2 GL_ARB_transform_feedback3 GL_ARB_transform_feedback_instanced GL_ARB_transform_feedback_overflow_query GL_ARB_uniform_buffer_object GL_ARB_vertex_array_bgra GL_ARB_vertex_array_object GL_ARB_vertex_attrib_64bit GL_ARB_vertex_attrib_binding GL_ARB_vertex_buffer_object GL_ARB_vertex_shader GL_ARB_vertex_type_10f_11f_11f_rev GL_ARB_vertex_type_2_10_10_10_rev GL_ARB_viewport_array GL_ATI_blend_equation_separate GL_ATI_texture_float GL_ATI_texture_mirror_once GL_EXT_EGL_image_storage GL_EXT_EGL_sync GLEXT abgr GL_EXT_blend_equation_separate GL_EXT_draw_buffers2 GL_EXT_draw_instanced GL_EXT_framebuffer_blit GL_EXT_framebuffer_multisample GL_EXT_framebuffer_multisample_blit_scaled GL_EXT_framebuffer_object GL_EXT_framebuffer_sRGB GL_EXT_memory_object GL_EXT_memory_object_fd GL_EXT_packed_depth_stencil GL_EXT_packed_float GL_EXT_pixel_buffer_object GL_EXT_polygon_offset_clamp GL_EXT_provoking_vertex GL_EXT_shader_framebuffer_fetch_non_coherent GL_EXT_shader_integer_mix GL_EXT_texture_array GL_EXT_texture_compression_dxt1 GL_EXT_texture_compression_rgtc GL_EXT_texture_compression_s3tc GL_EXT_texture_filter_anisotropic GL_EXT_texture_filter_minmax GL_EXT_texture_integer GL_EXT_texture_mirror_clamp GL_EXT_texture_sRGB GL_EXT_texture_sRGB_R8 GL_EXT_texture_sRGB_decode GL_EXT_texture_shadow_lod GL_EXT_texture_shared_exponent GL_EXT_texture_snorm GL_EXT_texture_swizzle GL_EXT_timer_query GL_EXT_transform_feedback GL_EXT_vertex_array_bgra GL_EXT_vertex_attrib_64bit GL_IBM_multimode_draw_arrays GL_KHR_blend_equation_adv anced GL_KHR_context_flush_control GL_KHR_debug GL_KHR_no_error GL_KHR_parallel_shader_compile GL_KHR_robust_buffer_access_behavior GL_KHR_robustness GL_KHR_texture_compression_astc_ldr GL_KHR_texture_compression_astc_sliced_3d GL_MESA_framebuffer_flip_y GL_MESA_pack_invert GL_MESA_shader_integer_functions GL_MESA_texture_signed_rgba GL_MESA_ycbcr_texture GL_NV_conditional_render GL_NV_copy_image GL_NV_depth_clamp GL_NV_packed_depth_stencil GL_NV_texture_barrier GL_OES_EGL_image GL_S3_s3tc OPENGL LOG: Creating OpenGL 4.5 graphics device ; Context level <OpenGL 4.5> ; Context handle 27908272 ALSA lib confmisc.c:855:(parse_card) cannot find card '0' ALSA lib conf.c:5178:(_snd_config_evaluate) function snd_func_card_inum returned error: No such file or directory ALSA lib confmisc.c:422:(snd_func_concat) error evaluating strings ALSA lib conf.c:5178:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory ALSA lib confmisc.c:1334:(snd_func_refer) error evaluating name ALSA lib conf.c:5178:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory ALSA lib conf.c:5701:(snd_config_expand) Evaluate error: No such file or directory ALSA lib pcm.c:2664:(snd_pcm_open_noupdate) Unknown PCM default FMOD failed to initialize the output device.: "Error initializing output device. " (60) Forced to initialize FMOD to to the device driver's system output rate 48000, this may impact performance and/or give inconsistent experiences compared to selected sample rate 48000 ALSA lib confmisc.c:855:(parse_card) cannot find card '0' ALSA lib conf.c:5178:(_snd_config_evaluate) function snd_func_card_inum returned error: No such file or directory ALSA lib confmisc.c:422:(snd_func_concat) error evaluating strings ALSA lib conf.c:5178:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory ALSA lib confmisc.c:1334:(snd_func_refer) error evaluating name ALSA lib conf.c:5178:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory ALSA lib conf.c:5701:(snd_config_expand) Evaluate error: No such file or directory ALSA lib pcm.c:2664:(snd_pcm_open_noupdate) Unknown PCM default FMOD failed to initialize the output device.: "Error initializing output device. " (60) FMOD failed to initialize any audio devices, running on emulated software output with no sound. Please check your audio drivers and/or hardware for malfunction. Begin MonoManager ReloadAssembly

Setting up 64 worker threads for Enlighten. Thread -> id: 7f74021eb700 -> priority: 1 Thread -> id: 7f74019ea700 -> priority: 1 Thread -> id: 7f74011e9700 -> priority: 1 Thread -> id: 7f74009e8700 -> priority: 1 Thread -> id: 7f73d7fff700 -> priority: 1 Thread -> id: 7f73d77fe700 -> priority: 1 Thread -> id: 7f73d6ffd700 -> priority: 1 Thread -> id: 7f73d67fc700 -> priority: 1 Thread -> id: 7f73d5ffb700 -> priority: 1 Thread -> id: 7f73d57fa700 -> priority: 1 Thread -> id: 7f73d4ff9700 -> priority: 1 requesting resize 256 x 256 Thread -> id: 7f738ffff700 -> priority: 1 Thread -> id: 7f738f7fe700 -> priority: 1 Thread -> id: 7f738effd700 -> priority: 1 Thread -> id: 7f738e7fc700 -> priority: 1 Thread -> id: 7f738dffb700 -> priority: 1 Thread -> id: 7f738d7fa700 -> priority: 1 Thread -> id: 7f738cff9700 -> priority: 1 Thread -> id: 7f738c7f8700 -> priority: 1 Thread -> id: 7f738bff7700 -> priority: 1 Thread -> id: 7f738b7f6700 -> priority: 1 Thread -> id: 7f738aff5700 -> priority: 1 Thread -> id: 7f738a7f4700 -> priority: 1 Thread -> id: 7f7389ff3700 -> priority: 1 Thread -> id: 7f73897f2700 -> priority: 1 Thread -> id: 7f7388ff1700 -> priority: 1 Thread -> id: 7f73887f0700 -> priority: 1 Thread -> id: 7f7387fef700 -> priority: 1 Thread -> id: 7f73877ee700 -> priority: 1 Thread -> id: 7f7386fed700 -> priority: 1 Thread -> id: 7f73867ec700 -> priority: 1 Thread -> id: 7f7385feb700 -> priority: 1 Thread -> id: 7f73857ea700 -> priority: 1 Thread -> id: 7f7384fe9700 -> priority: 1 Thread -> id: 7f73847e8700 -> priority: 1 Thread -> id: 7f7383fe7700 -> priority: 1 Thread -> id: 7f73837e6700 -> priority: 1 Thread -> id: 7f7382fe5700 -> priority: 1 Thread -> id: 7f73827e4700 -> priority: 1 Thread -> id: 7f7381fe3700 -> priority: 1 Thread -> id: 7f73817e2700 -> priority: 1 Thread -> id: 7f7380fe1700 -> priority: 1 Thread -> id: 7f73807e0700 -> priority: 1 Thread -> id: 7f737ffdf700 -> priority: 1 Thread -> id: 7f737f7de700 -> priority: 1 Thread -> id: 7f737efdd700 -> priority: 1 Thread -> id: 7f737e7dc700 -> priority: 1 Thread -> id: 7f737dfdb700 -> priority: 1 Thread -> id: 7f737d7da700 -> priority: 1 Thread -> id: 7f737cfd9700 -> priority: 1 Thread -> id: 7f737c7d8700 -> priority: 1 Thread -> id: 7f737bfd7700 -> priority: 1 Thread -> id: 7f737b7d6700 -> priority: 1 Thread -> id: 7f737afd5700 -> priority: 1 Thread -> id: 7f737a7d4700 -> priority: 1 Thread -> id: 7f7379fd3700 -> priority: 1 Thread -> id: 7f73797d2700 -> priority: 1 Thread -> id: 7f7378fd1700 -> priority: 1 Thread -> id: 7f73787d0700 -> priority: 1 Thread -> id: 7f7377fcf700 -> priority: 1 Thread -> id: 7f73777ce700 -> priority: 1 Thread -> id: 7f7376fcd700 -> priority: 1 Thread -> id: 7f73767cc700 -> priority: 1 Thread -> id: 7f7375fcb700 -> priority: 1 Fallback handler could not load library /home/hpczeji1/tdw_build/TDW/TDW_Data/Mono/libKernel32 Fallback handler could not load library /home/hpczeji1/tdw_build/TDW/TDW_Data/Mono/libKernel32.so Fallback handler could not load library /home/hpczeji1/tdw_build/TDW/TDW_Data/Mono/Kernel32 Fallback handler could not load library /home/hpczeji1/tdw_build/TDW/TDW_Data/Mono/libKernel32 Fallback handler could not load library /home/hpczeji1/tdw_build/TDW/TDW_Data/Mono/libKernel32.so Fallback handler could not load library /home/hpczeji1/tdw_build/TDW/TDW_Data/Mono/libKernel32 resizing window to 256 x 256 Desktop is 6080 x 2160 @ 0 Hz `

LuZeking commented 1 year ago

Sorry for the above messy report, thanks for your patience in reading until hear. Now I have a big Update, Interestingly, when I launch these shells with Xserver in WSL2 or VScode, it fails, but it works when I switch to Cygwin. _But I am now met with another problem, the simulation is too slow, and it seems like it didn't recognize the GPU cards, according to "ALSA lib confmisc.c:855:(parsecard) cannot find card '0'".

I used 4 A100 GPU to run a simple ur5 controller, which needs longer than 4.5 mins somehow, may I ask if you know how to solve it? PS. the Xorg-4gpus.conf is controlled by administrators, I am not sure if I could persuade them to change it.

2shells_for_GPU_TDW_simulation_compressed_from_4_5min

The Player log by running the ur5 example controller as shown below: `Mono path[0] = '/home/hpczeji1/tdw_build/TDW/TDW_Data/Managed' Mono config path = '/home/hpczeji1/tdw_build/TDW/TDW_Data/MonoBleedingEdge/etc' Preloaded 'libOni.so' Preloaded 'libaudiopluginresonanceaudio.so' Unable to preload the following plugins: libflexUtils.so Display 0 'default 45"': 4800x1440 (primary device). Desktop is 4800 x 1440 @ 0 Hz Initialize engine version: 2020.3.24f1 (79c78de19888) Plugins: Couldn't open OculusXRPlugin, error: OculusXRPlugin: cannot open shared object file: No such file or directory [Subsystems] Discovering subsystems at path /home/hpczeji1/tdw_build/TDW/TDW_Data/UnitySubsystems [Subsystems] No descriptors matched for examples in UnitySubsystems/OculusXRPlugin/UnitySubsystemsManifest.json. [Subsystems] 1 'inputs' descriptors matched in UnitySubsystems/OculusXRPlugin/UnitySubsystemsManifest.json [Subsystems] 1 'displays' descriptors matched in UnitySubsystems/OculusXRPlugin/UnitySubsystemsManifest.json [Subsystems] No descriptors matched for meshings in UnitySubsystems/OculusXRPlugin/UnitySubsystemsManifest.json. [Subsystems] No descriptors matched for examples in UnitySubsystems/WindowsMRXRSDK/UnitySubsystemsManifest.json. [Subsystems] 1 'inputs' descriptors matched in UnitySubsystems/WindowsMRXRSDK/UnitySubsystemsManifest.json [Subsystems] 1 'displays' descriptors matched in UnitySubsystems/WindowsMRXRSDK/UnitySubsystemsManifest.json [Subsystems] 1 'meshings' descriptors matched in UnitySubsystems/WindowsMRXRSDK/UnitySubsystemsManifest.json GfxDevice: creating device client; threaded=1 Renderer: llvmpipe (LLVM 13.0.1, 256 bits) Vendor: Mesa/X.org Version: 4.5 (Core Profile) Mesa 21.3.4 GLES: 0 GL_3DFX_texture_compression_FXT1 GL_AMD_conservative_depth GL_AMD_draw_buffers_blend GL_AMD_gpu_shader_int64 GL_AMD_multi_draw_indirect GL_AMD_pinned_memory GL_AMD_query_buffer_object GL_AMD_seamless_cubemap_per_texture GL_AMD_shader_stencil_export GL_AMD_shader_trinary_minmax GL_AMD_texture_texture4 GL_AMD_vertex_shader_layer GL_AMD_vertex_shader_viewport_index GL_ANGLE_texture_compression_dxt3 GL_ANGLE_texture_compression_dxt5 GL_ARB_ES2_compatibility GL_ARB_ES3_1_compatibility GL_ARB_ES3_2_compatibility GL_ARB_ES3_compatibility GL_ARB_arrays_of_arrays GL_ARB_base_instance GL_ARB_blend_func_extended GL_ARB_buffer_storage GL_ARB_clear_buffer_object GL_ARB_clear_texture GL_ARB_clip_control GL_ARB_compressed_texture_pixel_storage GL_ARB_compute_shader GL_ARB_conditional_render_inverted GL_ARB_conservative_depth GL_ARB_copy_buffer GL_ARB_copy_image GL_ARB_cull_distance GL_ARB_debug_output GL_ARB_depth_buffer_float GL_ARB_depth_clamp GL_ARB_derivative_control GL_ARB_direct_state_access GL_ARB_draw_buffers GL_A RB_draw_buffers_blend GL_ARB_draw_elements_base_vertex GL_ARB_draw_indirect GL_ARB_draw_instanced GL_ARB_enhanced_layouts GL_ARB_explicit_attrib_location GL_ARB_explicit_uniform_location GL_ARB_fragment_coord_conventions GL_ARB_fragment_layer_viewport GL_ARB_fragment_shader GL_ARB_framebuffer_no_attachments GL_ARB_framebuffer_object GL_ARB_framebuffer_sRGB GL_ARB_get_program_binary GL_ARB_get_texture_sub_image GL_ARB_gl_spirv GL_ARB_gpu_shader5 GL_ARB_gpu_shader_fp64 GL_ARB_gpu_shader_int64 GL_ARB_half_float_pixel GL_ARB_half_float_vertex GL_ARB_indirect_parameters GL_ARB_instanced_arrays GL_ARB_internalformat_query GL_ARB_internalformat_query2 GL_ARB_invalidate_subdata GL_ARB_map_buffer_alignment GL_ARB_map_buffer_range GL_ARB_multi_bind GL_ARB_multi_draw_indirect GL_ARB_occlusion_query2 GL_ARB_parallel_shader_compile GL_ARB_pipeline_statistics_query GL_ARB_pixel_buffer_object GL_ARB_point_sprite GL_ARB_polygon_offset_clamp GL_ARB_post_depth_coverage GL_ARB_program_interface_query GL_ARB_provoking_vertex GL _ARB_query_buffer_object GL_ARB_robust_buffer_access_behavior GL_ARB_robustness GL_ARB_sample_shading GL_ARB_sampler_objects GL_ARB_seamless_cube_map GL_ARB_seamless_cubemap_per_texture GL_ARB_separate_shader_objects GL_ARB_shader_atomic_counter_ops GL_ARB_shader_atomic_counters GL_ARB_shader_bit_encoding GL_ARB_shader_draw_parameters GL_ARB_shader_group_vote GL_ARB_shader_image_load_store GL_ARB_shader_image_size GL_ARB_shader_objects GL_ARB_shader_precision GL_ARB_shader_stencil_export GL_ARB_shader_storage_buffer_object GL_ARB_shader_subroutine GL_ARB_shader_texture_image_samples GL_ARB_shader_texture_lod GL_ARB_shader_viewport_layer_array GL_ARB_shading_language_420pack GL_ARB_shading_language_include GL_ARB_shading_language_packing GL_ARB_spirv_extensions GL_ARB_stencil_texturing GL_ARB_sync GL_ARB_tessellation_shader GL_ARB_texture_barrier GL_ARB_texture_buffer_object GL_ARB_texture_buffer_object_rgb32 GL_ARB_texture_buffer_range GL_ARB_texture_compression_bptc GL_ARB_texture_compression_rgtc GL_ARB_te xture_cube_map_array GL_ARB_texture_filter_anisotropic GL_ARB_texture_filter_minmax GL_ARB_texture_float GL_ARB_texture_gather GL_ARB_texture_mirror_clamp_to_edge GL_ARB_texture_multisample GL_ARB_texture_non_power_of_two GL_ARB_texture_query_levels GL_ARB_texture_query_lod GL_ARB_texture_rectangle GL_ARB_texture_rg GL_ARB_texture_rgb10_a2ui GL_ARB_texture_stencil8 GL_ARB_texture_storage GL_ARB_texture_storage_multisample GL_ARB_texture_swizzle GL_ARB_texture_view GL_ARB_timer_query GL_ARB_transform_feedback2 GL_ARB_transform_feedback3 GL_ARB_transform_feedback_instanced GL_ARB_transform_feedback_overflow_query GL_ARB_uniform_buffer_object GL_ARB_vertex_array_bgra GL_ARB_vertex_array_object GL_ARB_vertex_attrib_64bit GL_ARB_vertex_attrib_binding GL_ARB_vertex_buffer_object GL_ARB_vertex_shader GL_ARB_vertex_type_10f_11f_11f_rev GL_ARB_vertex_type_2_10_10_10_rev GL_ARB_viewport_array GL_ATI_blend_equation_separate GL_ATI_texture_float GL_ATI_texture_mirror_once GL_EXT_EGL_image_storage GL_EXT_EGL_sync GLEXT abgr GL_EXT_blend_equation_separate GL_EXT_draw_buffers2 GL_EXT_draw_instanced GL_EXT_framebuffer_blit GL_EXT_framebuffer_multisample GL_EXT_framebuffer_multisample_blit_scaled GL_EXT_framebuffer_object GL_EXT_framebuffer_sRGB GL_EXT_memory_object GL_EXT_memory_object_fd GL_EXT_packed_depth_stencil GL_EXT_packed_float GL_EXT_pixel_buffer_object GL_EXT_polygon_offset_clamp GL_EXT_provoking_vertex GL_EXT_shader_framebuffer_fetch_non_coherent GL_EXT_shader_integer_mix GL_EXT_texture_array GL_EXT_texture_compression_dxt1 GL_EXT_texture_compression_rgtc GL_EXT_texture_compression_s3tc GL_EXT_texture_filter_anisotropic GL_EXT_texture_filter_minmax GL_EXT_texture_integer GL_EXT_texture_mirror_clamp GL_EXT_texture_sRGB GL_EXT_texture_sRGB_R8 GL_EXT_texture_sRGB_decode GL_EXT_texture_shadow_lod GL_EXT_texture_shared_exponent GL_EXT_texture_snorm GL_EXT_texture_swizzle GL_EXT_timer_query GL_EXT_transform_feedback GL_EXT_vertex_array_bgra GL_EXT_vertex_attrib_64bit GL_IBM_multimode_draw_arrays GL_KHR_blend_equation_adv anced GL_KHR_context_flush_control GL_KHR_debug GL_KHR_no_error GL_KHR_parallel_shader_compile GL_KHR_robust_buffer_access_behavior GL_KHR_robustness GL_KHR_texture_compression_astc_ldr GL_KHR_texture_compression_astc_sliced_3d GL_MESA_framebuffer_flip_y GL_MESA_pack_invert GL_MESA_shader_integer_functions GL_MESA_texture_signed_rgba GL_MESA_ycbcr_texture GL_NV_conditional_render GL_NV_copy_image GL_NV_depth_clamp GL_NV_packed_depth_stencil GL_NV_texture_barrier GL_OES_EGL_image GL_S3_s3tc OPENGL LOG: Creating OpenGL 4.5 graphics device ; Context level <OpenGL 4.5> ; Context handle 6645008 ALSA lib confmisc.c:855:(parse_card) cannot find card '0' ALSA lib conf.c:5178:(_snd_config_evaluate) function snd_func_card_inum returned error: No such file or directory ALSA lib confmisc.c:422:(snd_func_concat) error evaluating strings ALSA lib conf.c:5178:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory ALSA lib confmisc.c:1334:(snd_func_refer) error evaluating name ALSA lib conf.c:5178:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory ALSA lib conf.c:5701:(snd_config_expand) Evaluate error: No such file or directory ALSA lib pcm.c:2664:(snd_pcm_open_noupdate) Unknown PCM default FMOD failed to initialize the output device.: "Error initializing output device. " (60) Forced to initialize FMOD to to the device driver's system output rate 48000, this may impact performance and/or give inconsistent experiences compared to selected sample rate 48000 ALSA lib confmisc.c:855:(parse_card) cannot find card '0' ALSA lib conf.c:5178:(_snd_config_evaluate) function snd_func_card_inum returned error: No such file or directory ALSA lib confmisc.c:422:(snd_func_concat) error evaluating strings ALSA lib conf.c:5178:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory ALSA lib confmisc.c:1334:(snd_func_refer) error evaluating name ALSA lib conf.c:5178:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory ALSA lib conf.c:5701:(snd_config_expand) Evaluate error: No such file or directory ALSA lib pcm.c:2664:(snd_pcm_open_noupdate) Unknown PCM default FMOD failed to initialize the output device.: "Error initializing output device. " (60) FMOD failed to initialize any audio devices, running on emulated software output with no sound. Please check your audio drivers and/or hardware for malfunction. Begin MonoManager ReloadAssembly

Setting up 64 worker threads for Enlighten. Thread -> id: 7f5f0924f700 -> priority: 1 Thread -> id: 7f5f08a4e700 -> priority: 1 Thread -> id: 7f5f03fff700 -> priority: 1 Thread -> id: 7f5f037fe700 -> priority: 1 Thread -> id: 7f5f02ffd700 -> priority: 1 Thread -> id: 7f5f027fc700 -> priority: 1 Thread -> id: 7f5f01ffb700 -> priority: 1 Thread -> id: 7f5f017fa700 -> priority: 1 Thread -> id: 7f5f00ff9700 -> priority: 1 Thread -> id: 7f5ee7fff700 -> priority: 1 Thread -> id: 7f5ee77fe700 -> priority: 1 Thread -> id: 7f5ee6ffd700 -> priority: 1 Thread -> id: 7f5ee67fc700 -> priority: 1 Thread -> id: 7f5ee5ffb700 -> priority: 1 Thread -> id: 7f5ee57fa700 -> priority: 1 Thread -> id: 7f5ee4ff9700 -> priority: 1 requesting resize 256 x 256 Thread -> id: 7f5ee47f8700 -> priority: 1 Thread -> id: 7f5ee3ff7700 -> priority: 1 Thread -> id: 7f5ee37f6700 -> priority: 1 Thread -> id: 7f5ee2ff5700 -> priority: 1 Thread -> id: 7f5ee27f4700 -> priority: 1 Thread -> id: 7f5ee1ff3700 -> priority: 1 Thread -> id: 7f5ee17f2700 -> priority: 1 Thread -> id: 7f5ee0ff1700 -> priority: 1 Thread -> id: 7f5e9ffff700 -> priority: 1 Thread -> id: 7f5e9f7fe700 -> priority: 1 Thread -> id: 7f5e9effd700 -> priority: 1 Thread -> id: 7f5e9e7fc700 -> priority: 1 Thread -> id: 7f5e9dffb700 -> priority: 1 Thread -> id: 7f5e9d7fa700 -> priority: 1 Thread -> id: 7f5e9cff9700 -> priority: 1 Thread -> id: 7f5e9c7f8700 -> priority: 1 Thread -> id: 7f5e9bff7700 -> priority: 1 Thread -> id: 7f5e9b7f6700 -> priority: 1 Thread -> id: 7f5e9aff5700 -> priority: 1 Thread -> id: 7f5e9a7f4700 -> priority: 1 Thread -> id: 7f5e99ff3700 -> priority: 1 Thread -> id: 7f5e997f2700 -> priority: 1 Thread -> id: 7f5e98ff1700 -> priority: 1 Thread -> id: 7f5e987f0700 -> priority: 1 Thread -> id: 7f5e97fef700 -> priority: 1 Thread -> id: 7f5e977ee700 -> priority: 1 Thread -> id: 7f5e96fed700 -> priority: 1 Thread -> id: 7f5e967ec700 -> priority: 1 Thread -> id: 7f5e95feb700 -> priority: 1 Thread -> id: 7f5e957ea700 -> priority: 1 Thread -> id: 7f5e94fe9700 -> priority: 1 Thread -> id: 7f5e947e8700 -> priority: 1 Thread -> id: 7f5e93fe7700 -> priority: 1 Thread -> id: 7f5e937e6700 -> priority: 1 Thread -> id: 7f5e92fe5700 -> priority: 1 Thread -> id: 7f5e927e4700 -> priority: 1 Thread -> id: 7f5e91fe3700 -> priority: 1 Thread -> id: 7f5e917e2700 -> priority: 1 Thread -> id: 7f5e90fe1700 -> priority: 1 Thread -> id: 7f5e907e0700 -> priority: 1 Thread -> id: 7f5e8ffdf700 -> priority: 1 Thread -> id: 7f5e8f7de700 -> priority: 1 Thread -> id: 7f5e8efdd700 -> priority: 1 Thread -> id: 7f5e8e7dc700 -> priority: 1 Thread -> id: 7f5e8dfdb700 -> priority: 1 Thread -> id: 7f5e8d7da700 -> priority: 1 Thread -> id: 7f5e8cfd9700 -> priority: 1 Thread -> id: 7f5e8c7d8700 -> priority: 1 resizing window to 256 x 256 Desktop is 4800 x 1440 @ 0 Hz TDW v1.11.1 Fallback handler could not load library /home/hpczeji1/tdw_build/TDW/TDW_Data/Mono/libKernel32 Fallback handler could not load library /home/hpczeji1/tdw_build/TDW/TDW_Data/Mono/libKernel32.so Fallback handler could not load library /home/hpczeji1/tdw_build/TDW/TDW_Data/Mono/Kernel32 Fallback handler could not load library /home/hpczeji1/tdw_build/TDW/TDW_Data/Mono/libKernel32 Fallback handler could not load library /home/hpczeji1/tdw_build/TDW/TDW_Data/Mono/libKernel32.so Fallback handler could not load library /home/hpczeji1/tdw_build/TDW/TDW_Data/Mono/libKernel32 `

alters-mit commented 1 year ago

Now I have a big Update, Interestingly, when I launch these shells with Xserver in WSL2 or VScode, it fails, but it works when I switch to Cygwin.

If you're using Windows 10, that's to be expected. WSL2 in Windows 10 doesn't support UI applications.

the simulation is too slow, and it seems like it didn't recognize the GPU cards, according to "ALSA lib confmisc.c:855:(parse_card) cannot find card '0'". Is this on Cygwin? If so, I don't think Cygwin can find the GPU. I don't know if you're trying to test the build before running it on a server, but if not, you should be using the Windows build instead.

If you're using a Linux server, the most likely problem is that your DISPLAY variable isn't correct. The reason for the slowdown could be either obsolete GPU drivers, or someone is currently using the GPU, or both.

Please send the output of nvidia-smi

according to "ALSA lib confmisc.c:855:(parse_card) cannot find card '0'". Again, this could be a Cygwin issue. Either way, it's harmless as long as you don't need to use audio.

LuZeking commented 1 year ago

Thanks for your reply! Yes you are right, I ran Cygwin shells on Windows 10 to connect to a remote Linux server, and the builder and controller programs both ran on the remote Linux server.

But after using the Cygwin terminal to connect to a remote Linux server, I apply for an interactive 4-GPU job by Slurm, the build and controllers both ran on the launched X-server Window, and it ran very slowly. So you mean when we use Cygwin to connect to the remote Linux server with X-server, the running programs cannot find the GPU there?

I checked the driver version(510.85.02) and GPU usage (after, only I can use it). And the output of nvidia-smi is shown below: image

alters-mit commented 1 year ago

If you're running the build on a server, it shouldn't matter if you're using Cygwin.

I took a closer look at your Player log and it looks like you're using Mesa drivers. That might not be optimal. If possible replace the drivers with the latest NVIDIA proprietary drivers (if you're using Ubuntu and have the required permissions, there's a ppa and it's pretty easy to do).

meti-94 commented 2 weeks ago

Hello everyone,

I recently came across this issue and I'm wondering if anything has happened in the last two years that would facilitate running TDW on a slurm environment. Are there any reliable resources or best practices available? The institue I am working for, uses apptainer containers in HPC