Very low speed when running projects/habitat_ovmm/eval_baselines_agent.py

[RanDing2000]

Hi, in my machine, I use a P5000 with 16384MiB. I run the code of ovmm projects/habitat_ovmm/eval_baselines_agent.py, but is very slow. It takes 30 min per episode. Is it normal? Thanks.

[chrisalbertson]

You are sure it is running on the GPU?

On Aug 7, 2024, at 5:26 AM, Ran Ding @.***> wrote:

Hi, in my machine, I use a P5000 with 16384MiB. I run the code of ovmm projects/habitat_ovmm/eval_baselines_agent.py, but is very slow. It takes 10 min per episode. Is it normal? Thanks.

— Reply to this email directly, view it on GitHub https://github.com/facebookresearch/home-robot/issues/513, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQKNRSCQGFBKUGJXO6DOSTZQIG65AVCNFSM6AAAAABMEJVFYSVHI2DSMVQWIX3LMV43ASLTON2WKOZSGQ2TGMZVHA2DIMA. You are receiving this because you are subscribed to this thread.

[RanDing2000]

Yes, cuda is available, there are some additional warnings when I run this code, is it normal?

pybullet build time: Nov 28 2023 23:52:03 An exception occurred in telemetry logging.Disabling telemetry to prevent further exceptions. Traceback (most recent call last): File "/usr/stud/dira/miniconda3/envs/home-robot/lib/python3.9/site-packages/iopath/common/file_io.py", line 946, in __log_tmetry_keys handler.log_event() File "/usr/stud/dira/miniconda3/envs/home-robot/lib/python3.9/site-packages/iopath/common/event_logger.py", line 97, in log_event del self._evt AttributeError: _evt Loading pretrained CLIP /usr/stud/dira/miniconda3/envs/home-robot/lib/python3.9/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1670525539683/work/aten/src/ATen/native/TensorShape.cpp:3190.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] 2024-08-07 19:58:39,056 Initializing dataset OVMMDataset-v0 2024-08-07 19:58:39,591 initializing sim OVMMSim-v0 Renderer: Quadro P5000/PCIe/SSE2 by NVIDIA Corporation OpenGL version: 4.6.0 NVIDIA 535.183.01 Using optional features: GL_ARB_vertex_array_object GL_ARB_separate_shader_objects GL_ARB_robustness GL_ARB_texture_storage GL_ARB_texture_view GL_ARB_framebuffer_no_attachments GL_ARB_invalidate_subdata GL_ARB_texture_storage_multisample GL_ARB_multi_bind GL_ARB_direct_state_access GL_ARB_get_texture_sub_image GL_ARB_texture_filter_anisotropic GL_KHR_debug GL_KHR_parallel_shader_compile GL_NV_depth_bufferfloat Using driver workarounds: no-forward-compatible-core-context nv-egl-incorrect-gl11-function-pointers no-layout-qualifiers-on-old-glsl nv-zero-context-profile-mask nv-implementation-color-read-format-dsa-broken nv-cubemap-inconsistent-compressed-image-size nv-cubemap-broken-full-compressed-image-query nv-compressed-block-size-in-bits [19:58:46:474598]:[Warning]:[Sim] Simulator.cpp(508)::instanceStageForSceneAttributes : The active scene does not contain semantic annotations : activeSemanticSceneID = 0 2024-08-07 19:58:48,281 Initializing task OVMMNavToObjTask-v0 /usr/stud/dira/miniconda3/envs/home-robot/lib/python3.9/site-packages/gym/spaces/box.py:84: UserWarning: WARN: Box bound precision lowered by casting to float32 logger.warn(f"Box bound precision lowered by casting to {self.dtype}") 0%| | 0/1199 [00:00<?, ?it/s]Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black [19:58:49:209893]:[Warning]:[Assets] ResourceManager.cpp(929)::createRenderAssetInstance : Instantiating render asset data/robots/hab_stretch/urdf/../meshes/link_gripper_fingertip_right.obj with incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file. Trade::AssimpImporter::material(): white ambient detected, forcing back to black [19:58:49:213567]:[Warning]:[Assets] ResourceManager.cpp(929)::createRenderAssetInstance : Instantiating render asset data/robots/hab_stretch/urdf/../meshes/link_gripper_fingertip_left.obj with incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file. Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black [19:58:49:697287]:[Warning]:[Assets] ResourceManager.cpp(929)::createRenderAssetInstance : Instantiating render asset data/robots/hab_stretch/urdf/../meshes/link_aruco_top_wrist.STL with incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file. Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::BasisImporter::image2D(): Y-flipping a compressed image that's not whole blocks, the result will be shifted by 2 pixels Trade::BasisImporter::image2D(): Y-flipping a compressed image that's not whole blocks, the result will be shifted by 3 pixels Trade::BasisImporter::image2D(): Y-flipping a compressed image that's not whole blocks, the result will be shifted by 2 pixels Trade::BasisImporter::image2D(): Y-flipping a compressed image that's not whole blocks, the result will be shifted by 3 pixels Trade::BasisImporter::image2D(): Y-flipping a compressed image that's not whole blocks, the result will be shifted by 2 pixels Trade::BasisImporter::image2D(): Y-flipping a compressed image that's not whole blocks, the result will be shifted by 3 pixels Trade::BasisImporter::image2D(): Y-flipping a compressed image that's not whole blocks, the result will be shifted by 2 pixels Trade::BasisImporter::image2D(): Y-flipping a compressed image that's not whole blocks, the result will be shifted by 3 pixels Trade::BasisImporter::image2D(): Y-flipping a compressed image that's not whole blocks, the result will be shifted by 2 pixels Trade::BasisImporter::image2D(): Y-flipping a compressed image that's not whole blocks, the result will be shifted by 3 pixels Trade::GltfImporter::material(): property fromMI is an object, skipping Trade::GltfImporter::material(): property fromMI is an object, skipping Trade::GltfImporter::material(): property fromMI is an object, skipping Loading pretrained CLIP Renderer: Quadro P5000/PCIe/SSE2 by NVIDIA Corporation OpenGL version: 4.6.0 NVIDIA 535.183.01 Using optional features: GL_ARB_vertex_array_object GL_ARB_separate_shader_objects GL_ARB_robustness GL_ARB_texture_storage GL_ARB_texture_view GL_ARB_framebuffer_no_attachments GL_ARB_invalidate_subdata GL_ARB_texture_storage_multisample GL_ARB_multi_bind GL_ARB_direct_state_access GL_ARB_get_texture_sub_image GL_ARB_texture_filter_anisotropic GL_KHR_debug GL_KHR_parallel_shader_compile GL_NV_depth_buffer_float Using driver workarounds: nv-egl-incorrect-gl11-function-pointers no-layout-qualifiers-on-old-glsl nv-zero-context-profile-mask nv-implementation-color-read-format-dsa-broken nv-cubemap-inconsistent-compressed-image-size nv-cubemap-broken-full-compressed-image-query nv-compressed-block-size-in-bits

[chrisalbertson]

On Aug 7, 2024, at 11:00 AM, Ran Ding @.***> wrote:

Yes, cuda is available, there are some additional warnings when I run this code, is it normal?

Yes, I see the p5000. But I don’t see any evidence of PyTorch using Coda. All of the text below seems to be from OpenGL. I am not saying what is happening, only that I don’t see evidence of it. I could be using the GPU, but we can’t know from the text below.

It is the nature of debugging that something you think is true is not. Finding it is hard and it is best to prove every assumption to yourself.

The simplest thing to do is put some print statements in the code. Some along the line of this maybe.

print(torch.cuda.current_device(), torch.cuda.device_count(), torch.cuda.get_device_name(0))

BNut really you should loop through all devices to be 100% correct and general

pybullet build time: Nov 28 2023 23:52:03 An exception occurred in telemetry logging.Disabling telemetry to prevent further exceptions. Traceback (most recent call last): File "/usr/stud/dira/miniconda3/envs/home-robot/lib/python3.9/site-packages/iopath/common/file_io.py", line 946, in __log_tmetry_keys handler.log_event() File "/usr/stud/dira/miniconda3/envs/home-robot/lib/python3.9/site-packages/iopath/common/event_logger.py", line 97, in log_event del self._evt AttributeError: _evt Loading pretrained CLIP /usr/stud/dira/miniconda3/envs/home-robot/lib/python3.9/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1670525539683/work/aten/src/ATen/native/TensorShape.cpp:3190.) return VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] 2024-08-07 19:58:39,056 Initializing dataset OVMMDataset-v0 2024-08-07 19:58:39,591 initializing sim OVMMSim-v0 Renderer: Quadro P5000/PCIe/SSE2 by NVIDIA Corporation OpenGL version: 4.6.0 NVIDIA 535.183.01 Using optional features: GL_ARB_vertex_array_object GL_ARB_separate_shader_objects GL_ARB_robustness GL_ARB_texture_storage GL_ARB_texture_view GL_ARB_framebuffer_no_attachments GL_ARB_invalidate_subdata GL_ARB_texture_storage_multisample GL_ARB_multi_bind GL_ARB_direct_state_access GL_ARB_get_texture_sub_image GL_ARB_texture_filter_anisotropic GL_KHR_debug GL_KHR_parallel_shader_compile GL_NV_depth_buffer_float Using driver workarounds: no-forward-compatible-core-context nv-egl-incorrect-gl11-function-pointers no-layout-qualifiers-on-old-glsl nv-zero-context-profile-mask nv-implementation-color-read-format-dsa-broken nv-cubemap-inconsistent-compressed-image-size nv-cubemap-broken-full-compressed-image-query nv-compressed-block-size-in-bits [19:58:46:474598]:[Warning]:[Sim] Simulator.cpp(508)::instanceStageForSceneAttributes : The active scene does not contain semantic annotations : activeSemanticSceneID = 0 2024-08-07 19:58:48,281 Initializing task OVMMNavToObjTask-v0 /usr/stud/dira/miniconda3/envs/home-robot/lib/python3.9/site-packages/gym/spaces/box.py:84: UserWarning: WARN: Box bound precision lowered by casting to float32 logger.warn(f"Box bound precision lowered by casting to {self.dtype}") 0%| | 0/1199 [00:00<?, ?it/s]Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black [19:58:49:209893]:[Warning]:[Assets] ResourceManager.cpp(929)::createRenderAssetInstance : Instantiating render asset data/robots/hab_stretch/urdf/../meshes/link_gripper_fingertip_right.obj with incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file. Trade::AssimpImporter::material(): white ambient detected, forcing back to black [19:58:49:213567]:[Warning]:[Assets] ResourceManager.cpp(929)::createRenderAssetInstance : Instantiating render asset data/robots/hab_stretch/urdf/../meshes/link_gripper_fingertip_left.obj with incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file. Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black [19:58:49:697287]:[Warning]:[Assets] ResourceManager.cpp(929)::createRenderAssetInstance : Instantiating render asset data/robots/hab_stretch/urdf/../meshes/link_aruco_top_wrist.STL with incompatible light setup, instance will not be correctly lit. For objects, please ensure 'requires lighting' is enabled in object config file. Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::AssimpImporter::material(): white ambient detected, forcing back to black Trade::BasisImporter::image2D(): Y-flipping a compressed image that's not whole blocks, the result will be shifted by 2 pixels Trade::BasisImporter::image2D(): Y-flipping a compressed image that's not whole blocks, the result will be shifted by 3 pixels Trade::BasisImporter::image2D(): Y-flipping a compressed image that's not whole blocks, the result will be shifted by 2 pixels Trade::BasisImporter::image2D(): Y-flipping a compressed image that's not whole blocks, the result will be shifted by 3 pixels Trade::BasisImporter::image2D(): Y-flipping a compressed image that's not whole blocks, the result will be shifted by 2 pixels Trade::BasisImporter::image2D(): Y-flipping a compressed image that's not whole blocks, the result will be shifted by 3 pixels Trade::BasisImporter::image2D(): Y-flipping a compressed image that's not whole blocks, the result will be shifted by 2 pixels Trade::BasisImporter::image2D(): Y-flipping a compressed image that's not whole blocks, the result will be shifted by 3 pixels Trade::BasisImporter::image2D(): Y-flipping a compressed image that's not whole blocks, the result will be shifted by 2 pixels Trade::BasisImporter::image2D(): Y-flipping a compressed image that's not whole blocks, the result will be shifted by 3 pixels Trade::GltfImporter::material(): property fromMI is an object, skipping Trade::GltfImporter::material(): property fromMI is an object, skipping Trade::GltfImporter::material(): property fromMI is an object, skipping Loading pretrained CLIP Renderer: Quadro P5000/PCIe/SSE2 by NVIDIA Corporation OpenGL version: 4.6.0 NVIDIA 535.183.01 Using optional features: GL_ARB_vertex_array_object GL_ARB_separate_shader_objects GL_ARB_robustness GL_ARB_texture_storage GL_ARB_texture_view GL_ARB_framebuffer_no_attachments GL_ARB_invalidate_subdata GL_ARB_texture_storage_multisample GL_ARB_multi_bind GL_ARB_direct_state_access GL_ARB_get_texture_sub_image GL_ARB_texture_filter_anisotropic GL_KHR_debug GL_KHR_parallel_shader_compile GL_NV_depth_buffer_float Using driver workarounds: nv-egl-incorrect-gl11-function-pointers no-layout-qualifiers-on-old-glsl nv-zero-context-profile-mask nv-implementation-color-read-format-dsa-broken nv-cubemap-inconsistent-compressed-image-size nv-cubemap-broken-full-compressed-image-query nv-compressed-block-size-in-bits

— Reply to this email directly, view it on GitHub https://github.com/facebookresearch/home-robot/issues/513#issuecomment-2274029548, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQKNRTI2WZWCIPXPAPFJWTZQJOD7AVCNFSM6AAAAABMEJVFYSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZUGAZDSNJUHA. You are receiving this because you commented.

[RanDing2000]

Thanks Chris for very detailed answer. It seems that I have already used cuda, but still very very slow.

print(torch.cuda.current_device(), torch.cuda.device_count(), torch.cuda.get_device_name(0)) 0 2 Quadro P5000

[RanDing2000]

Hi Chris, do you know what is the normal speed of running the eval baseline per episode? Thanks!