search

qcr / benchbot

BenchBot is a tool for seamlessly testing & evaluating semantic scene understanding tools in both realistic 3D simulation & on real robots
BSD 3-Clause "New" or "Revised" License
110 stars 12 forks source link

Benchbot Robot Controller Crash #92

Closed tan007 closed 1 year ago

tan007 commented 1 year ago

BenchBot start OK until ...

Supervisor is now available @ 'http://0.0.0.0:10000' ...

Waiting until a robot controller is found @ 'http://benchbot_robot:10000' ... 
    Found
Sending environment data & robot config to controller ... 
    Ready
Starting the robot controller ...

Then, the Python Issac Sim Console gives out this error message

2023-04-04 14:45:57  [Warning] [omni.kit.widget.live.cache_state_menu] Unable to detect Omniverse Cache Server. Consider installing it for better IO performance.

and crashes. Any idea to fix this..

david2611 commented 1 year ago

Hi @tan007. Thanks for lodging the issue. First off, are you using the most recent version of BenchBot and Omniverse? Can you tell me the version number of Omniverse when it appears in the pop up window?

What is it that is crashing? Just Omniverse or the benchbot_run command and the benchbot stack in general?

To try and get a better idea of what's going on, could you uncomment this line in your benchbot_run script and then after the crash has occurred run docker logs benchbot_robot and share the output of that here please?

Note that you will want to recomment the previous line when we are done debugging and will need to run benchbot_run -k to clean up the docker containers otherwise.

tan007 commented 1 year ago

Thanks for your reply. I am using the latest version of Benchbot( git default branch). The pop-up window (Issac sim python 2022.2.1), crashes and totally disappears. There is the yellow warning message " Unable to detect Omniverse Cache Server. Consider installing it for better IO performance. " at the bottom of the window.

The log file output is here , after uncommenting the "sleep infinity" 

Robot controller is now available @ 'http://0.0.0.0:10000' ...
172.20.0.102 - - [2023-04-04 14:45:43] "GET // HTTP/1.1" 200 152 0.000419
172.20.0.102 - - [2023-04-04 14:45:44] "POST //configure HTTP/1.1" 200 137 0.026483
Preparing the requested controller ... Done
172.20.0.102 - - [2023-04-04 14:46:00] "GET //prepare HTTP/1.1" 200 133 15.813365
Starting the requested run in running controller ... [INFO] [1680619580.108798]: FAILURE: timeout exceeded while waiting for message on topic /camera/depth/image_raw
publishing and latching message for 3.0 seconds

COMMAND TO START INSTANCE HAD NON-ZERO RETURN CODE (56)
Done
172.20.0.102 - - [2023-04-04 14:46:25] "GET //start HTTP/1.1" 200 132 25.774749
172.20.0.254 - - [2023-04-04 14:52:37] "GET /stop HTTP/1.1" 200 130 0.000308
172.20.0.254 - - [2023-04-04 15:13:54] "GET /stop HTTP/1.1" 200 130 0.000265
172.20.0.102 - - [2023-04-04 15:13:57] "GET // HTTP/1.1" 200 152 0.000250
[INFO] [1680621258.986191]: FAILURE: timeout exceeded while waiting for message on topic /camera/depth/image_raw

Controller isn't running, so nothing to stop.
172.20.0.102 - - [2023-04-04 15:14:18] "POST //configure HTTP/1.1" 200 137 20.009326
Preparing the requested controller ... Done
172.20.0.102 - - [2023-04-04 15:14:34] "GET //prepare HTTP/1.1" 200 133 15.356052
Starting the requested run in running controller ... [INFO] [1680621294.359775]: FAILURE: timeout exceeded while waiting for message on topic /camera/depth/image_raw
publishing and latching message for 3.0 seconds

COMMAND TO START INSTANCE HAD NON-ZERO RETURN CODE (56)
Done
172.20.0.102 - - [2023-04-04 15:14:59] "GET //start HTTP/1.1" 200 132 25.524091
172.20.0.254 - - [2023-04-05 01:34:15] "GET /stop HTTP/1.1" 200 130 0.000291
172.20.0.254 - - [2023-04-05 01:34:25] "GET /stop HTTP/1.1" 200 130 0.000263
172.20.0.102 - - [2023-04-05 01:34:28] "GET // HTTP/1.1" 200 152 0.000249
[INFO] [1680658489.851839]: FAILURE: timeout exceeded while waiting for message on topic /camera/depth/image_raw

Controller isn't running, so nothing to stop.
172.20.0.102 - - [2023-04-05 01:34:49] "POST //configure HTTP/1.1" 200 137 20.010748
Preparing the requested controller ... Done
172.20.0.102 - - [2023-04-05 01:35:05] "GET //prepare HTTP/1.1" 200 133 15.491744
Starting the requested run in running controller ... [INFO] [1680658525.361782]: FAILURE: timeout exceeded while waiting for message on topic /camera/depth/image_raw
publishing and latching message for 3.0 seconds

COMMAND TO START INSTANCE HAD NON-ZERO RETURN CODE (56)
Done
172.20.0.102 - - [2023-04-05 01:35:30] "GET //start HTTP/1.1" 200 132 25.523971
172.20.0.254 - - [2023-04-05 01:57:37] "GET /stop HTTP/1.1" 200 130 0.000438
172.20.0.254 - - [2023-04-05 01:57:48] "GET /stop HTTP/1.1" 200 130 0.000363
172.20.0.102 - - [2023-04-05 01:57:52] "GET // HTTP/1.1" 200 152 0.000361
[INFO] [1680659893.557523]: FAILURE: timeout exceeded while waiting for message on topic /camera/depth/image_raw

Controller isn't running, so nothing to stop.
172.20.0.102 - - [2023-04-05 01:58:13] "POST //configure HTTP/1.1" 200 137 20.008390
Preparing the requested controller ... Done
172.20.0.102 - - [2023-04-05 01:58:29] "GET //prepare HTTP/1.1" 200 133 15.475024
Starting the requested run in running controller ... [INFO] [1680659929.052392]: FAILURE: timeout exceeded while waiting for message on topic /camera/depth/image_raw
publishing and latching message for 3.0 seconds

COMMAND TO START INSTANCE HAD NON-ZERO RETURN CODE (52)
Done
172.20.0.102 - - [2023-04-05 01:59:10] "GET //start HTTP/1.1" 200 132 41.816659
172.20.0.254 - - [2023-04-05 02:00:25] "GET /stop HTTP/1.1" 200 130 0.000310
172.20.0.254 - - [2023-04-05 02:00:33] "GET /stop HTTP/1.1" 200 130 0.000255
172.20.0.102 - - [2023-04-05 02:00:37] "GET // HTTP/1.1" 200 152 0.000717
[INFO] [1680660058.302753]: FAILURE: timeout exceeded while waiting for message on topic /camera/depth/image_raw

Controller isn't running, so nothing to stop.
172.20.0.102 - - [2023-04-05 02:00:58] "POST //configure HTTP/1.1" 200 137 20.014455
Preparing the requested controller ... Done
172.20.0.102 - - [2023-04-05 02:01:13] "GET //prepare HTTP/1.1" 200 133 15.453577
Starting the requested run in running controller ... [INFO] [1680660093.774128]: FAILURE: timeout exceeded while waiting for message on topic /camera/depth/image_raw
publishing and latching message for 3.0 seconds

COMMAND TO START INSTANCE HAD NON-ZERO RETURN CODE (56)
Done
172.20.0.102 - - [2023-04-05 02:01:39] "GET //start HTTP/1.1" 200 132 25.525646
172.20.0.254 - - [2023-04-05 02:05:53] "GET /stop HTTP/1.1" 200 130 0.000313
172.20.0.254 - - [2023-04-05 02:06:59] "GET /stop HTTP/1.1" 200 130 0.000263
172.20.0.102 - - [2023-04-05 02:07:02] "GET // HTTP/1.1" 200 152 0.000253
[INFO] [1680660444.006010]: FAILURE: timeout exceeded while waiting for message on topic /camera/depth/image_raw

Controller isn't running, so nothing to stop.
172.20.0.102 - - [2023-04-05 02:07:24] "POST //configure HTTP/1.1" 200 137 20.008280
Preparing the requested controller ... Done
172.20.0.102 - - [2023-04-05 02:07:39] "GET //prepare HTTP/1.1" 200 133 15.393178
Starting the requested run in running controller ... [INFO] [1680660479.419428]: FAILURE: timeout exceeded while waiting for message on topic /camera/depth/image_raw
publishing and latching message for 3.0 seconds

COMMAND TO START INSTANCE HAD NON-ZERO RETURN CODE (56)
Done
172.20.0.102 - - [2023-04-05 02:08:04] "GET //start HTTP/1.1" 200 132 25.527548

Any idea to fix this ... Thanks

david2611 commented 1 year ago

Hmmm ok checking another couple of things.Is this your first time installing BenchBot or has it worked for you previously?

Am I correct in assuming after everything breaks and before you start a fresh attempt you have been running a benchbot_run -k command?

Using the same setup as previously, instead of the docker logs command can I get you to print the output of the following? docker exec benchbot_robot /bin/bash -c "cat /tmp/benchbot_logs/0"

That should give us the full terminal output from Isaac Sim. It is unusual that Isaac Sim itself is breaking.

tan007 commented 1 year ago

Yes. This is my first time installing BenchBot. The log printout is the following:

No python_sh_path provided. I SUPPOSE I SHALL BE Guessing ... 
Guessed: ./python.sh
Found existing installation: click 7.1.2
Uninstalling click-7.1.2:
  Successfully uninstalled click-7.1.2
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: click in ./kit/python/lib/python3.7/site-packages (8.1.3)
Requirement already satisfied: importlib-metadata in ./kit/extscore/omni.kit.pip_archive/pip_prebundle (from click) (5.0.0)
Requirement already satisfied: zipp>=0.5 in ./kit/extscore/omni.kit.pip_archive/pip_prebundle (from importlib-metadata->click) (3.10.0)
Requirement already satisfied: typing-extensions>=3.6.4 in ./kit/extscore/omni.kit.pip_archive/pip_prebundle (from importlib-metadata->click) (3.7.4.3)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 21.2.1+nv1; however, version 23.0.1 is available.
You should consider upgrading via the '/isaac-sim/kit/python/bin/python3 -m pip install --upgrade pip' command.
Found existing installation: typing-extensions 3.7.4.3
Uninstalling typing-extensions-3.7.4.3:
  Successfully uninstalled typing-extensions-3.7.4.3
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: typing-extensions in ./extscache/omni.pip.torch-1_13_1-0.1.4+104.2.lx64/torch-1-13-1 (4.4.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 21.2.1+nv1; however, version 23.0.1 is available.
You should consider upgrading via the '/isaac-sim/kit/python/bin/python3 -m pip install --upgrade pip' command.
HACK FIX FOR BROKEN PACKAGES
STARTING RUN.PY IN BENCHBOT_SIM_OMNI
inside run.py __main__
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0STARTING INSTANCE!!
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0[Warning] [omni.isaac.kit.simulation_app] Modules: ['omni.isaac.kit.app_framework'] were loaded before SimulationApp was started and might not be loaded correctly.
[Warning] [omni.isaac.kit.simulation_app] Please check to make sure no extra omniverse or pxr modules are imported before the call to SimulationApp(...)
[Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn't available.
[Info] [carb] Logging to file: /isaac-sim/kit/logs/Kit/Isaac-Sim/2022.2/kit_20230405_180052.log
2023-04-05 18:00:52 [15ms] [Warning] [omni.ext.plugin] [ext: omni.sensors.nv.lidar] Extensions config 'extension.toml' doesn't exist '/isaac-sim/exts/omni.sensors.nv.lidar' or '/isaac-sim/exts/omni.sensors.nv.lidar/config'
2023-04-05 18:00:52 [15ms] [Warning] [omni.ext.plugin] [ext: omni.sensors.nv.radar] Extensions config 'extension.toml' doesn't exist '/isaac-sim/exts/omni.sensors.nv.radar' or '/isaac-sim/exts/omni.sensors.nv.radar/config'
[0.167s] [ext: omni.stats-0.0.0] startup
[0.176s] [ext: omni.rtx.shadercache-1.0.0] startup
[0.180s] [ext: omni.assets.plugins-0.0.0] startup
[0.181s] [ext: omni.gpu_foundation-0.0.0] startup
[0.184s] [ext: carb.windowing.plugins-1.0.0] startup
[0.197s] [ext: omni.kit.renderer.init-0.0.0] startup

|---------------------------------------------------------------------------------------------|
| Driver Version: 530.30.02     | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name                             | Active | LDA | GPU Memory | Vendor-ID | LUID       |
|     |                                  |        |     |            | Device-ID | UUID       |
|---------------------------------------------------------------------------------------------|
| 0   | NVIDIA GeForce RTX 4090          | Yes: 0 |     | 24564   MB | 10de      | 0          |
|     |                                  |        |     |            | 2684      | 6daf179d.. |
|=============================================================================================|
| OS: Linux benchbot_robot, Version: 5.15.0-69-generic
| XServer Vendor: The X.Org Foundation, XServer Version: 12013000 (1.20.13.0)
| Processor: 13th Gen Intel(R) Core(TM) i9-13900 | Cores: Unknown | Logical: 32
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 128561 | Free Memory: 118100
| Total Page/Swap (MB): 2047 | Free Page/Swap: 2047
|---------------------------------------------------------------------------------------------|
[0.973s] [ext: omni.kit.pipapi-0.0.0] startup
[0.978s] [ext: omni.kit.pip_archive-0.0.0] startup
[0.979s] [ext: omni.kit.loop-isaac-1.0.0] startup
[0.980s] [ext: omni.kit.async_engine-0.0.0] startup
[0.980s] [ext: omni.kit.test-0.0.0] startup
[0.993s] [ext: omni.usd.config-1.0.0] startup
[0.996s] [ext: omni.usd.libs-1.0.0] startup
[1.126s] [ext: omni.isaac.core_archive-2.0.1] startup
[1.132s] [ext: omni.pip.torch-1_13_1-0.1.4] startup
[1.133s] [ext: omni.isaac.ml_archive-1.1.0] startup
[1.156s] [ext: omni.client-0.1.1] startup
[1.161s] [ext: omni.appwindow-1.0.1] startup
[1.163s] [ext: omni.kit.renderer.core-0.0.0] startup
[1.191s] [ext: omni.kit.renderer.capture-0.0.0] startup
[1.192s] [ext: omni.kit.renderer.imgui-0.0.0] startup
[1.235s] [ext: carb.audio-0.1.0] startup
[1.236s] [ext: omni.ui-2.14.4] startup
[1.242s] [ext: omni.uiaudio-1.0.0] startup
[1.243s] [ext: omni.kit.mainwindow-1.0.0] startup
[1.244s] [ext: omni.kit.uiapp-0.0.0] startup
[1.244s] [ext: omni.usd.schema.physics-1.0.0] startup
[1.274s] [ext: omni.usd.schema.isaac-1.1.0] startup
[1.286s] [ext: omni.usd.schema.semantics-0.0.0] startup
[1.293s] [ext: omni.usd.schema.omnigraph-1.0.0] startup
[1.298s] [ext: omni.usd.schema.forcefield-0.0.0] startup
[1.303s] [ext: omni.usd.schema.omniscripting-1.0.0] startup
[1.308s] [ext: omni.usd.schema.geospatial-0.0.0] startup
[1.313s] [ext: omni.usd.schema.anim-0.0.0] startup
[1.354s] [ext: omni.usd.schema.audio-0.0.0] startup
[1.358s] [ext: omni.usd.schema.scene.visualization-1.0.0] startup
[1.359s] [ext: omni.usd.schema.physx-0.0.0] startup
[1.378s] [ext: omni.gpucompute.plugins-0.0.0] startup
[1.378s] [ext: omni.timeline-1.0.5] startup
[1.379s] [ext: omni.kit.search_core-1.0.2] startup
[1.379s] [ext: omni.hydra.scene_delegate-0.3.0] startup
[1.383s] [ext: omni.kit.widget.path_field-2.0.4] startup
[1.386s] [ext: omni.kit.widget.filebrowser-2.3.10] startup
[1.389s] [ext: omni.kit.widget.browser_bar-2.0.5] startup
[1.389s] [ext: omni.kit.window.popup_dialog-2.0.16] startup
[1.392s] [ext: omni.kit.actions.core-1.0.0] startup
[1.393s] [ext: omni.usd.core-1.0.4] startup
[1.395s] [ext: omni.kit.audiodeviceenum-1.0.0] startup
[1.395s] [ext: omni.kit.widget.nucleus_connector-1.0.3] startup
[1.396s] [ext: omni.kit.commands-1.4.5] startup
[1.399s] [ext: omni.hydra.usdrt_delegate-4.3.2] startup
[1.408s] [ext: omni.kit.notification_manager-1.0.5] startup
[1.409s] [ext: omni.usd-1.6.30] startup
[1.436s] [ext: omni.kit.widget.versioning-1.3.8] startup
[1.437s] [ext: omni.iray.libs-0.0.0] startup
[1.439s] [ext: omni.kit.window.filepicker-2.7.15] startup
[1.470s] [ext: omni.mdl.neuraylib-0.1.0] startup
[1.472s] [ext: omni.kit.collaboration.channel_manager-1.0.9] startup
[1.473s] [ext: omni.kit.window.file_importer-1.0.10] startup
[1.474s] [ext: omni.mdl-0.1.0] startup
[1.515s] [ext: omni.kit.usd.layers-2.0.11] startup
[1.520s] [ext: omni.kit.menu.utils-1.4.7] startup
[1.526s] [ext: omni.kit.material.library-1.3.21] startup
[1.530s] [ext: omni.kit.menu.create-1.0.8] startup
[1.531s] [ext: omni.kit.clipboard-1.0.0] startup
[1.531s] [ext: omni.kit.widget.graph-1.5.6-104_2] startup
[1.547s] [ext: omni.kit.context_menu-1.5.12] startup
[1.549s] [ext: omni.kit.window.extensions-1.1.1] startup
[1.552s] [ext: omni.inspect-1.0.1] startup
[1.554s] [ext: omni.kit.window.property-1.8.2] startup
[1.555s] [ext: omni.kit.primitive.mesh-1.0.8] startup
[1.557s] [ext: omni.graph.core-2.65.4] startup
[1.559s] [ext: omni.isaac.version-1.0.0] startup
[1.559s] [ext: omni.kit.stage_templates-1.1.13] startup
[1.561s] [ext: omni.kit.usd_undo-0.1.2] startup
[1.561s] [ext: omni.graph.tools-1.17.2] startup
[1.571s] [ext: omni.ui.scene-1.5.18] startup
[1.574s] [ext: omni.graph-1.50.2] startup
[1.606s] [ext: omni.ui_query-1.1.1] startup
[1.607s] [ext: omni.volume-0.1.0] startup
[1.608s] [ext: omni.kit.ui_test-1.2.9] startup
[1.609s] [ext: omni.hydra.rtx-0.1.0] startup
[1.612s] [ext: omni.kit.widget.searchfield-1.0.10] startup
[1.613s] [ext: omni.kit.graph.delegate.default-1.0.18] startup
[1.615s] [ext: omni.debugdraw-0.1.1] startup
[1.618s] [ext: omni.kit.graph.editor.core-1.4.7] startup
[1.619s] [ext: omni.hydra.pxr-1.1.2] startup
[1.622s] [ext: omni.activity.core-1.0.1] startup
[1.624s] [ext: omni.kit.hydra_texture-1.0.11] startup
[1.627s] [ext: omni.kit.viewport.legacy_gizmos-1.0.10] startup
[1.630s] [ext: omni.kit.viewport.registry-104.0.2] startup
[1.631s] [ext: omni.kit.widget.viewport-104.1.16] startup
[1.632s] [ext: omni.kit.window.drop_support-1.0.1] startup
[1.633s] [ext: omni.kit.window.file_exporter-1.0.10] startup
[1.633s] [ext: omni.hydra.engine.stats-1.0.0] startup
[1.635s] [ext: omni.kit.window.content_browser_registry-0.0.1] startup
[1.636s] [ext: omni.kit.window.file-1.3.32] startup
[1.637s] [ext: omni.kit.viewport.window-104.1.23] startup
[1.653s] [ext: omni.kit.widget.settings-1.0.1] startup
[1.654s] [ext: omni.kit.widget.stage-2.7.24] startup
[1.657s] [ext: omni.kit.window.content_browser-2.6.8] startup
[1.670s] [ext: omni.kit.viewport.utility-1.0.14] startup
[1.670s] [ext: omni.kit.window.preferences-1.3.8] startup
[1.693s] [ext: omni.kit.widget.text_editor-1.0.2] startup
[1.694s] [ext: omni.kit.widget.prompt-1.0.5] startup
[1.695s] [ext: omni.kit.property.usd-3.18.17] startup
[1.702s] [ext: omni.kit.graph.delegate.modern-1.6.5] startup
[1.703s] [ext: omni.graph.ui-1.24.2] startup
[1.758s] [ext: omni.physics.tensors-0.1.0] startup
[1.763s] [ext: omni.kit.graph.widget.variables-2.0.4] startup
[1.764s] [ext: omni.kit.graph.usd.commands-1.2.0] startup
[1.764s] [ext: omni.convexdecomposition-104.2.4-5.1] startup
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0i.graph.window.core-1.43.3] startup
[1.771s] [ext: omni.kit.widget.zoombar-1.0.4] startup
[1.772s] [ext: omni.kvdb-0.0.0] startup
[1.773s] [ext: omni.usdphysics-104.2.4-5.1] startup
[1.775s] [ext: omni.localcache-0.0.0] startup
[1.776s] [ext: omni.graph.scriptnode-0.10.0] startup
[1.777s] [ext: omni.physx-104.2.4-5.1] startup
[1.849s] [ext: omni.command.usd-1.0.2] startup
[1.851s] [ext: omni.physx.tensors-0.1.0] startup
[1.854s] [ext: omni.graph.window.generic-1.3.15] startup
[1.855s] [ext: omni.kit.browser.core-2.2.2] startup
[1.857s] [ext: omni.graph.action-1.31.1] startup
[1.863s] [ext: omni.kit.browser.folder.core-1.7.3] startup
[1.865s] [ext: omni.kit.window.cursor-1.1.1] startup
[1.865s] [ext: omni.kit.widget.spinner-1.0.4] startup
[1.866s] [ext: omni.kit.browser.sample-1.2.5] startup
[1.867s] [ext: omni.graph.nodes-1.48.3] startup
[1.875s] [ext: omni.kit.viewport.menubar.core-104.4.3] startup
[1.886s] [ext: omni.kit.numpy.common-0.1.0] startup
[1.888s] [ext: omni.warp-0.6.3] startup
[2.049s] [ext: omni.syntheticdata-0.2.4] startup
2023-04-05 18:00:54 [2,050ms] [Warning] [omni.kit.viewport.menubar.core.menu_item.viewport_menu_item] ViewportMenuItem order argument is deprecated, use order_setting_path
[2.059s] [ext: omni.isaac.dynamic_control-1.2.3] startup
[2.065s] [ext: omni.replicator.core-1.7.8] startup
2023-04-05 18:00:54 [2,150ms] [Warning] [omni.replicator.core.scripts.annotators] Annotator PostProcessDispatch is already registered, overwriting annotator template
[2.221s] [ext: omni.isaac.core-1.46.3] startup
[2.746s] [ext: omni.kit.manipulator.viewport-104.0.7] startup
[2.747s] [ext: omni.isaac.core_nodes-0.23.2] startup
[2.755s] [ext: omni.isaac.cloner-0.4.1] startup
[2.756s] [ext: omni.isaac.lula-1.2.1] startup
[2.763s] [ext: omni.graph.visualization.nodes-2.0.0] startup
[2.767s] [ext: omni.isaac.debug_draw-0.2.3] startup
[2.770s] [ext: omni.isaac.gym-0.3.3] startup
[2.770s] [ext: omni.graph.window.action-1.5.5] startup
[2.771s] [ext: omni.isaac.ui-0.8.0] startup
[2.773s] [ext: usdrt.scenegraph-4.3.2] startup
[2.785s] [ext: omni.isaac.motion_planning-0.2.2] startup
[2.789s] [ext: omni.isaac.surface_gripper-0.4.0] startup
[2.792s] [ext: omni.isaac.range_sensor-0.8.1] startup
[2.811s] [ext: omni.isaac.motion_generation-4.5.6] startup
[2.815s] [ext: omni.isaac.manipulators-1.1.0] startup
[2.816s] [ext: omni.isaac.sensor-5.6.3] startup
2023-04-05 18:00:55 [2,810ms] [Warning] [omni.ext.plugin] [ext: omni.isaac.sensor-5.6.3] Native plugin search dir contains several files: /isaac-sim/exts/omni.isaac.sensor/bin . It makes dir walk slow, replace with the exact path to the plugin instead of a wildcard.
[2.837s] [ext: omni.isaac.dofbot-0.3.0] startup
[2.837s] [ext: omni.isaac.occupancy_map-0.2.7] startup
[2.845s] [ext: omni.isaac.quadruped-1.3.0] startup
[3.072s] [ext: omni.isaac.scene_blox-0.1.0] startup
[3.072s] [ext: omni.isaac.synthetic_utils-0.4.3] startup
[3.074s] [ext: omni.isaac.universal_robots-0.3.2] startup
[3.074s] [ext: omni.isaac.isaac_sensor-2.0.0] startup
2023-04-05 18:00:55 [3,068ms] [Warning] [omni.isaac.isaac_sensor] omni.isaac.isaac_sensor is deprecated and was renamed; use omni.isaac.sensor
[3.075s] [ext: omni.kit.widget.live_session_management-1.1.3] startup
[3.076s] [ext: omni.kit.hotkeys.core-1.0.8] startup
[3.080s] [ext: omni.isaac.wheeled_robots-0.6.3] startup
[3.085s] [ext: omni.kit.widget.stage_icons-1.0.2] startup
[3.086s] [ext: omni.kit.widget.layers-1.6.11] startup
[3.092s] [ext: omni.kit.selection-0.1.0] startup
[3.094s] [ext: omni.isaac.mjcf-0.3.1] startup
[3.098s] [ext: omni.kit.window.stage-2.3.12] startup
[3.100s] [ext: omni.kit.property.layer-1.1.4] startup
[3.101s] [ext: omni.kit.menu.edit-1.1.6] startup
[3.103s] [ext: omni.kit.menu.file-1.1.4] startup
[3.103s] [ext: omni.kit.menu.stage-1.2.1] startup
[3.104s] [ext: omni.kit.property.audio-1.0.6] startup
[3.105s] [ext: omni.kit.property.camera-1.0.3] startup
[3.105s] [ext: omni.kit.property.geometry-1.2.2] startup
[3.107s] [ext: omni.kit.property.light-1.0.6] startup
[3.108s] [ext: omni.hydra.scene_api-0.1.1] startup
100     3    0     3    0     0      0      0 --:--:--  0:00:05 --:--:--     0
{}
Starting kit application with the following args:  ['/isaac-sim/exts/omni.isaac.kit/omni/isaac/kit/simulation_app.py', '/isaac-sim/apps/omni.isaac.sim.python.kit', '--/app/tokens/exe-path=/isaac-sim/kit', '--/persistent/app/viewport/displayOptions=3094', '--/rtx/materialDb/syncLoads=True', '--/rtx/hydra/materialSyncLoads=True--/omni.kit.plugin/syncUsdLoads=True', '--/app/renderer/resolution/width=1280', '--/app/renderer/resolution/height=720', '--/app/window/width=1440', '--/app/window/height=900', '--/renderer/multiGpu/enabled=True', '--/app/fastShutdown=True', '--ext-folder', '/isaac-sim/exts', '--ext-folder', '/isaac-sim/apps', '--/physics/cudaDevice=0', '--portable', '--allow-root']
Passing the following args to the base kit application:  []
Warp 0.6.3 initialized:
   CUDA Toolkit: 11.5, Driver: 12.1
   Devices:
     "cpu"    | x86_64
     "cuda:0" | NVIDIA GeForce RTX 4090 (sm_89)
   Kernel cache: /root/.cache/warp/0.6.3
[5.534s] RTX ready
Skipping. No running simulation to stop
xt: omni.kit.property.material-1.8.19] startup
[3.113s] [ext: omni.kit.property.render-1.1.0] startup
[3.113s] [ext: omni.kit.property.skel-1.0.1] startup
[3.114s] [ext: omni.kit.property.transform-1.3.5] startup
[3.116s] [ext: omni.kit.manipulator.transform-104.6.12] startup
[3.117s] [ext: omni.kit.widget.toolbar-1.4.0] startup
[3.121s] [ext: omni.kit.profiler.window-2.0.5] startup
[3.165s] [ext: omni.kit.property.bundle-1.2.6] startup
[3.166s] [ext: omni.kit.manipulator.selector-1.0.1] startup
[3.167s] [ext: omni.kit.manipulator.tool.snap-1.2.4] startup
[3.170s] [ext: omni.kit.window.toolbar-1.4.0] startup
[3.171s] [ext: omni.kit.property.isaac-0.2.0] startup
[3.173s] [ext: omni.kit.stage_column.payload-1.0.8] startup
[3.174s] [ext: omni.kit.manipulator.camera-104.0.23] startup
[3.174s] [ext: omni.kit.manipulator.prim-104.10.7] startup
[3.177s] [ext: omni.kit.manipulator.selection-104.0.6] startup
[3.178s] [ext: omni.kit.viewport.menubar.display-104.1.11] startup
[3.179s] [ext: omni.kit.viewport.menubar.camera-104.1.9] startup
[3.181s] [ext: omni.rtx.window.settings-0.6.2] startup
[3.184s] [ext: omni.kit.viewport.menubar.settings-104.3.5] startup
[3.187s] [ext: omni.kit.viewport.menubar.render-104.0.20] startup
[3.189s] [ext: omni.physx.commands-104.2.4-5.1] startup
[3.192s] [ext: omni.kit.stage_column.variant-1.0.9] startup
[3.192s] [ext: omni.hydra.pxr.settings-1.0.3] startup
[3.193s] [ext: omni.kit.viewport.bundle-104.0.2] startup
[3.193s] [ext: omni.physx.ui-104.2.4-5.1] startup
[3.226s] [ext: omni.kit.viewport.menubar.lighting-104.0.9] startup
[3.227s] [ext: omni.kit.viewport.pxr-104.0.1] startup
[3.227s] [ext: omni.kit.viewport.ready-1.0.2] startup
[3.228s] [ext: omni.physx.demos-104.2.4-5.1] startup
[3.230s] [ext: omni.kit.property.physx-0.1.0] startup
[3.258s] [ext: omni.isaac.urdf-0.5.9] startup
[3.275s] [ext: omni.kit.widget.highlight_label-1.0.0] startup
[3.275s] [ext: omni.physx.supportui-104.2.4-5.1] startup
[3.284s] [ext: omni.physx.vehicle-104.2.4-5.1] startup
[3.291s] [ext: omni.physx.cct-104.2.4-5.1] startup
[3.296s] [ext: omni.physx.camera-104.2.4-5.1] startup
[3.299s] [ext: omni.kit.actions.window-1.1.0] startup
[3.302s] [ext: omni.kit.window.commands-0.2.4] startup
[3.303s] [ext: omni.kit.window.console-0.2.3] startup
[3.308s] [ext: omni.physx.bundle-104.2.4-5.1] startup
[3.308s] [ext: omni.graph.bundle.action-1.3.0] startup
[3.308s] [ext: omni.kit.window.script_editor-1.7.2] startup
[3.309s] [ext: omni.kit.hotkeys.window-1.1.3] startup
[3.312s] [ext: omni.kit.window.status_bar-0.1.5] startup
[3.314s] [ext: omni.kit.window.title-1.1.2] startup
[3.315s] [ext: omni.replicator.composer-1.2.10] startup
[3.318s] [ext: omni.replicator.isaac-1.7.4] startup
[3.327s] [ext: omni.resourcemonitor-1.0.0] startup
[3.330s] [ext: omni.rtx.settings.core-0.5.8] startup
[3.334s] [ext: omni.isaac.franka-0.4.0] startup
[3.334s] [ext: omni.kit.viewport.rtx-104.0.0] startup
[3.334s] [ext: semantics.schema.editor-0.3.3] startup
[3.337s] [ext: omni.kit.widget.live-2.0.3] startup
2023-04-05 18:00:55 [3,331ms] [Warning] [omni.kit.widget.live.cache_state_menu] Unable to detect Omniverse Cache Server. Consider installing it for better IO performance.
[3.338s] [ext: omni.isaac.utils-0.2.4] startup
[3.340s] [ext: omni.isaac.kit-1.4.1] startup
[3.340s] [ext: omni.isaac.cortex-0.3.2] startup
[3.341s] [ext: omni.kit.window.stats-0.1.2] startup
[3.341s] [ext: omni.isaac.sim.python-2022.2.1] startup
[3.342s] Simulation App Starting
[5.414s] app ready
[5.663s] Simulation App Startup Complete
[5.671s] [ext: omni.isaac.ros_bridge-1.11.2] startup
2023-04-05 18:01:22 [29,776ms] [Error] [carb.graphics-vulkan.plugin] GPU crash is detected. Crash dump is written into: /isaac-sim/kit/logs/Kit/Isaac-Sim/2022.2/kit_20230405_180052.nv-gpudmp
2023-04-05 18:01:22 [29,776ms] [Warning] [carb] Acquiring non optional plugin interface which is not listed as dependency: [carb::filesystem::IFileSystem v1.2] (plugin: (null)), by client: carb.graphics-vulkan.plugin. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client.
2023-04-05 18:01:22 [29,776ms] [Error] [carb.graphics-vulkan.plugin] GPU crash dump is successfully written
2023-04-05 18:01:22 [29,843ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_floor_01.mdl?watch=00007f3e1a1ddb00(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,844ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_floor_01.mdl?watch=00007f3e1a1ddb00(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,844ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_floor_01.mdl?watch=00007f3e1a1ddb00(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,844ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_floor_01.mdl?watch=00007f3e1a1ddb00(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,844ms] [Warning] [rtx.neuraylib.plugin] Compiler Core: file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_floor_01.mdl?watch=00007f3e1a1ddb00(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,844ms] [Warning] [rtx.neuraylib.plugin] Compiler Core: file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_floor_01.mdl?watch=00007f3e1a1ddb00(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,852ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_radio_01.mdl?watch=00007f3e083cbc10(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,852ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_radio_01.mdl?watch=00007f3e083cbc10(189,9): C181 unused variable 'CustomizedUV1_mdl'
2023-04-05 18:01:22 [29,853ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_radio_01.mdl?watch=00007f3e083cbc10(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,853ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_radio_01.mdl?watch=00007f3e083cbc10(189,9): C181 unused variable 'CustomizedUV1_mdl'
2023-04-05 18:01:22 [29,853ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_radio_01.mdl?watch=00007f3e083cbc10(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,853ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_radio_01.mdl?watch=00007f3e083cbc10(189,9): C181 unused variable 'CustomizedUV1_mdl'
2023-04-05 18:01:22 [29,853ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_radio_01.mdl?watch=00007f3e083cbc10(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,853ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_radio_01.mdl?watch=00007f3e083cbc10(189,9): C181 unused variable 'CustomizedUV1_mdl'
2023-04-05 18:01:22 [29,853ms] [Warning] [rtx.neuraylib.plugin] Compiler Core: file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_radio_01.mdl?watch=00007f3e083cbc10(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,853ms] [Warning] [rtx.neuraylib.plugin] Compiler Core: file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_radio_01.mdl?watch=00007f3e083cbc10(189,9): C181 unused variable 'CustomizedUV1_mdl'
2023-04-05 18:01:22 [29,853ms] [Warning] [rtx.neuraylib.plugin] Compiler Core: file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_radio_01.mdl?watch=00007f3e083cbc10(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,853ms] [Warning] [rtx.neuraylib.plugin] Compiler Core: file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_radio_01.mdl?watch=00007f3e083cbc10(189,9): C181 unused variable 'CustomizedUV1_mdl'
2023-04-05 18:01:22 [29,858ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_table_01.mdl?watch=00007f3be7ee2120(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,858ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_table_01.mdl?watch=00007f3be7ee2120(189,9): C181 unused variable 'CustomizedUV1_mdl'
2023-04-05 18:01:22 [29,858ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_table_01.mdl?watch=00007f3be7ee2120(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,858ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_table_01.mdl?watch=00007f3be7ee2120(189,9): C181 unused variable 'CustomizedUV1_mdl'
2023-04-05 18:01:22 [29,859ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_table_01.mdl?watch=00007f3be7ee2120(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,859ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_table_01.mdl?watch=00007f3be7ee2120(189,9): C181 unused variable 'CustomizedUV1_mdl'
2023-04-05 18:01:22 [29,859ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_table_01.mdl?watch=00007f3be7ee2120(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,859ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_table_01.mdl?watch=00007f3be7ee2120(189,9): C181 unused variable 'CustomizedUV1_mdl'
2023-04-05 18:01:22 [29,859ms] [Warning] [rtx.neuraylib.plugin] Compiler Core: file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_table_01.mdl?watch=00007f3be7ee2120(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,859ms] [Warning] [rtx.neuraylib.plugin] Compiler Core: file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_table_01.mdl?watch=00007f3be7ee2120(189,9): C181 unused variable 'CustomizedUV1_mdl'
2023-04-05 18:01:22 [29,859ms] [Warning] [rtx.neuraylib.plugin] Compiler Core: file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_table_01.mdl?watch=00007f3be7ee2120(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,859ms] [Warning] [rtx.neuraylib.plugin] Compiler Core: file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_table_01.mdl?watch=00007f3be7ee2120(189,9): C181 unused variable 'CustomizedUV1_mdl'
2023-04-05 18:01:22 [29,885ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_metal_dirty_01.mdl?watch=00007f3e03950730(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,885ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_metal_dirty_01.mdl?watch=00007f3e03950730(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,886ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_metal_dirty_01.mdl?watch=00007f3e03950730(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,886ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_metal_dirty_01.mdl?watch=00007f3e03950730(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,886ms] [Warning] [rtx.neuraylib.plugin] Compiler Core: file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_metal_dirty_01.mdl?watch=00007f3e03950730(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,886ms] [Warning] [rtx.neuraylib.plugin] Compiler Core: file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_metal_dirty_01.mdl?watch=00007f3e03950730(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,895ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_cuttingboard_03.mdl?watch=00007f3e0387efb0(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,896ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_cuttingboard_03.mdl?watch=00007f3e0387efb0(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,896ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_cuttingboard_03.mdl?watch=00007f3e0387efb0(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,896ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_cuttingboard_03.mdl?watch=00007f3e0387efb0(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,896ms] [Warning] [rtx.neuraylib.plugin] Compiler Core: file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_cuttingboard_03.mdl?watch=00007f3e0387efb0(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,896ms] [Warning] [rtx.neuraylib.plugin] Compiler Core: file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_cuttingboard_03.mdl?watch=00007f3e0387efb0(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,902ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_chair_02.mdl?watch=00007f2ccc566230(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,902ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_chair_02.mdl?watch=00007f2ccc566230(189,9): C181 unused variable 'CustomizedUV1_mdl'
2023-04-05 18:01:22 [29,903ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_chair_02.mdl?watch=00007f2ccc566230(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,903ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_chair_02.mdl?watch=00007f2ccc566230(189,9): C181 unused variable 'CustomizedUV1_mdl'
2023-04-05 18:01:22 [29,903ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_chair_02.mdl?watch=00007f2ccc566230(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,903ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_chair_02.mdl?watch=00007f2ccc566230(189,9): C181 unused variable 'CustomizedUV1_mdl'
2023-04-05 18:01:22 [29,903ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_chair_02.mdl?watch=00007f2ccc566230(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,903ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_chair_02.mdl?watch=00007f2ccc566230(189,9): C181 unused variable 'CustomizedUV1_mdl'
2023-04-05 18:01:22 [29,903ms] [Warning] [rtx.neuraylib.plugin] Compiler Core: file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_chair_02.mdl?watch=00007f2ccc566230(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,903ms] [Warning] [rtx.neuraylib.plugin] Compiler Core: file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_chair_02.mdl?watch=00007f2ccc566230(189,9): C181 unused variable 'CustomizedUV1_mdl'
2023-04-05 18:01:22 [29,903ms] [Warning] [rtx.neuraylib.plugin] Compiler Core: file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_chair_02.mdl?watch=00007f2ccc566230(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,903ms] [Warning] [rtx.neuraylib.plugin] Compiler Core: file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_chair_02.mdl?watch=00007f2ccc566230(189,9): C181 unused variable 'CustomizedUV1_mdl'
2023-04-05 18:01:22 [29,938ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_1_wood_table_black_01.mdl?watch=00007f2ce50b2e10(287,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,938ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_1_wood_table_black_01.mdl?watch=00007f2ce50b2e10(287,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,939ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_1_wood_table_black_01.mdl?watch=00007f2ce50b2e10(287,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,939ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_1_wood_table_black_01.mdl?watch=00007f2ce50b2e10(287,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,939ms] [Warning] [rtx.neuraylib.plugin] Compiler Core: file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_1_wood_table_black_01.mdl?watch=00007f2ce50b2e10(287,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,939ms] [Warning] [rtx.neuraylib.plugin] Compiler Core: file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_1_wood_table_black_01.mdl?watch=00007f2ce50b2e10(287,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,965ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_branches_02.mdl?watch=00007f3e03a57d30(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,965ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_branches_02.mdl?watch=00007f3e03a57d30(189,9): C181 unused variable 'CustomizedUV1_mdl'
2023-04-05 18:01:22 [29,965ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_branches_02.mdl?watch=00007f3e03a57d30(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,965ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_branches_02.mdl?watch=00007f3e03a57d30(189,9): C181 unused variable 'CustomizedUV1_mdl'
2023-04-05 18:01:22 [29,966ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_branches_02.mdl?watch=00007f3e03a57d30(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,966ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_branches_02.mdl?watch=00007f3e03a57d30(189,9): C181 unused variable 'CustomizedUV1_mdl'
2023-04-05 18:01:22 [29,966ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_branches_02.mdl?watch=00007f3e03a57d30(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,966ms] [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn : file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_branches_02.mdl?watch=00007f3e03a57d30(189,9): C181 unused variable 'CustomizedUV1_mdl'
2023-04-05 18:01:22 [29,966ms] [Warning] [rtx.neuraylib.plugin] Compiler Core: file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_branches_02.mdl?watch=00007f3e03a57d30(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,966ms] [Warning] [rtx.neuraylib.plugin] Compiler Core: file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_branches_02.mdl?watch=00007f3e03a57d30(189,9): C181 unused variable 'CustomizedUV1_mdl'
2023-04-05 18:01:22 [29,966ms] [Warning] [rtx.neuraylib.plugin] Compiler Core: file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_branches_02.mdl?watch=00007f3e03a57d30(188,9): C181 unused variable 'CustomizedUV0_mdl'
2023-04-05 18:01:22 [29,966ms] [Warning] [rtx.neuraylib.plugin] Compiler Core: file:/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/.sim_data/Materials/MI_AI_vol3_3_branches_02.mdl?watch=00007f3e03a57d30(189,9): C181 unused variable 'CustomizedUV1_mdl'
DoExecute : m_NvOFAPI->GetAPI()->nvOFExecuteVk(m_NvOFAPI->GetHandle(), &executeInParamsVk, &executeOutParamsVk)returned error 11 at /dvs/p4/build/sw/devrel/libdev/NGX/snippets/rel_tot/source/frameworks/opticalflow/v7/dev/Common/NvOFBase/NvOFVulkan.cpp;421
127.0.0.1 - - [2023-04-05 18:00:45] "GET /started HTTP/1.1" 200 126 0.000923
127.0.0.1 - - [2023-04-05 18:00:45] "GET /started HTTP/1.1" 200 126 0.000206
127.0.0.1 - - [2023-04-05 18:00:45] "GET /started HTTP/1.1" 200 126 0.000844
127.0.0.1 - - [2023-04-05 18:00:45] "GET /started HTTP/1.1" 200 126 0.000368
127.0.0.1 - - [2023-04-05 18:00:46] "GET /started HTTP/1.1" 200 126 0.000352
127.0.0.1 - - [2023-04-05 18:00:46] "GET /started HTTP/1.1" 200 126 0.000227
127.0.0.1 - - [2023-04-05 18:00:46] "GET /started HTTP/1.1" 200 126 0.000204
127.0.0.1 - - [2023-04-05 18:00:46] "GET /started HTTP/1.1" 200 126 0.000264
127.0.0.1 - - [2023-04-05 18:00:47] "GET /started HTTP/1.1" 200 126 0.000907
127.0.0.1 - - [2023-04-05 18:00:47] "GET /started HTTP/1.1" 200 126 0.000872
127.0.0.1 - - [2023-04-05 18:00:47] "GET /started HTTP/1.1" 200 126 0.000873
127.0.0.1 - - [2023-04-05 18:00:48] "GET /started HTTP/1.1" 200 126 0.000242
127.0.0.1 - - [2023-04-05 18:00:48] "GET /started HTTP/1.1" 200 126 0.000259
127.0.0.1 - - [2023-04-05 18:00:48] "GET /started HTTP/1.1" 200 126 0.000238
127.0.0.1 - - [2023-04-05 18:00:48] "GET /started HTTP/1.1" 200 126 0.000789
127.0.0.1 - - [2023-04-05 18:00:49] "GET /started HTTP/1.1" 200 126 0.000247
127.0.0.1 - - [2023-04-05 18:00:49] "GET /started HTTP/1.1" 200 126 0.000258
127.0.0.1 - - [2023-04-05 18:00:49] "GET /started HTTP/1.1" 200 126 0.000782
127.0.0.1 - - [2023-04-05 18:00:49] "GET /started HTTP/1.1" 200 126 0.000358
127.0.0.1 - - [2023-04-05 18:00:50] "GET /started HTTP/1.1" 200 126 0.000367
127.0.0.1 - - [2023-04-05 18:00:50] "GET /started HTTP/1.1" 200 126 0.000364
127.0.0.1 - - [2023-04-05 18:00:50] "GET /started HTTP/1.1" 200 126 0.000213
127.0.0.1 - - [2023-04-05 18:00:50] "GET /started HTTP/1.1" 200 126 0.000214
127.0.0.1 - - [2023-04-05 18:00:51] "GET /started HTTP/1.1" 200 126 0.000347
127.0.0.1 - - [2023-04-05 18:00:51] "GET /started HTTP/1.1" 200 126 0.000218
127.0.0.1 - - [2023-04-05 18:00:51] "GET /started HTTP/1.1" 200 126 0.000203
127.0.0.1 - - [2023-04-05 18:00:51] "GET /started HTTP/1.1" 200 126 0.000199
127.0.0.1 - - [2023-04-05 18:00:52] "GET /started HTTP/1.1" 200 126 0.000351
127.0.0.1 - - [2023-04-05 18:00:52] "GET /started HTTP/1.1" 200 126 0.000245
127.0.0.1 - - [2023-04-05 18:00:58] "POST /start HTTP/1.1" 200 110 5.784261
127.0.0.1 - - [2023-04-05 18:00:58] "GET /started HTTP/1.1" 200 125 0.000323
[2023-04-05 18:01:23,202] ERROR in app: Exception on /open_environment [POST]
Traceback (most recent call last):
  File "/isaac-sim/kit/python/lib/python3.7/site-packages/flask/app.py", line 2528, in wsgi_app
    response = self.full_dispatch_request()
  File "/isaac-sim/kit/python/lib/python3.7/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/isaac-sim/kit/python/lib/python3.7/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "/isaac-sim/kit/python/lib/python3.7/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/benchbot/benchbot_simulator/run.py", line 183, in __open_env
    self.open_usd()
  File "/benchbot/benchbot_simulator/run.py", line 116, in open_usd
    update_stage()
  File "/isaac-sim/exts/omni.isaac.core/omni/isaac/core/utils/stage.py", line 35, in update_stage
    omni.kit.app.get_app_interface().update()
RuntimeError: DoExecute : m_NvOFAPI->GetAPI()->nvOFExecuteVk(m_NvOFAPI->GetHandle(), &executeInParamsVk, &executeOutParamsVk)returned error 11 at /dvs/p4/build/sw/devrel/libdev/NGX/snippets/rel_tot/source/frameworks/opticalflow/v7/dev/Common/NvOFBase/NvOFVulkan.cpp;421

127.0.0.1 - - [2023-04-05 18:01:23] "POST /open_environment HTTP/1.1" 500 401 0.909125
Traceback (most recent call last):
  File "/benchbot/benchbot_simulator/run.py", line 333, in <module>
    sd.run()
  File "/benchbot/benchbot_simulator/run.py", line 237, in run
    self.tick_simulator()
  File "/benchbot/benchbot_simulator/run.py", line 311, in tick_simulator
    self.inst.update()
  File "/isaac-sim/exts/omni.isaac.kit/omni/isaac/kit/simulation_app.py", line 424, in update
    self._app.update()
RuntimeError: Operation not permitted
2023-04-05 18:01:23 [31,025ms] [Warning] [carb.audio.context] 1 contexts were leaked
./python.sh: line 41:   211 Segmentation fault      (core dumped) $python_exe "$@" $args
There was an error running python

Do I need to install any pre-requisite package before installing BenchBot ?

Thanks again for your help...

david2611 commented 1 year ago

Sorry for the delay in response.

The benchbot_install script should handle any packages that are required prior to installation. The Isaac Sim python environment is also separate to whatever you have on your own system.

This might be a hardware or driver issue. Can you tell me what hardware (GPU in particular) you are using and give me the output of nvidia-smi?

When you installed benchbot initially did you pass all of the driver checks or did you need to install drivers through benchbot?

btalb commented 1 year ago

Thanks for raising this @tan007 .

Can you get us a couple of things so we can dig a bit further please:

When done, please post the benchbot.log & vulkan.log files back here for us.

tan007 commented 1 year ago

Thanks for your replay Ben, The benchbot.log is the following.

################################################################################
###################### CHECKING BENCHBOT SCRIPTS VERSION #######################
################################################################################

Fetching latest hash for Benchbot scripts ... 
        f7573674da2905b12d62345721595de7f26a777e.

BenchBot scripts are up-to-date.

################################################################################
######################## PART 1: EXAMINING SYSTEM STATE ########################
################################################################################

Core host system checks:
    Ubuntu version >= 20.04:                                  Passed (20.04)

Running Nvidia related system checks:
    NVIDIA GPU available:                     Found card of type '10de:2684'
    NVIDIA driver is running:                                          Found
    NVIDIA driver version valid:                           Valid (530.30.02)
    NVIDIA driver from a standard PPA:                          PPA is valid
    CUDA drivers installed:                                    Drivers found
    CUDA drivers version valid:                          Valid (530.30.02-1)
    CUDA drivers from the NVIDIA PPA:                           PPA is valid
    CUDA is installed:                                            CUDA found
    CUDA version valid:                                         Valid (12.0)
    CUDA is from the NVIDIA PPA:                                PPA is valid

Running Docker related system checks:
    Docker is available:                                               Found
    Docker version valid:                                     Valid (23.0.3)
    NVIDIA Container Toolkit installed:                       Found (1.13.0)
    Docker runs without root:                                         Passed

Running checks of filesystem used for Docker:
    /var/lib/docker on ext4 filesystem:                 Yes (/dev/nvme0n1p2)
    /var/lib/docker supports suid:                                   Enabled
    /var/lib/docker drive space check:              Sufficient space (1658G)

Miscellaneous requirements:
    Pip python package manager available:                     Found (23.0.1)
    Tkinter for Python installed:                                      Found
    PIL (with ImageTk) for Python install                              Found

Manual installation steps for Omniverse-powered Isaac Sim:
    License accepted for Omniverse:                                      Yes
    Access to nvcr.io Docker registry:                                   Yes

All requirements & dependencies fulfilled. Docker containers for the BenchBot
software stack will now be built (which may take anywhere from a few seconds
to many hours). Would you like to proceed (y/N)? y

Proceeding with install ... 

################################################################################
################ PART 2: FETCHING LATEST BENCHBOT VERSION INFO #################
################################################################################

Fetching latest hash for BenchBot Simulator 'sim_omni' ... 
        c979337d0e0f2d0ff7113d79d126ca62d79dcc5a.
Fetching latest hash for BenchBot Robot Controller ... 
        142365e774cb9bdbcf6681ef4147f93154a4a7ba.
Fetching latest hash for BenchBot Supervisor ... 
        e5e50c46373d24176eb6e99dcc93661086571a68.
Fetching latest hash for BenchBot API ... 
        b8422ccb152332125b4ecb9ed253338fb495dd2d.
Fetching latest hash for BenchBot ROS Messages ... 
        29153dd20b7e611088318cc4b3ef5f1770f44e87.

################################################################################
######################## PART 3: BUILDING DOCKER IMAGES ########################
################################################################################

BUILDING BENCHBOT CORE DOCKER IMAGE:
#1 [internal] load build definition from core.Dockerfile
#1 transferring dockerfile: 2.92kB done
#1 DONE 0.1s

#2 [internal] load .dockerignore
#2 transferring context: 52B done
#2 DONE 0.1s

#3 [internal] load metadata for docker.io/library/ubuntu:focal
#3 DONE 1.6s

#4 [1/5] FROM docker.io/library/ubuntu:focal@sha256:24a0df437301598d1a4b62ddf59fa0ed2969150d70d748c84225e6501e9c36b9
#4 DONE 0.0s

#5 [2/5] RUN echo "Asia/Hong_Kong" > /etc/timezone && ln -s /usr/share/zoneinfo/"Asia/Hong_Kong"     /etc/localtime && apt update && apt -y install tzdata
#5 CACHED

#6 [3/5] RUN apt update && apt install -yq wget gnupg2 software-properties-common git     vim ipython3 tmux iputils-ping
#6 CACHED

#7 [4/5] RUN add-apt-repository ppa:graphics-drivers &&     wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin &&   
  mv -v cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600 &&     apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/u
buntu2004/x86_64/3bf863cc.pub &&     add-apt-repository -n "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /" &&     apt update
#7 CACHED

#8 [5/5] RUN CUDA_NAME="cuda-$(echo "12.0.1-1" |     sed 's/\([0-9]*\)\.\([0-9]*\).*/\1\.\2/; s/\./-/')" &&     NVIDIA_NAME="nvidia-driver-$(echo "530.30.02-0ubuntu1
" |     sed 's/\(^[0-9]*\).*/\1/')" &&     NVIDIA_DEPS="$(apt depends "$NVIDIA_NAME=530.30.02-0ubuntu1" 2>/dev/null |     grep '^ *Depends:' | sed 's/.*Depends: \([^
 ]*\) (.\?= \([^)]*\))/\1 \2/' |     while read d; do read a b <<< "$d"; v=$(apt policy "$a" 2>/dev/null |     grep "$b" | grep -vE "(Installed|Candidate)" | sed "s/
.*($b[^ ]*).*/1/");     echo "$a=$v"; done)" &&     CUDA_DRIVERS_DEPS="$(apt depends "cuda-drivers=530.30.02-1" 2>/dev/null |     grep '^ *Depends:' | sed 's/.*Depen
ds: \([^ ]*\) (.\?= \([^)]*\))/\1 \2/' |     while read d; do read a b <<< "$d"; v=$(apt policy "$a" 2>/dev/null |     grep "$b" | grep -vE "(Installed|Candidate)" |
 sed "s/.*($b[^ ]*).*/1/");     echo "$a=$v"; done)" &&     CUDA_DEPS="$(apt depends "$CUDA_NAME=12.0.1-1" 2>/dev/null |     grep '^ *Depends:' | sed 's/.*Depends: \
([^ ]*\) (.\?= \([^)]*\))/\1 \2/' |     while read d; do read a b <<< "$d"; v=$(apt policy "$a" 2>/dev/null |     grep "$b" | grep -vE "(Installed|Candidate)" | sed 
"s/.*($b[^ ]*).*/1/");     echo "$a=$v"; done)" &&     TARGETS="$(echo "$NVIDIA_DEPS $NVIDIA_NAME=530.30.02-0ubuntu1"     "$CUDA_DRIVERS_DEPS cuda-drivers=530.30.02-
1"     "$CUDA_DEPS $CUDA_NAME=12.0.1-1" |     tr '\n' ' ')" &&     DEBIAN_FRONTEND=noninteractive apt install -yq $TARGETS
#8 CACHED

#9 exporting to image
#9 exporting layers done
#9 writing image sha256:bacb20977f3f36bdd58ca135c1bee1cd3e9cf99cde6ba8d4639c3770204553b6 done
#9 naming to docker.io/benchbot/core:base done
#9 DONE 0.0s

BUILDING BENCHBOT BACKEND DOCKER IMAGE:
#1 [internal] load .dockerignore
#1 transferring context: 52B done
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 3.11kB done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/benchbot/core:base
#3 DONE 0.0s

#4 [1/9] FROM docker.io/benchbot/core:base
#4 DONE 0.0s

#5 [5/9] RUN apt update && apt install -y curl &&     echo "deb http://packages.ros.org/ros/ubuntu focal main" >     /etc/apt/sources.list.d/ros-latest.list &&     c
url -s https://raw.githubusercontent.com/ros/rosdistro/master/ros.asc |     apt-key add - && apt update &&     apt install -y ros-noetic-ros-base python3-rosdep     
python3-rosinstall python3-rosinstall-generator python3-wstool     python3-catkin-tools python3-pip build-essential     ros-noetic-tf2-ros ros-noetic-tf
#5 CACHED

#6 [3/9] RUN apt update && apt install -y python3 python3-pip &&     git clone https://github.com/qcr/benchbot_supervisor /benchbot/benchbot_supervisor &&     pushd 
/benchbot/benchbot_supervisor &&     git checkout e5e50c46373d24176eb6e99dcc93661086571a68 && pip3 install .
#6 CACHED

#7 [6/9] RUN rosdep init && rosdep update && mkdir -p /benchbot/ros_ws/src &&     source /opt/ros/noetic/setup.bash &&     pushd /benchbot/ros_ws && catkin_make && s
ource devel/setup.bash && popd
#7 CACHED

#8 [4/9] RUN mkdir -p /benchbot
#8 CACHED

#9 [7/9] RUN git clone https://github.com/qcr/benchbot_msgs /benchbot/benchbot_msgs &&     pushd /benchbot/benchbot_msgs && git checkout 29153dd20b7e611088318cc4b3ef
5f1770f44e87 &&     pip install -r requirements.txt && pushd /benchbot/ros_ws &&     ln -sv /benchbot/benchbot_msgs src/ && source devel/setup.bash && catkin_make
#9 CACHED

#10 [2/9] WORKDIR /benchbot
#10 CACHED

#11 [8/9] RUN git clone https://github.com/qcr/benchbot_robot_controller /benchbot/benchbot_robot_controller &&     pushd /benchbot/benchbot_robot_controller && git 
checkout 142365e774cb9bdbcf6681ef4147f93154a4a7ba &&     pip install -r requirements.txt &&     sed -i 's/np.float/float/g' /usr/local/lib/python3.8/dist-packages/tr
ansforms3d/quaternions.py &&     pushd /benchbot/ros_ws &&     pushd src && git clone https://github.com/eric-wieser/ros_numpy &&     sed -i 's/np.float/float/g' /be
nchbot/ros_ws/src/ros_numpy/src/ros_numpy/point_cloud2.py &&     popd &&     ln -sv /benchbot/benchbot_robot_controller src/ && source devel/setup.bash && catkin_mak
e
#11 CACHED

#12 [9/9] RUN apt update && apt install -y python3 python3-pip &&     python3 -m pip install --upgrade pip setuptools wheel pyyaml &&     mkdir -p /benchbot/addons
#12 CACHED

#13 exporting to image
#13 exporting layers done
#13 writing image sha256:6d3a5be975d2ae287481d946a7d0b681b0fda59e3402f74ede776dbb3bdd9543 done
#13 naming to docker.io/benchbot/backend:base done
#13 DONE 0.0s

BUILDING BENCHBOT SIMULATOR 'sim_omni' DOCKER IMAGE:
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 3.91kB done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 transferring context: 52B done
#2 DONE 0.0s

#3 [internal] load metadata for nvcr.io/nvidia/isaac-sim:2022.2.1
#3 ...

#4 [auth] nvidia/isaac-sim:pull,push token for nvcr.io
#4 DONE 0.0s
#3 [internal] load metadata for nvcr.io/nvidia/isaac-sim:2022.2.1
#3 DONE 2.1s

#5 [ 1/10] FROM nvcr.io/nvidia/isaac-sim:2022.2.1@sha256:48ccef298a8a7be20a3325a8e9c76546b5bc18ee1480e9594a5977cd2f04a54a
#5 DONE 0.0s

#6 [ 8/10] RUN git clone https://github.com/qcr/benchbot_msgs /benchbot/benchbot_msgs &&     pushd /benchbot/benchbot_msgs && git checkout 29153dd20b7e611088318cc4b3
ef5f1770f44e87 &&     pip install -r requirements.txt && pushd /benchbot/ros_ws &&     ln -sv /benchbot/benchbot_msgs src/ && source devel/setup.bash && catkin_make
#6 CACHED

#7 [ 7/10] RUN rosdep init && rosdep update && mkdir -p /benchbot/ros_ws/src &&     source /opt/ros/noetic/setup.bash &&     pushd /benchbot/ros_ws && catkin_make &&
 source devel/setup.bash && popd
#7 CACHED

#8 [ 9/10] RUN git clone https://github.com/qcr/benchbot_robot_controller /benchbot/benchbot_robot_controller &&     pushd /benchbot/benchbot_robot_controller && git
 checkout 142365e774cb9bdbcf6681ef4147f93154a4a7ba &&     pip install -r requirements.txt &&     sed -i 's/np.float/float/g' /usr/local/lib/python3.8/dist-packages/t
ransforms3d/quaternions.py &&     pushd /benchbot/ros_ws &&     pushd src && git clone https://github.com/eric-wieser/ros_numpy &&     sed -i 's/np.float/float/g' /b
enchbot/ros_ws/src/ros_numpy/src/ros_numpy/point_cloud2.py &&     popd &&     ln -sv /benchbot/benchbot_robot_controller src/ && source devel/setup.bash && catkin_ma
ke
#8 CACHED

#9 [ 2/10] RUN echo "$TZ" > /etc/timezone && ln -s /usr/share/zoneinfo/"$TZ"     /etc/localtime && apt update && apt -y install tzdata
#9 CACHED

#10 [ 4/10] RUN mkdir -p /benchbot/benchbot_simulator &&     git clone https://github.com/qcr/benchbot_sim_omni /benchbot/benchbot_simulator &&     pushd /benchbot/b
enchbot_simulator && git checkout c979337d0e0f2d0ff7113d79d126ca62d79dcc5a &&     /isaac-sim/kit/python/bin/python3 -m pip install -r ./.custom_deps
#10 CACHED

#11 [ 3/10] RUN apt update && apt install -y git
#11 CACHED

#12 [ 5/10] RUN mkdir -p /benchbot
#12 CACHED

#13 [ 6/10] RUN apt update && apt install -y curl &&     echo "deb http://packages.ros.org/ros/ubuntu focal main" >     /etc/apt/sources.list.d/ros-latest.list &&   
  curl -s https://raw.githubusercontent.com/ros/rosdistro/master/ros.asc |     apt-key add - && apt update &&     apt install -y ros-noetic-ros-base python3-rosdep  
   python3-rosinstall python3-rosinstall-generator python3-wstool     python3-catkin-tools python3-pip build-essential     ros-noetic-tf2-ros ros-noetic-tf
#13 CACHED

#14 [10/10] RUN apt update && apt install -y python3 python3-pip &&     python3 -m pip install --upgrade pip setuptools wheel pyyaml &&     mkdir -p /benchbot/addons
#14 CACHED

#15 exporting to image
#15 exporting layers done
#15 writing image sha256:a9e0addd8d2761526a80f030cad3251f1ceee8d2e883f500736a40ab35e9e012 done
#15 naming to docker.io/benchbot/simulator:sim_omni done
#15 DONE 0.0s

BAKING SIMULATOR DATA INTO 'sim_omni' DOCKER IMAGE:
[Warning] [omni.isaac.kit.simulation_app] Modules: ['omni.isaac.kit.app_framework'] were loaded before SimulationApp was started and might not be loaded corre
ctly.
[Warning] [omni.isaac.kit.simulation_app] Please check to make sure no extra omniverse or pxr modules are imported before the call to SimulationApp(...)
Starting kit application with the following args:  ['/isaac-sim/exts/omni.isaac.kit/omni/isaac/kit/simulation_app.py', '/isaac-sim/apps/omni.isaac.sim.python.kit', '
--/app/tokens/exe-path=/isaac-sim/kit', '--/persistent/app/viewport/displayOptions=3094', '--/rtx/materialDb/syncLoads=True', '--/rtx/hydra/materialSyncLoads=True--/
omni.kit.plugin/syncUsdLoads=True', '--/app/renderer/resolution/width=1280', '--/app/renderer/resolution/height=720', '--/app/window/width=1440', '--/app/window/heig
ht=900', '--/renderer/multiGpu/enabled=True', '--/app/fastShutdown=True', '--ext-folder', '/isaac-sim/exts', '--ext-folder', '/isaac-sim/apps', '--/physics/cudaDevic
e=0', '--portable', '--allow-root']
Passing the following args to the base kit application:  []
[Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn't available.
[Info] [carb] Logging to file: /isaac-sim/kit/logs/Kit/Isaac-Sim/2022.2/kit_20230412_034242.log
2023-04-12 03:42:42 [13ms] [Warning] [omni.ext.plugin] [ext: omni.sensors.nv.lidar] Extensions config 'extension.toml' doesn't exist '/isaac-sim/exts/omni.sen
sors.nv.lidar' or '/isaac-sim/exts/omni.sensors.nv.lidar/config'
2023-04-12 03:42:42 [13ms] [Warning] [omni.ext.plugin] [ext: omni.sensors.nv.radar] Extensions config 'extension.toml' doesn't exist '/isaac-sim/exts/omni.s
ensors.nv.radar' or '/isaac-sim/exts/omni.sensors.nv.radar/config'
[0.147s] [ext: omni.stats-0.0.0] startup
[0.159s] [ext: omni.rtx.shadercache-1.0.0] startup
[0.163s] [ext: omni.assets.plugins-0.0.0] startup
[0.163s] [ext: omni.gpu_foundation-0.0.0] startup
[0.167s] [ext: carb.windowing.plugins-1.0.0] startup
[0.179s] [ext: omni.kit.renderer.init-0.0.0] startup

|---------------------------------------------------------------------------------------------|
| Driver Version: 530.30.02     | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name                             | Active | LDA | GPU Memory | Vendor-ID | LUID       |
|     |                                  |        |     |            | Device-ID | UUID       |
|---------------------------------------------------------------------------------------------|
| 0   | NVIDIA GeForce RTX 4090          | Yes: 0 |     | 24564   MB | 10de      | 0          |
|     |                                  |        |     |            | 2684      | 6daf179d.. |
|=============================================================================================|
| OS: Linux c02cca7a8030, Version: 5.15.0-69-generic
| XServer Vendor: The X.Org Foundation, XServer Version: 12013000 (1.20.13.0)
| Processor: 13th Gen Intel(R) Core(TM) i9-13900 | Cores: Unknown | Logical: 32
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 128561 | Free Memory: 123281
| Total Page/Swap (MB): 2047 | Free Page/Swap: 2047
|---------------------------------------------------------------------------------------------|
[0.586s] [ext: omni.kit.pipapi-0.0.0] startup
[0.594s] [ext: omni.kit.pip_archive-0.0.0] startup
[0.602s] [ext: omni.kit.loop-isaac-1.0.0] startup
[0.603s] [ext: omni.kit.async_engine-0.0.0] startup
[0.604s] [ext: omni.kit.test-0.0.0] startup
[0.692s] [ext: omni.usd.config-1.0.0] startup
[0.695s] [ext: omni.usd.libs-1.0.0] startup
[0.803s] [ext: omni.isaac.core_archive-2.0.1] startup
[1.039s] [ext: omni.pip.torch-1_13_1-0.1.4] startup
[1.041s] [ext: omni.isaac.ml_archive-1.1.0] startup
[1.094s] [ext: omni.client-0.1.1] startup
[1.102s] [ext: omni.appwindow-1.0.1] startup
[1.104s] [ext: omni.kit.renderer.core-0.0.0] startup
[1.133s] [ext: omni.kit.renderer.capture-0.0.0] startup
[1.135s] [ext: omni.kit.renderer.imgui-0.0.0] startup
[1.186s] [ext: carb.audio-0.1.0] startup
[1.207s] [ext: omni.ui-2.14.4] startup
[1.216s] [ext: omni.uiaudio-1.0.0] startup
[1.217s] [ext: omni.kit.mainwindow-1.0.0] startup
[1.218s] [ext: omni.kit.uiapp-0.0.0] startup
[1.218s] [ext: omni.usd.schema.physics-1.0.0] startup
[1.248s] [ext: omni.usd.schema.isaac-1.1.0] startup
[1.260s] [ext: omni.usd.schema.semantics-0.0.0] startup
[1.267s] [ext: omni.usd.schema.omnigraph-1.0.0] startup
[1.272s] [ext: omni.usd.schema.forcefield-0.0.0] startup
[1.277s] [ext: omni.usd.schema.omniscripting-1.0.0] startup
[1.282s] [ext: omni.usd.schema.geospatial-0.0.0] startup
[1.287s] [ext: omni.usd.schema.anim-0.0.0] startup
[1.330s] [ext: omni.usd.schema.audio-0.0.0] startup
[1.334s] [ext: omni.usd.schema.scene.visualization-1.0.0] startup
[1.335s] [ext: omni.usd.schema.physx-0.0.0] startup
[1.354s] [ext: omni.gpucompute.plugins-0.0.0] startup
[1.355s] [ext: omni.timeline-1.0.5] startup
[1.355s] [ext: omni.kit.search_core-1.0.2] startup
[1.357s] [ext: omni.hydra.scene_delegate-0.3.0] startup
[1.360s] [ext: omni.kit.widget.path_field-2.0.4] startup
[1.364s] [ext: omni.kit.widget.filebrowser-2.3.10] startup
[1.397s] [ext: omni.kit.widget.browser_bar-2.0.5] startup
[1.398s] [ext: omni.kit.window.popup_dialog-2.0.16] startup
[1.404s] [ext: omni.kit.actions.core-1.0.0] startup
[1.406s] [ext: omni.usd.core-1.0.4] startup
[1.407s] [ext: omni.kit.audiodeviceenum-1.0.0] startup
[1.408s] [ext: omni.kit.widget.nucleus_connector-1.0.3] startup
[1.411s] [ext: omni.kit.commands-1.4.5] startup
[1.417s] [ext: omni.hydra.usdrt_delegate-4.3.2] startup
[1.424s] [ext: omni.kit.notification_manager-1.0.5] startup
[1.429s] [ext: omni.usd-1.6.30] startup
[1.477s] [ext: omni.kit.widget.versioning-1.3.8] startup
[1.482s] [ext: omni.iray.libs-0.0.0] startup
[1.485s] [ext: omni.kit.window.filepicker-2.7.15] startup
[1.531s] [ext: omni.mdl.neuraylib-0.1.0] startup
[1.533s] [ext: omni.kit.collaboration.channel_manager-1.0.9] startup
[1.536s] [ext: omni.kit.window.file_importer-1.0.10] startup
[1.538s] [ext: omni.mdl-0.1.0] startup
[1.628s] [ext: omni.kit.usd.layers-2.0.11] startup
[1.642s] [ext: omni.kit.menu.utils-1.4.7] startup
[1.656s] [ext: omni.kit.material.library-1.3.21] startup
[1.664s] [ext: omni.kit.menu.create-1.0.8] startup
[1.666s] [ext: omni.kit.clipboard-1.0.0] startup
[1.666s] [ext: omni.kit.widget.graph-1.5.6-104_2] startup
[1.752s] [ext: omni.kit.context_menu-1.5.12] startup
[1.762s] [ext: omni.kit.window.extensions-1.1.1] startup
[1.780s] [ext: omni.inspect-1.0.1] startup
[1.782s] [ext: omni.kit.window.property-1.8.2] startup
[1.785s] [ext: omni.kit.primitive.mesh-1.0.8] startup
[1.795s] [ext: omni.graph.core-2.65.4] startup
[1.796s] [ext: omni.isaac.version-1.0.0] startup
[1.797s] [ext: omni.kit.stage_templates-1.1.13] startup
[1.800s] [ext: omni.kit.usd_undo-0.1.2] startup
[1.801s] [ext: omni.graph.tools-1.17.2] startup
[1.844s] [ext: omni.ui.scene-1.5.18] startup
[1.848s] [ext: omni.graph-1.50.2] startup
[1.936s] [ext: omni.ui_query-1.1.1] startup
[1.937s] [ext: omni.volume-0.1.0] startup
[1.939s] [ext: omni.kit.ui_test-1.2.9] startup
[1.943s] [ext: omni.hydra.rtx-0.1.0] startup
[1.946s] [ext: omni.kit.widget.searchfield-1.0.10] startup
[1.948s] [ext: omni.kit.graph.delegate.default-1.0.18] startup
[1.950s] [ext: omni.debugdraw-0.1.1] startup
[1.953s] [ext: omni.kit.graph.editor.core-1.4.7] startup
[1.960s] [ext: omni.hydra.pxr-1.1.2] startup
[1.963s] [ext: omni.activity.core-1.0.1] startup
[1.965s] [ext: omni.kit.hydra_texture-1.0.11] startup
[1.968s] [ext: omni.kit.viewport.legacy_gizmos-1.0.10] startup
[1.971s] [ext: omni.kit.viewport.registry-104.0.2] startup
[1.972s] [ext: omni.kit.widget.viewport-104.1.16] startup
[1.982s] [ext: omni.kit.window.drop_support-1.0.1] startup
[1.983s] [ext: omni.kit.window.file_exporter-1.0.10] startup
[1.984s] [ext: omni.hydra.engine.stats-1.0.0] startup
[1.986s] [ext: omni.kit.window.content_browser_registry-0.0.1] startup
[1.987s] [ext: omni.kit.window.file-1.3.32] startup
[1.995s] [ext: omni.kit.viewport.window-104.1.23] startup
[2.035s] [ext: omni.kit.widget.settings-1.0.1] startup
[2.040s] [ext: omni.kit.widget.stage-2.7.24] startup
[2.055s] [ext: omni.kit.window.content_browser-2.6.8] startup
[2.086s] [ext: omni.kit.viewport.utility-1.0.14] startup
[2.089s] [ext: omni.kit.window.preferences-1.3.8] startup
[2.118s] [ext: omni.kit.widget.text_editor-1.0.2] startup
[2.119s] [ext: omni.kit.widget.prompt-1.0.5] startup
[2.121s] [ext: omni.kit.property.usd-3.18.17] startup
[2.152s] [ext: omni.kit.graph.delegate.modern-1.6.5] startup
[2.158s] [ext: omni.graph.ui-1.24.2] startup
[2.240s] [ext: omni.physics.tensors-0.1.0] startup
[2.249s] [ext: omni.kit.graph.widget.variables-2.0.4] startup
[2.252s] [ext: omni.kit.graph.usd.commands-1.2.0] startup
[2.254s] [ext: omni.convexdecomposition-104.2.4-5.1] startup
[2.255s] [ext: omni.graph.window.core-1.43.3] startup
[2.283s] [ext: omni.kit.widget.zoombar-1.0.4] startup
[2.284s] [ext: omni.kvdb-0.0.0] startup
[2.286s] [ext: omni.usdphysics-104.2.4-5.1] startup
[2.287s] [ext: omni.localcache-0.0.0] startup
[2.288s] [ext: omni.graph.scriptnode-0.10.0] startup
[2.292s] [ext: omni.physx-104.2.4-5.1] startup
[2.406s] [ext: omni.command.usd-1.0.2] startup
[2.414s] [ext: omni.physx.tensors-0.1.0] startup
[2.417s] [ext: omni.graph.window.generic-1.3.15] startup
[2.419s] [ext: omni.kit.browser.core-2.2.2] startup
[2.429s] [ext: omni.graph.action-1.31.1] startup
[2.439s] [ext: omni.kit.browser.folder.core-1.7.3] startup
[2.448s] [ext: omni.kit.window.cursor-1.1.1] startup
[2.449s] [ext: omni.kit.widget.spinner-1.0.4] startup
[2.450s] [ext: omni.kit.browser.sample-1.2.5] startup
[2.452s] [ext: omni.graph.nodes-1.48.3] startup
[2.494s] [ext: omni.kit.viewport.menubar.core-104.4.3] startup
[2.527s] [ext: omni.kit.numpy.common-0.1.0] startup
[2.529s] [ext: omni.warp-0.6.3] startup
Warp 0.6.3 initialized:
   CUDA Toolkit: 11.5, Driver: 12.1
   Devices:
     "cpu"    | x86_64
     "cuda:0" | NVIDIA GeForce RTX 4090 (sm_89)
   Kernel cache: /root/.cache/warp/0.6.3
WARNING: Attribute 'inputs:execIn' description should not be empty
WARNING: Attribute 'inputs:sdf_min' description should not be empty
WARNING: Attribute 'inputs:sdf_max' description should not be empty
WARNING: Attribute 'inputs:spacing' description should not be empty
WARNING: Attribute 'inputs:spacing_jitter' description should not be empty
WARNING: Attribute 'inputs:velocity' description should not be empty
WARNING: Attribute 'inputs:max_points' description should not be empty
WARNING: Attribute 'inputs:mass' description should not be empty
WARNING: Attribute 'inputs:radius' description should not be empty
WARNING: Attribute 'inputs:k_contact_elastic' description should not be empty
WARNING: Attribute 'inputs:k_contact_damp' description should not be empty
WARNING: Attribute 'inputs:k_contact_friction' description should not be empty
WARNING: Attribute 'inputs:k_contact_mu' description should not be empty
WARNING: Attribute 'inputs:k_contact_cohesion' description should not be empty
WARNING: Attribute 'inputs:k_contact_adhesion' description should not be empty
WARNING: Attribute 'inputs:num_substeps' description should not be empty
WARNING: Attribute 'inputs:gravity' description should not be empty
WARNING: Attribute 'inputs:ground' description should not be empty
WARNING: Attribute 'inputs:ground_plane' description should not be empty
WARNING: Attribute 'inputs:asynchronous' description should not be empty
WARNING: Attribute 'inputs:fps' description should not be empty
WARNING: Attribute 'inputs:start' description should not be empty
WARNING: Attribute 'inputs:end' description should not be empty
WARNING: Attribute 'outputs:time' description should not be empty
WARNING: Node description should not be empty
WARNING: Attribute 'inputs:resolution' description should not be empty
WARNING: Attribute 'inputs:speed' description should not be empty
WARNING: Attribute 'inputs:damp' description should not be empty
WARNING: Attribute 'inputs:grid' description should not be empty
WARNING: Attribute 'inputs:collider_0' description should not be empty
WARNING: Attribute 'inputs:collider_1' description should not be empty
WARNING: Attribute 'inputs:collider_2' description should not be empty
WARNING: Attribute 'inputs:collider_3' description should not be empty
WARNING: Attribute 'inputs:buoyancy_enabled' description should not be empty
WARNING: Attribute 'inputs:density_0' description should not be empty
WARNING: Attribute 'inputs:density_1' description should not be empty
WARNING: Attribute 'inputs:density_2' description should not be empty
WARNING: Attribute 'inputs:density_3' description should not be empty
WARNING: Attribute 'inputs:displace' description should not be empty
WARNING: Attribute 'inputs:gravity' description should not be empty
WARNING: Attribute 'inputs:buoyancy' description should not be empty
WARNING: Attribute 'inputs:buoyancy_damp' description should not be empty
WARNING: Attribute 'inputs:delay' description should not be empty
WARNING: Attribute 'outputs:vertices' description should not be empty
WARNING: Attribute 'outputs:face_indices' description should not be empty
WARNING: Attribute 'outputs:face_counts' description should not be empty
WARNING: Attribute 'inputs:collider_offset' description should not be empty
WARNING: Attribute 'inputs:ground' description should not be empty
WARNING: Attribute 'inputs:ground_plane' description should not be empty
WARNING: Attribute 'inputs:density' description should not be empty
WARNING: Attribute 'inputs:k_tri_elastic' description should not be empty
WARNING: Attribute 'inputs:k_tri_area' description should not be empty
WARNING: Attribute 'inputs:k_tri_damp' description should not be empty
WARNING: Attribute 'inputs:k_edge_bend' description should not be empty
WARNING: Attribute 'inputs:k_edge_damp' description should not be empty
WARNING: Attribute 'inputs:k_contact_elastic' description should not be empty
WARNING: Attribute 'inputs:k_contact_damp' description should not be empty
WARNING: Attribute 'inputs:k_contact_friction' description should not be empty
WARNING: Attribute 'inputs:k_contact_mu' description should not be empty
WARNING: Attribute 'inputs:num_substeps' description should not be empty
WARNING: Attribute 'inputs:gravity' description should not be empty
[2.757s] [ext: omni.syntheticdata-0.2.4] startup
2023-04-12 03:42:44 [2,779ms] [Warning] [omni.kit.viewport.menubar.core.menu_item.viewport_menu_item] ViewportMenuItem order argument is deprecated, use order
_setting_path
[2.791s] [ext: omni.isaac.dynamic_control-1.2.3] startup
[2.804s] [ext: omni.replicator.core-1.7.8] startup
2023-04-12 03:42:45 [3,024ms] [Warning] [omni.replicator.core.scripts.annotators] Annotator PostProcessDispatch is already registered, overwriting annotator t
emplate
[3.248s] [ext: omni.isaac.core-1.46.3] startup
[4.207s] [ext: omni.kit.manipulator.viewport-104.0.7] startup
[4.210s] [ext: omni.isaac.core_nodes-0.23.2] startup
[4.234s] [ext: omni.isaac.cloner-0.4.1] startup
[4.236s] [ext: omni.isaac.lula-1.2.1] startup
[4.243s] [ext: omni.graph.visualization.nodes-2.0.0] startup
[4.252s] [ext: omni.isaac.debug_draw-0.2.3] startup
[4.255s] [ext: omni.isaac.gym-0.3.3] startup
[4.255s] [ext: omni.graph.window.action-1.5.5] startup
[4.257s] [ext: omni.isaac.ui-0.8.0] startup
[4.268s] [ext: usdrt.scenegraph-4.3.2] startup
[4.280s] [ext: omni.isaac.motion_planning-0.2.2] startup
[4.285s] [ext: omni.isaac.surface_gripper-0.4.0] startup
[4.327s] [ext: omni.isaac.range_sensor-0.8.1] startup
[4.375s] [ext: omni.isaac.motion_generation-4.5.6] startup
[4.394s] [ext: omni.isaac.manipulators-1.1.0] startup
[4.397s] [ext: omni.isaac.sensor-5.6.3] startup
2023-04-12 03:42:46 [4,392ms] [Warning] [omni.ext.plugin] [ext: omni.isaac.sensor-5.6.3] Native plugin search dir contains several files: /isaac-sim/exts/omni
.isaac.sensor/bin . It makes dir walk slow, replace with the exact path to the plugin instead of a wildcard.
[4.439s] [ext: omni.isaac.dofbot-0.3.0] startup
[4.440s] [ext: omni.isaac.occupancy_map-0.2.7] startup
[4.452s] [ext: omni.isaac.quadruped-1.3.0] startup
[5.040s] [ext: omni.isaac.scene_blox-0.1.0] startup
[5.040s] [ext: omni.isaac.synthetic_utils-0.4.3] startup
[5.046s] [ext: omni.isaac.universal_robots-0.3.2] startup
[5.047s] [ext: omni.isaac.isaac_sensor-2.0.0] startup
2023-04-12 03:42:47 [5,042ms] [Warning] [omni.isaac.isaac_sensor] omni.isaac.isaac_sensor is deprecated and was renamed; use omni.isaac.sensor
[5.048s] [ext: omni.kit.widget.live_session_management-1.1.3] startup
[5.054s] [ext: omni.kit.hotkeys.core-1.0.8] startup
[5.072s] [ext: omni.isaac.wheeled_robots-0.6.3] startup
[5.127s] [ext: omni.kit.widget.stage_icons-1.0.2] startup
[5.129s] [ext: omni.kit.widget.layers-1.6.11] startup
[5.158s] [ext: omni.kit.selection-0.1.0] startup
[5.160s] [ext: omni.isaac.mjcf-0.3.1] startup
[5.167s] [ext: omni.kit.window.stage-2.3.12] startup
[5.170s] [ext: omni.kit.property.layer-1.1.4] startup
[5.174s] [ext: omni.kit.menu.edit-1.1.6] startup
[5.179s] [ext: omni.kit.menu.file-1.1.4] startup
[5.181s] [ext: omni.kit.menu.stage-1.2.1] startup
[5.182s] [ext: omni.kit.property.audio-1.0.6] startup
[5.185s] [ext: omni.kit.property.camera-1.0.3] startup
[5.187s] [ext: omni.kit.property.geometry-1.2.2] startup
[5.191s] [ext: omni.kit.property.light-1.0.6] startup
[5.193s] [ext: omni.hydra.scene_api-0.1.1] startup
[5.196s] [ext: omni.kit.property.material-1.8.19] startup
[5.203s] [ext: omni.kit.property.render-1.1.0] startup
[5.205s] [ext: omni.kit.property.skel-1.0.1] startup
[5.207s] [ext: omni.kit.property.transform-1.3.5] startup
[5.218s] [ext: omni.kit.manipulator.transform-104.6.12] startup
[5.220s] [ext: omni.kit.widget.toolbar-1.4.0] startup
[5.230s] [ext: omni.kit.profiler.window-2.0.5] startup
[5.354s] [ext: omni.kit.property.bundle-1.2.6] startup
[5.356s] [ext: omni.kit.manipulator.selector-1.0.1] startup
[5.358s] [ext: omni.kit.manipulator.tool.snap-1.2.4] startup
[5.373s] [ext: omni.kit.window.toolbar-1.4.0] startup
[5.375s] [ext: omni.kit.property.isaac-0.2.0] startup
[5.378s] [ext: omni.kit.stage_column.payload-1.0.8] startup
[5.381s] [ext: omni.kit.manipulator.camera-104.0.23] startup
[5.381s] [ext: omni.kit.manipulator.prim-104.10.7] startup
[5.393s] [ext: omni.kit.manipulator.selection-104.0.6] startup
[5.394s] [ext: omni.kit.viewport.menubar.display-104.1.11] startup
[5.397s] [ext: omni.kit.viewport.menubar.camera-104.1.9] startup
[5.406s] [ext: omni.rtx.window.settings-0.6.2] startup
[5.413s] [ext: omni.kit.viewport.menubar.settings-104.3.5] startup
[5.423s] [ext: omni.kit.viewport.menubar.render-104.0.20] startup
[5.428s] [ext: omni.physx.commands-104.2.4-5.1] startup
[5.440s] [ext: omni.kit.stage_column.variant-1.0.9] startup
[5.442s] [ext: omni.hydra.pxr.settings-1.0.3] startup
[5.444s] [ext: omni.kit.viewport.bundle-104.0.2] startup
[5.444s] [ext: omni.physx.ui-104.2.4-5.1] startup
[5.495s] [ext: omni.kit.viewport.menubar.lighting-104.0.9] startup
[5.501s] [ext: omni.kit.viewport.pxr-104.0.1] startup
[5.501s] [ext: omni.kit.viewport.ready-1.0.2] startup
[5.502s] [ext: omni.physx.demos-104.2.4-5.1] startup
[5.508s] [ext: omni.kit.property.physx-0.1.0] startup
[5.646s] [ext: omni.isaac.urdf-0.5.9] startup
[5.669s] [ext: omni.kit.widget.highlight_label-1.0.0] startup
[5.670s] [ext: omni.physx.supportui-104.2.4-5.1] startup
[5.688s] [ext: omni.physx.vehicle-104.2.4-5.1] startup
[5.712s] [ext: omni.physx.cct-104.2.4-5.1] startup
[5.720s] [ext: omni.physx.camera-104.2.4-5.1] startup
[5.726s] [ext: omni.kit.actions.window-1.1.0] startup
[5.737s] [ext: omni.kit.window.commands-0.2.4] startup
[5.740s] [ext: omni.kit.window.console-0.2.3] startup
[5.746s] [ext: omni.physx.bundle-104.2.4-5.1] startup
[5.746s] [ext: omni.graph.bundle.action-1.3.0] startup
[5.746s] [ext: omni.kit.window.script_editor-1.7.2] startup
[5.748s] [ext: omni.kit.hotkeys.window-1.1.3] startup
[5.761s] [ext: omni.kit.window.status_bar-0.1.5] startup
[5.763s] [ext: omni.kit.window.title-1.1.2] startup
[5.764s] [ext: omni.replicator.composer-1.2.10] startup
[5.776s] [ext: omni.replicator.isaac-1.7.4] startup
[5.818s] [ext: omni.resourcemonitor-1.0.0] startup
[5.822s] [ext: omni.rtx.settings.core-0.5.8] startup
[5.830s] [ext: omni.isaac.franka-0.4.0] startup
[5.832s] [ext: omni.kit.viewport.rtx-104.0.0] startup
[5.832s] [ext: semantics.schema.editor-0.3.3] startup
[5.836s] [ext: omni.kit.widget.live-2.0.3] startup
2023-04-12 03:42:48 [5,834ms] [Warning] [omni.kit.widget.live.cache_state_menu] Unable to detect Omniverse Cache Server. Consider installing it for better IO 
performance.
[5.840s] [ext: omni.isaac.utils-0.2.4] startup
[5.844s] [ext: omni.isaac.kit-1.4.1] startup
[5.844s] [ext: omni.isaac.cortex-0.3.2] startup
[5.845s] [ext: omni.kit.window.stats-0.1.2] startup
[5.846s] [ext: omni.isaac.sim.python-2022.2.1] startup
[5.847s] Simulation App Starting
[7.212s] [ext: omni.isaac.sim.python-2022.2.1] shutdown
[7.212s] [ext: omni.isaac.universal_robots-0.3.2] shutdown
[7.301s] [ext: omni.isaac.cortex-0.3.2] shutdown
[7.388s] [ext: omni.isaac.franka-0.4.0] shutdown
[7.475s] [ext: omni.isaac.dofbot-0.3.0] shutdown
[7.562s] [ext: omni.isaac.manipulators-1.1.0] shutdown
[7.648s] [ext: omni.isaac.surface_gripper-0.4.0] shutdown
2023-04-12 03:42:49 [7,756ms] [Warning] [omni.ext._impl._internal] omni.isaac.surface_gripper-0.4.0 -> <class 'omni.isaac.surface_gripper.scripts.extension.Ex
tension'>: extension object is still alive, something holds a reference on it. References: ["[0]:type: <class 'method'>, id: 140247690004592"]
[7.850s] [ext: omni.isaac.wheeled_robots-0.6.3] shutdown
[7.943s] [ext: omni.isaac.surface_gripper-0.4.0] startup
[7.949s] [ext: omni.isaac.manipulators-1.1.0] startup
[7.950s] [ext: omni.isaac.dofbot-0.3.0] startup
[7.951s] [ext: omni.isaac.universal_robots-0.3.2] startup
[7.952s] [ext: omni.isaac.wheeled_robots-0.6.3] startup
[7.955s] [ext: omni.isaac.franka-0.4.0] startup
[7.956s] [ext: omni.isaac.cortex-0.3.2] startup
[7.956s] [ext: omni.isaac.sim.python-2022.2.1] startup
2023-04-12 03:43:23 [40,862ms] [Warning] [gpu.foundation.plugin] Waiting for compilation of ray tracing shaders by GPU driver: 30 seconds so far
2023-04-12 03:43:53 [70,862ms] [Warning] [gpu.foundation.plugin] Waiting for compilation of ray tracing shaders by GPU driver: 60 seconds so far
2023-04-12 03:44:23 [100,862ms] [Warning] [gpu.foundation.plugin] Waiting for compilation of ray tracing shaders by GPU driver: 90 seconds so far
2023-04-12 03:44:53 [130,862ms] [Warning] [gpu.foundation.plugin] Waiting for compilation of ray tracing shaders by GPU driver: 120 seconds so far
2023-04-12 03:45:23 [160,862ms] [Warning] [gpu.foundation.plugin] Waiting for compilation of ray tracing shaders by GPU driver: 150 seconds so far
2023-04-12 03:45:46 [184,446ms] [Warning] [gpu.foundation.plugin] Ray tracing shader compilation finished after 173 seconds
[184.622s] app ready
[184.867s] RTX ready
[185.002s] Simulation App Startup Complete
[185.051s] Simulation App Shutting Down
sha256:a35052eda7eba07550850b6f0e47fe39d6fcab4854e09f5954945f6f3fb23c8e
tmp

BUILDING BENCHBOT SUBMISSION DOCKER IMAGE:
#1 [internal] load .dockerignore
#1 transferring context: 52B done
#1 DONE 0.0s

#2 [internal] load build definition from submission.Dockerfile
#2 transferring dockerfile: 1.28kB done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/benchbot/core:base
#3 DONE 0.0s

#4 [1/4] FROM docker.io/benchbot/core:base
#4 DONE 0.0s

#5 [2/4] RUN apt update && apt install -y libsm6 libxext6 libxrender-dev python3     python3-pip python3-tk
#5 CACHED

#6 [3/4] RUN git clone https://github.com/qcr/benchbot_api && pushd benchbot_api &&     git checkout b8422ccb152332125b4ecb9ed253338fb495dd2d && pip3 install .
#6 CACHED
#7 [4/4] WORKDIR /benchbot_submission
#7 CACHED

#8 exporting to image
#8 exporting layers done
#8 writing image sha256:297ad9d5df02952fae439c0e43ea36f1632e3eb153845f37eeb32624090de00d done
#8 naming to docker.io/benchbot/submission:base done
#8 DONE 0.0s

CLEANING UP OUTDATED BENCHBOT REMNANTS:

Deleted the following containers:
Deleted Images:
deleted: sha256:cea0df35acf816073a08294547dbe6ab073ed20ca1b15ba7e6a5637fc0e5c7d5
deleted: sha256:8a730417cfc46254e116d9e60b064a5a123f14fd295492bd7de89a24111d9059
deleted: sha256:cc04a1c129e5cc1f6b7496eca1536c00864ce7fd01f7521bd9a50d9c5579c110
deleted: sha256:dcb801a2e2eb29d8d7e9424973ef166806fb780305138d393ec848aab2370a80
deleted: sha256:b3ea5c256c30ee03e18746c5b9e048d029dc8fcb6928f5e2aea670712b35ae0b
deleted: sha256:103696436975b480d2dd1e93602bb29a5df5c0281b8193ae0dbef602e8a51695
deleted: sha256:d539f2d2f655b6d933697aded420fbe9870194c20f7ced4b8ac47b4f83359edd

Deleted build cache objects:
dnmgd2toy389nbrg4jccanm3s
vl4st9ldinyxrbvpid2qqcp42

Total reclaimed space: 793MB

Finished cleaning!

################################################################################
#################### PART 4: RUNNING POST-BUILD HOST CHECKS ####################
################################################################################

Validating the build against the host system:
    CUDA / NVIDIA versions match:                                    Matches

Validating BenchBot libraries on the host system:
    BenchBot Add-ons Manager cloned:                                     Yes
    BenchBot Add-ons Manager up-to-date:                          Up-to-date
    BenchBot Add-ons Manager installed:                            Available
    BenchBot API cloned:                                                 Yes
    BenchBot API up-to-date:                                      Up-to-date
    BenchBot API installed:                                        Available
    BenchBot evaluation cloned:                                          Yes
    BenchBot evaluation up-to-date:                               Up-to-date
    BenchBot evaluation installed:                                 Available

Integrating BenchBot with the host system:
    BenchBot hosts available:                                          Found
    BenchBot symlinks available:                                       Found

################################################################################
##################### PART 5: INSTALLING BENCHBOT ADD-ONS ######################
################################################################################

Configuring locations for local add-on content:

Created '/home/rp5/benchbot/addons/benchbot_addons/.local/my_addons/{batches,environments,evaluation_methods,examples,formats,ground_truths,robots,tasks}'.

Installing add-ons based on the request string 'benchbot-addons/ssu':

/usr/bin/unzip
Installing addon 'benchbot-addons/ssu' in '/home/rp5/benchbot/addons/benchbot_addons':
    Found install path './benchbot-addons/ssu'.
    No action - latest already installed.
Installing addon 'benchbot-addons/tasks_ssu' in '/home/rp5/benchbot/addons/benchbot_addons':
    Found install path './benchbot-addons/tasks_ssu'.
    No action - latest already installed.
Installing addon 'benchbot-addons/formats_object_map' in '/home/rp5/benchbot/addons/benchbot_addons':
    Found install path './benchbot-addons/formats_object_map'.
    No action - latest already installed.
Installing addon 'benchbot-addons/envs_bear_develop_sim_omni' in '/home/rp5/benchbot/addons/benchbot_addons':
    Found install path './benchbot-addons/envs_bear_develop_sim_omni'.
    No action - latest already installed.
Found remote content to install to 'environments': https://drive.google.com/file/d/1jGJKyg-adoIxpW4y5MRXLVjREfesV68-/view?usp=share_link
    No action - remote content is already installed.
Installing addon 'benchbot-addons/robots_sim_omni' in '/home/rp5/benchbot/addons/benchbot_addons':
    Found install path './benchbot-addons/robots_sim_omni'.
    No action - latest already installed.
    Found remote content to install to 'robots': https://drive.google.com/file/d/1mAvuJocmiOxgg-CwCppxyfJyuCusutJd/view?usp=share_link
    No action - remote content is already installed.
Installing addon 'benchbot-addons/envs_bear_challenge_sim_omni' in '/home/rp5/benchbot/addons/benchbot_addons':
    Found install path './benchbot-addons/envs_bear_challenge_sim_omni'.
    No action - latest already installed.
    Found remote content to install to 'environments': https://drive.google.com/file/d/1cZ-h6w1EuD9J6Qh7QLk1LUJf1omVCHqx/view?usp=share_link
    No action - remote content is already installed.
Installing addon 'benchbot-addons/robots_sim_omni' in '/home/rp5/benchbot/addons/benchbot_addons':
    Found install path './benchbot-addons/robots_sim_omni'.
    No action - latest already installed.
    Found remote content to install to 'robots': https://drive.google.com/file/d/1mAvuJocmiOxgg-CwCppxyfJyuCusutJd/view?usp=share_link
    No action - remote content is already installed.
Installing addon 'benchbot-addons/robots_sim_omni' in '/home/rp5/benchbot/addons/benchbot_addons':
    Found install path './benchbot-addons/robots_sim_omni'.
    No action - latest already installed.
    Found remote content to install to 'robots': https://drive.google.com/file/d/1mAvuJocmiOxgg-CwCppxyfJyuCusutJd/view?usp=share_link
    No action - remote content is already installed.
Installing addon 'benchbot-addons/ground_truths_bear_develop' in '/home/rp5/benchbot/addons/benchbot_addons':
    Found install path './benchbot-addons/ground_truths_bear_develop'.
    No action - latest already installed.
Installing addon 'benchbot-addons/batches_bear' in '/home/rp5/benchbot/addons/benchbot_addons':
    Found install path './benchbot-addons/batches_bear'.
    No action - latest already installed.
Installing addon 'benchbot-addons/eval_omq' in '/home/rp5/benchbot/addons/benchbot_addons':
    Found install path './benchbot-addons/eval_omq'.
    No action - latest already installed.
Installing addon 'benchbot-addons/formats_object_map' in '/home/rp5/benchbot/addons/benchbot_addons':
    Found install path './benchbot-addons/formats_object_map'.
    No action - latest already installed.
Installing addon 'benchbot-addons/examples_ssu' in '/home/rp5/benchbot/addons/benchbot_addons':
    Found install path './benchbot-addons/examples_ssu'.
    No action - latest already installed.
Installing addon 'benchbot-addons/examples_base' in '/home/rp5/benchbot/addons/benchbot_addons':
    Found install path './benchbot-addons/examples_base'.
    No action - latest already installed.

Installing external add-on dependencies:

Requirement already satisfied: matplotlib in /home/rp5/.local/lib/python3.8/site-packages (3.7.1)
Requirement already satisfied: numpy in /home/rp5/.local/lib/python3.8/site-packages (1.24.2)
Requirement already satisfied: scipy in /home/rp5/.local/lib/python3.8/site-packages (1.10.1)
Requirement already satisfied: shapely in /home/rp5/.local/lib/python3.8/site-packages (2.0.1)
Requirement already satisfied: cycler>=0.10 in /home/rp5/.local/lib/python3.8/site-packages (from matplotlib) (0.11.0)
Requirement already satisfied: packaging>=20.0 in /home/rp5/.local/lib/python3.8/site-packages (from matplotlib) (23.0)
Requirement already satisfied: fonttools>=4.22.0 in /home/rp5/.local/lib/python3.8/site-packages (from matplotlib) (4.39.3)
Requirement already satisfied: contourpy>=1.0.1 in /home/rp5/.local/lib/python3.8/site-packages (from matplotlib) (1.0.7)
Requirement already satisfied: importlib-resources>=3.2.0; python_version < "3.10" in /home/rp5/.local/lib/python3.8/site-packages (from matplotlib) (5.12.0)
Requirement already satisfied: python-dateutil>=2.7 in /usr/lib/python3/dist-packages (from matplotlib) (2.7.3)
Requirement already satisfied: pyparsing>=2.3.1 in /home/rp5/.local/lib/python3.8/site-packages (from matplotlib) (3.0.9)
Requirement already satisfied: pillow>=6.2.0 in /usr/lib/python3/dist-packages (from matplotlib) (7.0.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /home/rp5/.local/lib/python3.8/site-packages (from matplotlib) (1.4.4)
Requirement already satisfied: zipp>=3.1.0; python_version < "3.10" in /home/rp5/.local/lib/python3.8/site-packages (from importlib-resources>=3.2.0; python_version <
 "3.10"->matplotlib) (3.15.0)
Running the following pip install command:
    pip3 install matplotlib numpy scipy shapely

    Done.

Baking external add-on dependencies into 'benchbot/simulator:sim_omni' container:

d48ac5bf4c9ef35975220dea440c7f34753979d078156fd39dcaac83924440d6
Requirement already satisfied: numpy in /usr/lib/python3/dist-packages (1.17.4)
Collecting matplotlib
  Downloading matplotlib-3.7.1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (9.2 MB)

Collecting shapely
  Downloading shapely-2.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB)

Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (1.10.1)
Collecting fonttools>=4.22.0
  Downloading fonttools-4.39.3-py3-none-any.whl (1.0 MB)

Collecting numpy
  Downloading numpy-1.24.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)

Collecting packaging>=20.0
  Using cached packaging-23.0-py3-none-any.whl (42 kB)
Collecting importlib-resources>=3.2.0
  Downloading importlib_resources-5.12.0-py3-none-any.whl (36 kB)
Requirement already satisfied: pillow>=6.2.0 in /usr/lib/python3/dist-packages (from matplotlib) (7.0.0)
Requirement already satisfied: pyparsing>=2.3.1 in /usr/lib/python3/dist-packages (from matplotlib) (2.4.6)
Collecting cycler>=0.10
  Using cached cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting kiwisolver>=1.0.1
  Downloading kiwisolver-1.4.4-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.2 MB)

Collecting contourpy>=1.0.1
  Downloading contourpy-1.0.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (300 kB)

Requirement already satisfied: python-dateutil>=2.7 in /usr/lib/python3/dist-packages (from matplotlib) (2.7.3)
Collecting zipp>=3.1.0
  Using cached zipp-3.15.0-py3-none-any.whl (6.8 kB)
Installing collected packages: zipp, packaging, numpy, kiwisolver, fonttools, cycler, shapely, importlib-resources, contourpy, matplotlib
  Attempting uninstall: zipp
    Found existing installation: zipp 1.0.0
    Uninstalling zipp-1.0.0:
      Successfully uninstalled zipp-1.0.0
  Attempting uninstall: numpy
    Found existing installation: numpy 1.17.4
    Uninstalling numpy-1.17.4:
      Successfully uninstalled numpy-1.17.4
Successfully installed contourpy-1.0.7 cycler-0.11.0 fonttools-4.39.3 importlib-resources-5.12.0 kiwisolver-1.4.4 matplotlib-3.7.1 numpy-1.24.2 packaging-23.0 shapely
-2.0.1 zipp-3.15.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a vir
tual environment instead: https://pip.pypa.io/warnings/venv
sha256:de8ac19aa9c44989a8bb5846fee85cca28de2565ec29660cad45df38ca577d20
tmp

Baking external add-on dependencies into 'benchbot/backend:base' container:

2332d94b822c2a8411383dfedbe79d084a57bcb6495a63650c7236124f8e263f
Collecting matplotlib
  Downloading matplotlib-3.7.1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (9.2 MB)

Collecting shapely
  Downloading shapely-2.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB)

Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (1.10.1)
Requirement already satisfied: numpy in /usr/lib/python3/dist-packages (1.17.4)
Collecting fonttools>=4.22.0
  Downloading fonttools-4.39.3-py3-none-any.whl (1.0 MB)

Collecting importlib-resources>=3.2.0
  Downloading importlib_resources-5.12.0-py3-none-any.whl (36 kB)
Collecting contourpy>=1.0.1
  Downloading contourpy-1.0.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (300 kB)

Requirement already satisfied: pyparsing>=2.3.1 in /usr/lib/python3/dist-packages (from matplotlib) (2.4.6)
Collecting numpy
  Downloading numpy-1.24.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)

Collecting cycler>=0.10
  Downloading cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting packaging>=20.0
  Downloading packaging-23.0-py3-none-any.whl (42 kB)

Requirement already satisfied: pillow>=6.2.0 in /usr/lib/python3/dist-packages (from matplotlib) (7.0.0)
Requirement already satisfied: python-dateutil>=2.7 in /usr/lib/python3/dist-packages (from matplotlib) (2.7.3)
Collecting kiwisolver>=1.0.1
  Downloading kiwisolver-1.4.4-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.2 MB)

Requirement already satisfied: zipp>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=3.2.0->matplotlib) (3.15.0)
Installing collected packages: packaging, numpy, kiwisolver, importlib-resources, fonttools, cycler, shapely, contourpy, matplotlib
  Attempting uninstall: numpy
    Found existing installation: numpy 1.17.4
    Uninstalling numpy-1.17.4:
      Successfully uninstalled numpy-1.17.4
Successfully installed contourpy-1.0.7 cycler-0.11.0 fonttools-4.39.3 importlib-resources-5.12.0 kiwisolver-1.4.4 matplotlib-3.7.1 numpy-1.24.2 packaging-23.0 shapely
-2.0.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
sha256:ec41d72a577ebbf847562143854ba9d5d5c0acc96506fdbf8a6daa2a953f885b
tmp

Finished!

However, vulkan.log is not in text format

The output of nvidia-smi is shown below. 

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.30.02              Driver Version: 530.30.02    CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 4090         On | 00000000:01:00.0 Off |                  Off |
| 30%   34C    P8               38W / 450W|     12MiB / 24564MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      1202      G   /usr/lib/xorg/Xorg                            4MiB |
|    0   N/A  N/A      2063      G   /usr/lib/xorg/Xorg                            4MiB |
+---------------------------------------------------------------------------------------+

I cannot find the ppa link to install cuda-12.0 and nvidia-driver-525 The current ppa link installs cuda-12.1 and nvidia-driver-530.
Could the latest cuda-driver and nvidia-driver cause the probem ?

Looking forward to receiving your reply.. Thx.

david2611 commented 1 year ago

We have been advised that the 530 drivers might be causing issues within Omniverse. Were your 530 drivers installed through the BenchBot install process or did you have those installed prior? The latest stable 525 drivers are what is recommended.

A dirty fix may be to uninstall your 530 drivers then run the benchbot_install script again.

tan007 commented 1 year ago

Thanks for your replay David, I installed 525 Nividia-drivers at the beginning, but the Cuda-driver installation process re-installed the Nvidia-driver to 530. I found that the installation script actually installs the latest 12.1 Cuda-driver and it also installs Nvidia-driver 530. The previous 12.0.1 or 12.0.0 Cuda-driver works with Nvidia-driver 525. Any quick fix to make the installation script of Benchbot to install the Cuda-12.0 or 12.01 driver, not the latest Cuda-12.1 driver?

Looking forward to your suggestion....Thanks

btalb commented 1 year ago

Thanks for the details @tan007 . Sorry about the troubles, it's hard when we can't rely on stability in latest versions...

Simplest option:

  1. Uninstall everything NVIDIA & CUDA (when you're done apt list --installed | grep '\(nvidia\|cuda\)' should be empty)
  2. Manually install specific versions you need of both NVIDIA drivers and CUDA (might be worth putting an apt-mark hold on them)
  3. Re-run benchbot_install; it should not touch what's installed on your host, and re-build the dockers to match the host

Easiest way to get BenchBot to help sort this for you (this code path isn't heavily used so might not work perfectly sorry):

  1. Uninstall everything NVIDIA & CUDA (when you're done apt list --installed | grep '\(nvidia\|cuda\)' should be empty)
  2. Change the default fallback CUDA version here, e.g.: 
    # Defaults for the installation process
ADDONS_DEFAULT='benchbot-addons/ssu'
- CUDA_DEFAULT='cuda'
+ CUDA_DEFAULT='cuda=12.0.0-1'
NVIDIA_DEFAULT='nvidia-driver-525'
SIMULATOR_DEFAULT='sim_omni'
  3. Re-run benchbot_install

Let me know how this goes.

david2611 commented 1 year ago

The driver may not be the issue here. I have just re-installed benchbot on a test machine and have updated CUDA (which seems to update NVIDIA drivers even when you set it to the old version) and once installation completed, BenchBot operated as normal.

System state output from test machine

################################################################################
######################## PART 1: EXAMINING SYSTEM STATE ########################
################################################################################

Core host system checks:
    Ubuntu version >= 20.04:                                  Passed (20.04)

Running Nvidia related system checks:
    NVIDIA GPU available:                     Found card of type '10de:2230'
    NVIDIA driver is running:                                          Found
    NVIDIA driver version valid:                           Valid (530.30.02)
    NVIDIA driver from a standard PPA:                          PPA is valid
    CUDA drivers installed:                                    Drivers found
    CUDA drivers version valid:                          Valid (530.30.02-1)
    CUDA drivers from the NVIDIA PPA:                           PPA is valid
    CUDA is installed:                                            CUDA found
    CUDA version valid:                                         Valid (12.0)
    CUDA is from the NVIDIA PPA:                                PPA is valid

Running Docker related system checks:
    Docker is available:                                               Found
    Docker version valid:                                     Valid (23.0.3)
    NVIDIA Container Toolkit installed:                       Found (1.13.0)
    Docker runs without root:                                         Passed

Running checks of filesystem used for Docker:
    /var/lib/docker on ext4 filesystem:                 Yes (/dev/nvme0n1p2)
    /var/lib/docker supports suid:                                   Enabled
    /var/lib/docker drive space check:               Sufficient space (402G)

Miscellaneous requirements:
    Pip python package manager available:                     Found (20.0.2)
    Tkinter for Python installed:                                      Found
    PIL (with ImageTk) for Python install                              Found

Manual installation steps for Omniverse-powered Isaac Sim:
    License accepted for Omniverse:                                      Yes
    Access to nvcr.io Docker registry:                                   Yes

Does this match what you currently get when you run benchbot_install @tan007 ?

tan007 commented 1 year ago

Thanks for your reply. David. However, I am still stuck in the installation process, even though I have the same system state as yours. 

################################################################################
######################## PART 1: EXAMINING SYSTEM STATE ########################
################################################################################

Core host system checks:
    Ubuntu version >= 20.04:                                  Passed (20.04)

Running Nvidia related system checks:
    NVIDIA GPU available:                     Found card of type '10de:2684'
    NVIDIA driver is running:                                          Found
    NVIDIA driver version valid:                           Valid (530.30.02)
    NVIDIA driver from a standard PPA:                          PPA is valid
    CUDA drivers installed:                                    Drivers found
    CUDA drivers version valid:                          Valid (530.30.02-1)
    CUDA drivers from the NVIDIA PPA:                           PPA is valid
    CUDA is installed:                                            CUDA found
    CUDA version valid:                                         Valid (12.0)
    CUDA is from the NVIDIA PPA:                                PPA is valid

Running Docker related system checks:
    Docker is available:                                               Found
    Docker version valid:                                     Valid (23.0.3)
    NVIDIA Container Toolkit installed:                       Found (1.13.0)
    Docker runs without root:                                         Passed

Running checks of filesystem used for Docker:
    /var/lib/docker on ext4 filesystem:                 Yes (/dev/nvme0n1p2)
    /var/lib/docker supports suid:                                   Enabled
    /var/lib/docker drive space check:              Sufficient space (1721G)

Miscellaneous requirements:
    Pip python package manager available:                     Found (20.0.2)
    Tkinter for Python installed:                                      Found
    PIL (with ImageTk) for Python install                              Found

Manual installation steps for Omniverse-powered Isaac Sim:
    License accepted for Omniverse:                                      Yes
    Access to nvcr.io Docker registry:                                   Yes

I still got the following error messages from the output 

[2.781s] [ext: omni.kit.widget.spinner-1.0.4] startup
[2.782s] [ext: omni.kit.browser.sample-1.2.5] startup
**2023-04-13 08:48:37 [2,778ms] [Error] [carb.scripting-python.plugin] FileNotFoundError: [Errno 2] No such file or directory: '/root/.nvidia-omniverse/config/omniverse.toml'

At:
  /isaac-sim/kit/extscore/omni.kit.pip_archive/pip_prebundle/toml/decoder.py(133): load
  /isaac-sim/extscache/omni.kit.browser.folder.core-1.7.3/omni/kit/browser/folder/core/models/folder_browser_model.py(108): __init__
  /isaac-sim/extscache/omni.kit.browser.folder.core-1.7.3/omni/kit/browser/folder/core/models/tree_folder_browser_model.py(16): __init__
  /isaac-sim/extscache/omni.kit.browser.sample-1.2.5/omni/kit/browser/sample/model.py(20): __init__
  /isaac-sim/extscache/omni.kit.browser.sample-1.2.5/omni/kit/browser/sample/extension.py(31): on_startup
  /isaac-sim/kit/kernel/py/omni/ext/_impl/_internal.py(147): _startup_ext
  /isaac-sim/kit/kernel/py/omni/ext/_impl/_internal.py(198): startup
  /isaac-sim/kit/kernel/py/omni/ext/_impl/_internal.py(285): startup_extension
  PythonExtension.cpp::startup()(2): <module>
  /isaac-sim/exts/omni.isaac.kit/omni/isaac/kit/simulation_app.py(301): _start_app
  /isaac-sim/exts/omni.isaac.kit/omni/isaac/kit/simulation_app.py(186): __init__
  <string>(1): <module>

2023-04-13 08:48:37 [2,778ms] [Error] [omni.ext.plugin] [ext: omni.kit.browser.sample-1.2.5] Failed to startup python extension.**
[2.855s] [ext: omni.graph.nodes-1.48.3] startup
[2.869s] [ext: omni.kit.viewport.menubar.core-104.4.3] startup
[2.902s] [ext: omni.kit.numpy.common-0.1.0] startup
=======================================================
and  warning messages
2023-04-13 08:48:38 [3,142ms] [Warning] [omni.kit.viewport.menubar.core.menu_item.viewport_menu_item] ViewportMenuItem order argument is deprecated, use order_setting_path

2023-04-13 08:48:38 [3,384ms] [Warning] [omni.replicator.core.scripts.annotators] Annotator PostProcessDispatch is already registered, overwriting annotator template

2023-04-13 08:48:39 [4,707ms] [Warning] [omni.ext.plugin] [ext: omni.isaac.sensor-5.6.3] Native plugin search dir contains several files: /isaac-sim/exts/omni.isaac.sensor/bin . It makes dir walk slow, replace with the exact path to the plugin instead of a wildcard.

2023-04-13 08:48:40 [5,362ms] [Warning] [omni.isaac.isaac_sensor] omni.isaac.isaac_sensor is deprecated and was renamed; use omni.isaac.sensor

2023-04-13 08:48:41 [6,148ms] [Warning] [omni.kit.widget.live.cache_state_menu] Unable to detect Omniverse Cache Server. Consider installing it for better IO performance.

The pop up window also crashes after showing the warning message "Unable to detect Omniverse Cache Server. Consider installing it for better IO performance."

By the way, I am using RTX 4090. Could hardware be the problem?

Looking forward to receiving your reply ....

david2611 commented 1 year ago

OK so just confirming I understand rightly, the system was able to install previously but is now giving you errors when you try to reinstall?

Has anything about your system changed between the first and second installs of BenchBot? The reason I ask is that the install process can only complete when the pop-up window appears and does not crash immediately (caching a bunch of things to speed up subsequent times using Isaac Sim).

Can I confirm that in your benchbot folder you have a .cache folder that contains an isaac-sim folder?

tan007 commented 1 year ago

I was able to modify the benchbot_install script to use the the nvidia-driver-525 and cuda-12.0

################################################################################
######################## PART 1: EXAMINING SYSTEM STATE ########################
################################################################################

Core host system checks:
    Ubuntu version >= 20.04:                                  Passed (20.04)

Running Nvidia related system checks:
    NVIDIA GPU available:                     Found card of type '10de:2684'
    NVIDIA driver is running:                                          Found
    NVIDIA driver version valid:                          Valid (525.105.17)
    NVIDIA driver from a standard PPA:                          PPA is valid
    CUDA drivers installed:                                    Drivers found
    CUDA drivers version valid:                         Valid (525.105.17-1)
    CUDA drivers from the NVIDIA PPA:                           PPA is valid
    CUDA is installed:                                            CUDA found
    CUDA version valid:                                         Valid (12.0)
    CUDA is from the NVIDIA PPA:                                PPA is valid

Running Docker related system checks:
    Docker is available:                                               Found
    Docker version valid:                                     Valid (23.0.3)
    NVIDIA Container Toolkit installed:                       Found (1.13.0)
    Docker runs without root:                                         Passed

Running checks of filesystem used for Docker:
    /var/lib/docker on ext4 filesystem:                 Yes (/dev/nvme0n1p2)
    /var/lib/docker supports suid:                                   Enabled
    /var/lib/docker drive space check:              Sufficient space (1667G)

Miscellaneous requirements:
    Pip python package manager available:                     Found (20.0.2)
    Tkinter for Python installed:                                      Found
    PIL (with ImageTk) for Python install                              Found

Manual installation steps for Omniverse-powered Isaac Sim:
    License accepted for Omniverse:                                      Yes
    Access to nvcr.io Docker registry:                                   Yes

I do have the .cache folder that contains an Isaac-sim folder in the benchbot folder.

I suspect that the docker file does not install the Omniverse properly. I check that my PC configuration can run nvidia-onmiverse and Issac-smi by installing the Nvidia omniverse-launcher-linux.App.image .

By the way, is it possible to launch all the benchbot applications without using docker to run Issac-sim and Omniverse? I can easily run them from the Nvidia omniverse-launcher-Linux.App.image.

Thanks again for your help...

tan007 commented 1 year ago

I guess I also need to define somewhere in the installation script the path of .nvidia-omniverse folder. It is in my home directory. Is that right?

david2611 commented 1 year ago

Sorry for the delay.

If you are able to install and run NVIDIA Omniverse outside of BenchBot that is a good sign that things should work in BenchBot.

Unfortunately, BenchBot uses docker for a lot of its moving parts (from earlier design decisions to get it to work in both simulations and on physical hardware). @btalb may know of a way of using the external app in some way shape or form but you would have to do a lot of adaptation to base code that would be more hassle than it is work. If you are able to run Omniverse on your machine you should be able to run it in the docker.

Did you reinstall BenchBot after you changed your drivers? Did it go through the entire install procedure.

It might be a good idea to do a benchbot_install -u to uninstall benchbot before going through the install script again (install script might be using cached images if not). Because you have passed all the install checks with the output you have shown above, BenchBot shouldn't touch your drivers at all.

Nuclear option for a "clean install" would be a docker system prune -a command but that will get rid of ALL docker images.

tan007 commented 1 year ago

Thanks for your reply. David I was able to start Benchbot properly after creating a static link of .nvidia-omniverse from my home directory to the Benchbot folder. The Issac-Sim dashboard starts and renders graphics of a scene. I can launch the hello_benchbot example as well. However, it still complains about not being able to connect to the omniverse-cache server and the example crashes after I run it for a number of steps. [1] How should I start and set up Omniverse-launcher outside of Benchbot, so Benchbot can find the cache-server ? [2] Should I also need to configure the docker to allow more memory for the Benchbot container ?

btalb commented 1 year ago

That's great progress @tan007 .

RE the App.Image: that wasn't available when we initially created BenchBot, so it's all been designed around the Docker installation methods. That being said, if you can run it through the flatpack, than BenchBot should be able to work on your system.

The symbolic linking of .nvidia-omniverse into the BenchBot directory is not something we've needed in the past... I wonder if it's popped up in the upgrade to the latest Isaac Sim and we haven't detected it due to our existing installs having the directory from the past.

Omniverse Cache has never been used, and shouldn't be required to run BenchBot. That is for if we were hosting an instance / assets in the cloud and wanted to interactively work on a simulation across multiple machines. @david2611 can you confirm nothing has changed in this regard with the newest Isaac Sim?

You shouldn't need to make any configuration changes to the Docker or setup cache for BenchBot to run. Have you got any updated logs of what is happening now when it "crashes"? Does the Isaac Sim window die? Is there anything in the logs @david2611 pointed you to above (e.g. docker exec benchbot_robot /bin/bash -c "cat /tmp/benchbot_logs/0").

Thanks for persevering with this with us.

tan007 commented 1 year ago

Thanks for your reply. I attach the last few lines of the log file below. Apparently, there is a core dump error. Any idea how to fix that? Looking forward to receiving your reply.

2023-04-18 06:30:32 [108,757ms] [Warning] [omni.physx.plugin] PhysicsUSD: Prim at path /robot/left_wheel_link/cylinder is using obsolete 'customGeometry' attribute. To toggle custom geome try for cylinders, use the physics settings option. 2023-04-18 06:30:32 [108,757ms] [Warning] [omni.physx.plugin] PhysicsUSD: Prim at path /robot/right_wheel_link/cylinder is using obsolete 'customGeometry' attribute. To toggle custom geom etry for cylinders, use the physics settings option. 2023-04-18 06:30:32 [108,757ms] [Warning] [omni.physx.plugin] PhysicsUSD: Prim at path /robot/rear_wheel_link/cylinder is using obsolete 'customGeometry' attribute. To toggle custom geome try for cylinders, use the physics settings option. 2023-04-18 06:30:33 [108,886ms] [Warning] [omni.physx.plugin] Per-joint collision setting is ignored for articulations. (/robot/chassis_link/com_link) 2023-04-18 06:30:33 [108,886ms] [Warning] [omni.physx.plugin] Translation DOF is expected to be locked for D6 joint in articulation hierarchy, the resulting joint in articulation will hav e all translation DOF locked. 2023-04-18 06:30:33 [108,886ms] [Warning] [omni.physx.plugin] Per-joint collision setting is ignored for articulations. (/robot/chassis_link/left_wheel) 2023-04-18 06:30:33 [108,886ms] [Warning] [omni.physx.plugin] Per-joint collision setting is ignored for articulations. (/robot/chassis_link/right_wheel) 2023-04-18 06:30:33 [108,886ms] [Warning] [omni.physx.plugin] Per-joint collision setting is ignored for articulations. (/robot/chassis_link/rear_pivot) 2023-04-18 06:30:33 [108,886ms] [Warning] [omni.physx.plugin] Per-joint collision setting is ignored for articulations. (/robot/chassis_link/imu_joint) 2023-04-18 06:30:33 [108,886ms] [Warning] [omni.physx.plugin] Per-joint collision setting is ignored for articulations. (/robot/rear_pivot_link/rear_axle) 2023-04-18 06:30:33 [108,886ms] [Warning] [omni.physx.plugin] Per-joint collision setting is ignored for articulations. (/robot/chassis_link/left_wheel) 2023-04-18 06:30:33 [108,886ms] [Warning] [omni.physx.plugin] Per-joint collision setting is ignored for articulations. (/robot/chassis_link/right_wheel) 2023-04-18 06:30:33 [108,886ms] [Warning] [omni.physx.plugin] Per-joint collision setting is ignored for articulations. (/robot/chassis_link/rear_pivot) 2023-04-18 06:30:33 [108,886ms] [Warning] [omni.physx.plugin] Per-joint collision setting is ignored for articulations. (/robot/chassis_link/imu_joint) 2023-04-18 06:30:33 [108,886ms] [Warning] [omni.physx.plugin] Per-joint collision setting is ignored for articulations. (/robot/rear_pivot_link/rear_axle) 2023-04-18 06:30:33 [108,886ms] [Warning] [omni.physx.plugin] Per-joint collision setting is ignored for articulations. (/robot/chassis_link/right_wheel) 2023-04-18 06:30:33 [108,886ms] [Warning] [omni.physx.plugin] Per-joint collision setting is ignored for articulations. (/robot/chassis_link/rear_pivot) 2023-04-18 06:30:33 [108,886ms] [Warning] [omni.physx.plugin] Per-joint collision setting is ignored for articulations. (/robot/chassis_link/imu_joint) 2023-04-18 06:30:33 [108,886ms] [Warning] [omni.physx.plugin] Per-joint collision setting is ignored for articulations. (/robot/rear_pivot_link/rear_axle) 2023-04-18 06:30:33 [108,886ms] [Warning] [omni.physx.plugin] Per-joint collision setting is ignored for articulations. (/robot/chassis_link/rear_pivot) 2023-04-18 06:30:33 [108,886ms] [Warning] [omni.physx.plugin] Per-joint collision setting is ignored for articulations. (/robot/chassis_link/imu_joint) 2023-04-18 06:30:33 [108,886ms] [Warning] [omni.physx.plugin] Per-joint collision setting is ignored for articulations. (/robot/rear_pivot_link/rear_axle) 2023-04-18 06:30:33 [108,886ms] [Warning] [omni.physx.plugin] Per-joint collision setting is ignored for articulations. (/robot/chassis_link/imu_joint) 2023-04-18 06:30:33 [108,886ms] [Warning] [omni.physx.plugin] Per-joint collision setting is ignored for articulations. (/robot/chassis_link/imu_joint) 2023-04-18 06:30:33 [108,886ms] [Warning] [omni.physx.plugin] Translation DOF is expected to be lo./python.sh: line 41: 243 Illegal instruction (core dumped) $python_exe "$@" $args There was an error running python

david2611 commented 1 year ago

Hi all, sorry for delayed replies.

So I will admit to being very confused on how you have installed BenchBot such that it works with your local install of Isaac Sim. You have done something in the install process itself to make it point to the local simulator itself? Without needing to use the IsaacSim docker container? You have installed everything using the AppImage omniverse-launcher?

So when you run benchbot_run it is running the simulation commands to the local version of IsaacSim?

When you installed with the launcher you should have been able to install Omniverse Cache which would give you access to the localhost version of cache.

@btalb the .nvidia-omniverse folder is a by-product of the local install of Isaac Sim and is not included normally in the standard BenchBot install of Isaac Sim which is done entirely through the IsaacSim docker container.

If you are still having issue with the local Isaac Sim and wanted to try the docker container, one trick to check would be the following commands from the Isaac Sim Docker instructions.

xhost +
docker run --name isaac-sim --entrypoint bash -it --gpus all -e "ACCEPT_EULA=Y" --rm --network=host \
 -v ~/docker/isaac-sim/kit/cache/Kit:/isaac-sim/kit/cache/Kit:rw \
 -v ~/docker/isaac-sim/cache/ov:/root/.cache/ov:rw \
 -v ~/docker/isaac-sim/cache/pip:/root/.cache/pip:rw \
 -v ~/docker/isaac-sim/cache/glcache:/root/.cache/nvidia/GLCache:rw \
 -v ~/docker/isaac-sim/cache/computecache:/root/.nv/ComputeCache:rw \
 -v ~/docker/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw \
 -v ~/docker/isaac-sim/data:/root/.local/share/ov/data:rw \
 -v ~/docker/isaac-sim/documents:/root/Documents:rw \
 nvcr.io/nvidia/isaac-sim:2022.2.1

Then within the interactive session ./runapp.sh If that works for you fine, the docker image for Isaac Sim as used in BenchBot should also be working.

tan007 commented 1 year ago

I am running the containerized issac-sim , and kill all the Nvidia Omniverse launcher background jobs. As part of the BenchBot containerized Issac-Sim installation process in my workstation (RTX 4090), the availability of .nvidia-omniverse folder is required. So I create a static link to the same folder in my home directory and Benchbot can launch successfully. Hope this helps to improve the Benchbot... rp5@dt4090:~$ docker container ls CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES c7530aa2b7fb benchbot/backend:base "/bin/bash" 19 minutes ago Up 19 minutes benchbot_debug e39aff1e9e0e benchbot/backend:base "/bin/bash -c 'sourc…" 19 minutes ago Up 19 minutes benchbot_supervisor 8762849e0896 benchbot/simulator:sim_omni "/bin/bash -c 'sourc…" 19 minutes ago Up 19 minutes 8011-8012/tcp, 8211/tcp, 8891/tcp, 8899/tcp, 47995-48012/tcp, 47995-48012/udp, 49000-49007/tcp, 49100/tcp, 49000-49007/udp benchbot_robot 80935f09feed benchbot/backend:base "/bin/bash -c 'sourc…" 19 minutes ago Up 19 minutes

rp5@dt4090:~$ ps -u USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND rp5 2384 0.0 0.0 172652 6392 tty2 Ssl+ 09:21 0:00 /usr/lib/gdm3/gdm-x-session --run-script env GNOME_SHELL_SESSION_MODE=ubuntu /usr/bin/gnome-session --systemd --session=ubuntu rp5 2435 0.0 0.0 199364 15164 tty2 Sl+ 09:21 0:00 /usr/libexec/gnome-session-binary --systemd --systemd --session=ubuntu rp5 3502 0.0 0.0 19644 5576 pts/0 Ss 09:21 0:00 bash rp5 5305 0.0 0.0 18880 4244 pts/0 S+ 09:25 0:00 bash /usr/local/bin/benchbot_run --robot carter_omni --env house:2 --task semantic_slam:passive:ground_truth rp5 6947 0.0 0.0 1625148 24968 pts/0 Sl+ 09:25 0:00 docker logs --follow benchbot_supervisor rp5 9805 0.0 0.0 19644 5296 pts/1 Ss 09:27 0:00 bash rp5 500940 0.0 0.0 19512 5036 pts/2 Ss 09:43 0:00 bash rp5 595699 0.0 0.0 18784 3964 pts/1 S+ 09:48 0:00 bash /usr/local/bin/benchbot_submit --example hello_passive rp5 595786 77.9 0.1 3001096 154068 pts/1 Sl+ 09:48 1:28 python3 .hello_passive/hello_passive

tan007 commented 1 year ago

I manage to fix the frequent crashing of Issac-sim on my workstation. After extensive troubleshooting, I find that the RAM clock frequency is set too high. After reducing the RAM clock frequency, Issac-sim runs well.

david2611 commented 1 year ago

Excellent work! Sorry for the limited communication on my end but happy to hear you reached a satisfactory conclusion. Are you satisfied to close the issue or is there something more that is still breaking in your use of BenchBot?

tan007 commented 1 year ago

Please close this ticket.