search

qcr / benchbot

BenchBot is a tool for seamlessly testing & evaluating semantic scene understanding tools in both realistic 3D simulation & on real robots
BSD 3-Clause "New" or "Revised" License
110 stars 12 forks source link

Simulation window not launching after executing $ benchbot_run --robot carter_omni --env office:2:3 --task scd:active:ground_trut #96

Open motonarikambara opened 1 year ago

motonarikambara commented 1 year ago

Hello, I hope you're doing well. I executed the following command: benchbot_run --robot carter_omni --env office:2:3 --task scd:active:ground_truth but the simulation window didn't launch as expected. Instead, the following log output was generated:

################################################################################
################# CHECKING FOR BENCHBOT SOFTWARE STACK UPDATES #################
################################################################################

Checking BenchBot version ...               Up-to-date.
Checking BenchBot API version ...           Up-to-date.
Checking BenchBot Eval version ...          Up-to-date.
Checking BenchBot Supervisor version ...        Up-to-date.
Checking installed BenchBot add-ons are up-to-date ...  Up-to-date.

BenchBot is up-to-date.

################################################################################
###################### CLEANING UP ALL BENCHBOT REMNANTS #######################
################################################################################

Sending stop request to running controller:
{"stop_success":true}

Deleted the following containers:
Total reclaimed space: 0B

Finished cleaning! (use 'benchbot_run -k' for a full clean)

################################################################################
##################### STARTING THE BENCHBOT SOFTWARE STACK #####################
################################################################################

Running the BenchBot system with the following settings:

    Selected task:        scd:active:ground_truth
    Task results format:  object_map_with_states
    Selected robot:       carter_omni
    Selected environment: office:2:3
    Scene/s:              office:2, starting @ pose [0.0, 0.0, 0.0, 1.0, 1.0, 5.5, 0.3]
                          (map_path = '.sim_data/office_2.usd')
                          office:3, starting @ pose [0.7154, -0.0029, 0.0028, 0.6987, 0.0, -1.5, 0.3]
                          (map_path = '.sim_data/office_3.usd')
    Simulator required:   Yes (sim_omni)

Creating shared network 'benchbot_network':

Starting persistent container for ROS core:
Skipping (already running)

Starting persistent container for BenchBot Robot Controller (sim_omni):
Skipping (already running)

Starting container for BenchBot Supervisor:
84d0c4732461de26e8dccc6f1db4dda72cec0fb621c1188b65f1970d78710614

Starting container for BenchBot Debugging:
1c00d1e0041b525bfaf18232a8a661b65ca9eac30efe3b1563b216971a6bc16a

################################################################################
################### BENCHBOT IS RUNNING (Ctrl^C to exit) ... ###################
################################################################################

Initialising supervisor...

Configuring the supervisor...
Starting a supervisor with the following configuration:

{'environments': [{'_file_path': '/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_challenge_sim_omni/environments/office_2.yaml',
                   'description': 'Large office environment with object '
                                  'variations under day lighting\n',
                   'map_path': '.sim_data/office_2.usd',
                   'name': 'office',
                   'object_labels': [...],
                   'robots': [...],
                   'start_pose': [...],
                   'trajectory_poses': [...],
                   'type': 'sim_omni',
                   'variant': 2},
                  {'_file_path': '/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_challenge_sim_omni/environments/office_3.yaml',
                   'description': 'Large office environment with object '
                                  'variations under night lighting\n',
                   'map_path': '.sim_data/office_3.usd',
                   'name': 'office',
                   'object_labels': [...],
                   'robots': [...],
                   'start_pose': [...],
                   'trajectory_poses': [...],
                   'type': 'sim_omni',
                   'variant': 3}],
 'results': {'_file_path': '/benchbot/addons/benchbot_addons/benchbot-addons/formats_object_map/formats/object_map_with_states.yaml',
             'description': 'The same as an "object map", but each object also '
                            'has field specifying a\n'
                            "probability distribution for the object's state. "
                            'Valid states are defined by \n'
                            "the top level 'state_list' field.\n",
             'functions': {'create': 'object_map_states.create_empty',
                           'create_object': 'object_map_states.create_empty_object',
                           'validate': 'object_map_states.validate'},
             'name': 'object_map_with_states'},
 'robot': {'_file_path': '/benchbot/addons/benchbot_addons/benchbot-addons/robots_sim_omni/robots/carter_omni.yaml',
           'address': 'http://benchbot_robot:10000',
           'connections': {'image_depth': {...},
                           'image_depth_info': {...},
                           'image_rgb': {...},
                           'image_rgb_info': {...},
                           'laser': {...},
                           'move_angle': {...},
                           'move_distance': {...},
                           'move_next': {...},
                           'poses': {...}},
           'global_frame': 'map',
           'name': 'carter_omni',
           'persistent_cmds': ['/benchbot/benchbot_simulator/run -P 10001 & '
                               'x=$! && sleep 10 && curl -X POST '
                               'http://localhost:10001/start && wait $x\n',
                               'rosrun benchbot_robot_controller noisify_odom '
                               '\\\n'
                               '  noise_linear:=0.2 noise_angular:=0.1\n'],
           'persistent_status': 'curl -s localhost:10001/started | grep -q '
                                "'true'\n",
           'poses': ['base_link', 'initial_pose', 'camera_left', 'lidar'],
           'robot_frame': 'base_link',
           'run_cmd': 'rostopic pub -1 /odom_start_pose std_msgs/String "data: '
                      '\'$START_POSE\'" && curl -s -o /dev/null '
                      'localhost:10001/open_environment \\\n'
                      '  -H "Content-Type: application/json" \\\n'
                      '  -d \'{"environment": "$ENVS_PATH/$MAP_PATH"}\' &&\n'
                      'curl -s -o /dev/null localhost:10001/place_robot \\\n'
                      '  -H "Content-Type: application/json" \\\n'
                      '  -d \'{"robot": "$ROBOT_PATH/.robot_data/carter.usd", '
                      '"start_pose": "$START_POSE"}\'\n',
           'stop_cmd': 'curl -s -o /dev/null -X POST '
                       'localhost:10001/stop_sim\n',
           'type': 'sim_omni'},
 'task': {'_file_path': '/benchbot/addons/benchbot_addons/benchbot-addons/tasks_ssu/tasks/scd_agt.yaml',
          'actions': ['move_angle', 'move_distance'],
          'description': 'Detect changes between two scenes of an environment. '
                         'An object map describes each changed object with a '
                         'probabilistic label suggestion, spatial location, '
                         'and probabilistic state change suggestion (added, '
                         'removed, changed). This task provides active robot '
                         'control, and sensor observations with ground truth '
                         'robot pose.\n',
          'localisation': 'ground_truth',
          'name': 'scd:active:ground_truth',
          'observations': ['image_depth',
                           'image_depth_info',
                           'image_rgb',
                           'image_rgb_info',
                           'laser',
                           'poses'],
          'results_format': 'object_map_with_states',
          'scene_count': 2,
          'type': 'sim_unreal'}}

Supervisor is now available @ 'http://0.0.0.0:10000' ...

Waiting until a robot controller is found @ 'http://benchbot_robot:10000' ... 
    Found
Sending environment data & robot config to controller ... 
    Ready
Starting the robot controller ... 
    Ready

I have tried modifying the task and environment options (--task and --env) but the issue persists.

The installation completed without any errors, and my environment is as follows:

If you have any ideas or suggestions, I would appreciate your assistance. Thank you.

david2611 commented 1 year ago

Thanks for your interest in BenchBot. Do you remember if the simulation window popped up for you during installation? It should have had to happen at least once as part of the install procedure.

We have been advised that the 530 drivers may be breaking with the current version of Omniverse. Are you able to go to the latest stable version of the 525 drivers? Were your drivers installed via the benchbot_install script or had you installed them separately yourself?

btalb commented 1 year ago

See this comment for suggestions on how to cleanly downgrade NVIDIA drivers and CUDA @motonarikambara .

Sorry about the inconvenience with flaky drivers.

david2611 commented 1 year ago

Note, within that issue we have found that the updated drivers are likely not the culprit. Are you able to provide any more information about what happens when the system fails?

motonarikambara commented 1 year ago

Sorry for being late.

Regarding the driver, CUDA 12.0 and NVIDIA driver 530 were used. Both drivers were installed using the ./install command.　During installation, the Isaac Sim window itself appeared (it naturally disappears after about 2 minutes of being left alone).

When executing benchbot_run, the window appears, but the simulation environment is not rendered. Additionally, no error message was specifically output.

During the execution of benchbot_run, the following output was found in /var/log/syslog:

Apr 14 11:04:22 d30 systemd[1]: run-docker-netns-5d56e965b5d2.mount: Succeeded.
Apr 14 11:04:22 d30 systemd[1330]: run-docker-netns-5d56e965b5d2.mount: Succeeded.
Apr 14 11:04:22 d30 systemd[1947]: run-docker-netns-5d56e965b5d2.mount: Succeeded.
Apr 14 11:04:22 d30 systemd[1]: var-lib-docker-overlay2-a8ccab5641ac416094893f896d049aa7fb4ef59995ee15ee90a9d61f2de1ae76-merged.mount: Succeeded.
Apr 14 11:04:22 d30 systemd[1947]: var-lib-docker-overlay2-a8ccab5641ac416094893f896d049aa7fb4ef59995ee15ee90a9d61f2de1ae76-merged.mount: Succeeded.
Apr 14 11:04:22 d30 systemd[1330]: var-lib-docker-overlay2-a8ccab5641ac416094893f896d049aa7fb4ef59995ee15ee90a9d61f2de1ae76-merged.mount: Succeeded.
Apr 14 11:04:26 d30 NetworkManager[1061]: <info>  [1681437866.1395] manager: (br-e58f095f66da): new Bridge device (/org/freedesktop/NetworkManager/Devices/136)
Apr 14 11:04:26 d30 systemd-udevd[63645]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Apr 14 11:04:26 d30 avahi-daemon[1044]: Joining mDNS multicast group on interface br-e58f095f66da.IPv4 with address 172.20.0.254.
Apr 14 11:04:26 d30 avahi-daemon[1044]: New relevant interface br-e58f095f66da.IPv4 for mDNS.
Apr 14 11:04:26 d30 NetworkManager[1061]: <info>  [1681437866.1495] device (br-e58f095f66da): state change: unmanaged -> unavailable (reason 'connection-assumed', sys-iface-state: 'external')
Apr 14 11:04:26 d30 avahi-daemon[1044]: Registering new address record for 172.20.0.254 on br-e58f095f66da.IPv4.
Apr 14 11:04:26 d30 NetworkManager[1061]: <info>  [1681437866.1502] device (br-e58f095f66da): state change: unavailable -> disconnected (reason 'connection-assumed', sys-iface-state: 'external')
Apr 14 11:04:26 d30 NetworkManager[1061]: <info>  [1681437866.1504] device (br-e58f095f66da): Activation: starting connection 'br-e58f095f66da' (aac33fa3-56e4-4a6c-8f6d-b190d372cce0)
Apr 14 11:04:26 d30 NetworkManager[1061]: <info>  [1681437866.1505] device (br-e58f095f66da): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'external')
Apr 14 11:04:26 d30 NetworkManager[1061]: <info>  [1681437866.1506] device (br-e58f095f66da): state change: prepare -> config (reason 'none', sys-iface-state: 'external')
Apr 14 11:04:26 d30 NetworkManager[1061]: <info>  [1681437866.1506] device (br-e58f095f66da): state change: config -> ip-config (reason 'none', sys-iface-state: 'external')
Apr 14 11:04:26 d30 NetworkManager[1061]: <info>  [1681437866.1507] device (br-e58f095f66da): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'external')
Apr 14 11:04:26 d30 dbus-daemon[1058]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.10' (uid=0 pid=1061 comm="/usr/sbin/NetworkManager --no-daemon " label="unconfined")
Apr 14 11:04:26 d30 systemd[1]: Starting Network Manager Script Dispatcher Service...
Apr 14 11:04:26 d30 dbus-daemon[1058]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Apr 14 11:04:26 d30 systemd[1]: Started Network Manager Script Dispatcher Service.
Apr 14 11:04:26 d30 NetworkManager[1061]: <info>  [1681437866.1581] device (br-e58f095f66da): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'external')
Apr 14 11:04:26 d30 NetworkManager[1061]: <info>  [1681437866.1582] device (br-e58f095f66da): state change: secondaries -> activated (reason 'none', sys-iface-state: 'external')
Apr 14 11:04:26 d30 NetworkManager[1061]: <info>  [1681437866.1585] device (br-e58f095f66da): Activation: successful, device activated.
Apr 14 11:04:26 d30 systemd[1947]: var-lib-docker-overlay2-37c86a02fcdc3797b72e49507f7b9c3cb80005562bcb5f3390b84c11f81cef48\x2dinit-merged.mount: Succeeded.
Apr 14 11:04:26 d30 systemd[1]: var-lib-docker-overlay2-37c86a02fcdc3797b72e49507f7b9c3cb80005562bcb5f3390b84c11f81cef48\x2dinit-merged.mount: Succeeded.
Apr 14 11:04:26 d30 systemd[1330]: var-lib-docker-overlay2-37c86a02fcdc3797b72e49507f7b9c3cb80005562bcb5f3390b84c11f81cef48\x2dinit-merged.mount: Succeeded.
Apr 14 11:04:26 d30 systemd-udevd[63645]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Apr 14 11:04:26 d30 systemd-udevd[63645]: veth2c1ffc3: Could not generate persistent MAC: No data available
Apr 14 11:04:26 d30 NetworkManager[1061]: <info>  [1681437866.2244] manager: (veth2c1ffc3): new Veth device (/org/freedesktop/NetworkManager/Devices/137)
Apr 14 11:04:26 d30 NetworkManager[1061]: <info>  [1681437866.2247] manager: (veth637e880): new Veth device (/org/freedesktop/NetworkManager/Devices/138)
Apr 14 11:04:26 d30 systemd-udevd[63810]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Apr 14 11:04:26 d30 systemd-udevd[63810]: veth637e880: Could not generate persistent MAC: No data available
Apr 14 11:04:26 d30 dockerd[1742]: time="2023-04-14T11:04:26.226325897+09:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Apr 14 11:04:26 d30 dockerd[1742]: time="2023-04-14T11:04:26.226345497+09:00" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Apr 14 11:04:26 d30 kernel: [ 2341.228695] br-e58f095f66da: port 1(veth637e880) entered blocking state
Apr 14 11:04:26 d30 kernel: [ 2341.228698] br-e58f095f66da: port 1(veth637e880) entered disabled state
Apr 14 11:04:26 d30 kernel: [ 2341.228734] device veth637e880 entered promiscuous mode
Apr 14 11:04:26 d30 kernel: [ 2341.228810] br-e58f095f66da: port 1(veth637e880) entered blocking state
Apr 14 11:04:26 d30 kernel: [ 2341.228811] br-e58f095f66da: port 1(veth637e880) entered forwarding state
Apr 14 11:04:26 d30 kernel: [ 2341.228966] br-e58f095f66da: port 1(veth637e880) entered disabled state
Apr 14 11:04:26 d30 containerd[1201]: time="2023-04-14T11:04:26.238816675+09:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Apr 14 11:04:26 d30 containerd[1201]: time="2023-04-14T11:04:26.238840328+09:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Apr 14 11:04:26 d30 containerd[1201]: time="2023-04-14T11:04:26.238844880+09:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Apr 14 11:04:26 d30 containerd[1201]: time="2023-04-14T11:04:26.238899966+09:00" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/cfeda62ebe910e327adb277dd17c147e616d5d38bfafe9f2d7522a14bcf56d4a pid=64582 runtime=io.containerd.runc.v2
Apr 14 11:04:26 d30 kernel: [ 2341.452741] eth0: renamed from veth2c1ffc3
Apr 14 11:04:26 d30 NetworkManager[1061]: <info>  [1681437866.4886] device (veth637e880): carrier: link connected
Apr 14 11:04:26 d30 gnome-shell[1501]: Removing a network device that was not added
Apr 14 11:04:26 d30 NetworkManager[1061]: <info>  [1681437866.4894] device (br-e58f095f66da): carrier: link connected
Apr 14 11:04:26 d30 gnome-shell[2177]: Removing a network device that was not added
Apr 14 11:04:26 d30 kernel: [ 2341.492704] IPv6: ADDRCONF(NETDEV_CHANGE): veth637e880: link becomes ready
Apr 14 11:04:26 d30 kernel: [ 2341.492725] br-e58f095f66da: port 1(veth637e880) entered blocking state
Apr 14 11:04:26 d30 kernel: [ 2341.492727] br-e58f095f66da: port 1(veth637e880) entered forwarding state
Apr 14 11:04:26 d30 kernel: [ 2341.492751] IPv6: ADDRCONF(NETDEV_CHANGE): br-e58f095f66da: link becomes ready
Apr 14 11:04:26 d30 systemd[1947]: var-lib-docker-overlay2-5266a38b0360237fd34cb8ce6ed272e7c68403eb12570a21a735eb501306612b\x2dinit-merged.mount: Succeeded.
Apr 14 11:04:26 d30 systemd[1]: var-lib-docker-overlay2-5266a38b0360237fd34cb8ce6ed272e7c68403eb12570a21a735eb501306612b\x2dinit-merged.mount: Succeeded.
Apr 14 11:04:26 d30 systemd-udevd[63810]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Apr 14 11:04:26 d30 systemd-udevd[63645]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Apr 14 11:04:26 d30 systemd-udevd[63810]: vethb64f481: Could not generate persistent MAC: No data available
Apr 14 11:04:26 d30 systemd-udevd[63645]: veth0bf2ce8: Could not generate persistent MAC: No data available
Apr 14 11:04:26 d30 NetworkManager[1061]: <info>  [1681437866.5898] manager: (veth0bf2ce8): new Veth device (/org/freedesktop/NetworkManager/Devices/139)
Apr 14 11:04:26 d30 NetworkManager[1061]: <info>  [1681437866.5903] manager: (vethb64f481): new Veth device (/org/freedesktop/NetworkManager/Devices/140)
Apr 14 11:04:26 d30 dockerd[1742]: time="2023-04-14T11:04:26.591258378+09:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Apr 14 11:04:26 d30 dockerd[1742]: time="2023-04-14T11:04:26.591271474+09:00" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Apr 14 11:04:26 d30 kernel: [ 2341.593722] br-e58f095f66da: port 2(vethb64f481) entered blocking state
Apr 14 11:04:26 d30 kernel: [ 2341.593724] br-e58f095f66da: port 2(vethb64f481) entered disabled state
Apr 14 11:04:26 d30 kernel: [ 2341.593745] device vethb64f481 entered promiscuous mode
Apr 14 11:04:26 d30 kernel: [ 2341.593793] br-e58f095f66da: port 2(vethb64f481) entered blocking state
Apr 14 11:04:26 d30 kernel: [ 2341.593793] br-e58f095f66da: port 2(vethb64f481) entered forwarding state
Apr 14 11:04:26 d30 containerd[1201]: time="2023-04-14T11:04:26.603357880+09:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Apr 14 11:04:26 d30 containerd[1201]: time="2023-04-14T11:04:26.603383053+09:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Apr 14 11:04:26 d30 containerd[1201]: time="2023-04-14T11:04:26.603389189+09:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Apr 14 11:04:26 d30 containerd[1201]: time="2023-04-14T11:04:26.603439228+09:00" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/49aef7684405c6b090d1fe093d8f3d3617be9fda804b4c9ac3b680cb812f3d27 pid=64796 runtime=io.containerd.runc.v2
Apr 14 11:04:26 d30 kernel: [ 2341.872567] eth0: renamed from veth0bf2ce8
Apr 14 11:04:26 d30 NetworkManager[1061]: <info>  [1681437866.9011] device (vethb64f481): carrier: link connected
Apr 14 11:04:26 d30 gnome-shell[2177]: Removing a network device that was not added
Apr 14 11:04:26 d30 gnome-shell[1501]: Removing a network device that was not added
Apr 14 11:04:26 d30 kernel: [ 2341.904632] IPv6: ADDRCONF(NETDEV_CHANGE): vethb64f481: link becomes ready
Apr 14 11:04:26 d30 systemd[1947]: var-lib-docker-overlay2-d1127c340f476ae356fc452e2f4a3aae93ff609c4619530ff2520fcbab218f88\x2dinit-merged.mount: Succeeded.
Apr 14 11:04:26 d30 systemd[1330]: var-lib-docker-overlay2-d1127c340f476ae356fc452e2f4a3aae93ff609c4619530ff2520fcbab218f88\x2dinit-merged.mount: Succeeded.
Apr 14 11:04:26 d30 systemd[1]: var-lib-docker-overlay2-d1127c340f476ae356fc452e2f4a3aae93ff609c4619530ff2520fcbab218f88\x2dinit-merged.mount: Succeeded.
Apr 14 11:04:26 d30 systemd-udevd[63645]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Apr 14 11:04:26 d30 systemd-udevd[63645]: veth27dfdf2: Could not generate persistent MAC: No data available
Apr 14 11:04:26 d30 NetworkManager[1061]: <info>  [1681437866.9629] manager: (veth27dfdf2): new Veth device (/org/freedesktop/NetworkManager/Devices/141)
Apr 14 11:04:26 d30 systemd-udevd[63810]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Apr 14 11:04:26 d30 systemd-udevd[63810]: veth8fdac26: Could not generate persistent MAC: No data available
Apr 14 11:04:26 d30 NetworkManager[1061]: <info>  [1681437866.9634] manager: (veth8fdac26): new Veth device (/org/freedesktop/NetworkManager/Devices/142)
Apr 14 11:04:26 d30 kernel: [ 2341.966784] br-e58f095f66da: port 3(veth8fdac26) entered blocking state
Apr 14 11:04:26 d30 kernel: [ 2341.966786] br-e58f095f66da: port 3(veth8fdac26) entered disabled state
Apr 14 11:04:26 d30 kernel: [ 2341.966806] device veth8fdac26 entered promiscuous mode
Apr 14 11:04:26 d30 kernel: [ 2341.966844] br-e58f095f66da: port 3(veth8fdac26) entered blocking state
Apr 14 11:04:26 d30 kernel: [ 2341.966845] br-e58f095f66da: port 3(veth8fdac26) entered forwarding state
Apr 14 11:04:26 d30 dockerd[1742]: time="2023-04-14T11:04:26.966971093+09:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Apr 14 11:04:26 d30 dockerd[1742]: time="2023-04-14T11:04:26.966988903+09:00" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Apr 14 11:04:26 d30 containerd[1201]: time="2023-04-14T11:04:26.980565890+09:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Apr 14 11:04:26 d30 containerd[1201]: time="2023-04-14T11:04:26.980617459+09:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Apr 14 11:04:26 d30 containerd[1201]: time="2023-04-14T11:04:26.980626425+09:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Apr 14 11:04:26 d30 containerd[1201]: time="2023-04-14T11:04:26.980758639+09:00" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/cfb9f295bf4d2a82ee56f1f42429e22688e68e1dac8f8a9ea825b1af4146c7bb pid=65050 runtime=io.containerd.runc.v2
Apr 14 11:04:27 d30 kernel: [ 2342.148403] br-e58f095f66da: port 3(veth8fdac26) entered disabled state
Apr 14 11:04:27 d30 kernel: [ 2342.236662] eth0: renamed from veth27dfdf2
Apr 14 11:04:27 d30 NetworkManager[1061]: <info>  [1681437867.2648] device (veth8fdac26): carrier: link connected
Apr 14 11:04:27 d30 gnome-shell[1501]: Removing a network device that was not added
Apr 14 11:04:27 d30 gnome-shell[2177]: Removing a network device that was not added
Apr 14 11:04:27 d30 kernel: [ 2342.268670] IPv6: ADDRCONF(NETDEV_CHANGE): veth8fdac26: link becomes ready
Apr 14 11:04:27 d30 kernel: [ 2342.268694] br-e58f095f66da: port 3(veth8fdac26) entered blocking state
Apr 14 11:04:27 d30 kernel: [ 2342.268695] br-e58f095f66da: port 3(veth8fdac26) entered forwarding state
Apr 14 11:04:27 d30 systemd[1947]: var-lib-docker-overlay2-b3912c5f70ecb31ff67ddee9fd777efe9794d373672868465781808903dba586\x2dinit-merged.mount: Succeeded.
Apr 14 11:04:27 d30 systemd[1330]: var-lib-docker-overlay2-b3912c5f70ecb31ff67ddee9fd777efe9794d373672868465781808903dba586\x2dinit-merged.mount: Succeeded.
Apr 14 11:04:27 d30 systemd-udevd[63645]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Apr 14 11:04:27 d30 systemd-udevd[63645]: veth79732f9: Could not generate persistent MAC: No data available
Apr 14 11:04:27 d30 kernel: [ 2342.319739] br-e58f095f66da: port 4(vetha7d2e7e) entered blocking state
Apr 14 11:04:27 d30 kernel: [ 2342.319742] br-e58f095f66da: port 4(vetha7d2e7e) entered disabled state
Apr 14 11:04:27 d30 kernel: [ 2342.319773] device vetha7d2e7e entered promiscuous mode
Apr 14 11:04:27 d30 kernel: [ 2342.319833] br-e58f095f66da: port 4(vetha7d2e7e) entered blocking state
Apr 14 11:04:27 d30 kernel: [ 2342.319835] br-e58f095f66da: port 4(vetha7d2e7e) entered forwarding state
Apr 14 11:04:27 d30 systemd-udevd[63810]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Apr 14 11:04:27 d30 NetworkManager[1061]: <info>  [1681437867.3159] manager: (veth79732f9): new Veth device (/org/freedesktop/NetworkManager/Devices/143)
Apr 14 11:04:27 d30 systemd-udevd[63810]: vetha7d2e7e: Could not generate persistent MAC: No data available
Apr 14 11:04:27 d30 NetworkManager[1061]: <info>  [1681437867.3164] manager: (vetha7d2e7e): new Veth device (/org/freedesktop/NetworkManager/Devices/144)
Apr 14 11:04:27 d30 dockerd[1742]: time="2023-04-14T11:04:27.317287597+09:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Apr 14 11:04:27 d30 dockerd[1742]: time="2023-04-14T11:04:27.317302197+09:00" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Apr 14 11:04:27 d30 containerd[1201]: time="2023-04-14T11:04:27.328898227+09:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Apr 14 11:04:27 d30 containerd[1201]: time="2023-04-14T11:04:27.328925160+09:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Apr 14 11:04:27 d30 containerd[1201]: time="2023-04-14T11:04:27.328930156+09:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Apr 14 11:04:27 d30 containerd[1201]: time="2023-04-14T11:04:27.328977335+09:00" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/74e49623a9967fc406bd3c9a67176ed4b00e8c92002f53f68747fe0ac36c74d0 pid=65284 runtime=io.containerd.runc.v2
Apr 14 11:04:27 d30 kernel: [ 2342.520762] eth0: renamed from veth79732f9
Apr 14 11:04:27 d30 avahi-daemon[1044]: Joining mDNS multicast group on interface veth637e880.IPv6 with address fe80::ca2:eff:fe34:e49b.
Apr 14 11:04:27 d30 avahi-daemon[1044]: New relevant interface veth637e880.IPv6 for mDNS.
Apr 14 11:04:27 d30 avahi-daemon[1044]: Registering new address record for fe80::ca2:eff:fe34:e49b on veth637e880.*.
Apr 14 11:04:27 d30 NetworkManager[1061]: <info>  [1681437867.5513] device (vetha7d2e7e): carrier: link connected
Apr 14 11:04:27 d30 kernel: [ 2342.554458] IPv6: ADDRCONF(NETDEV_CHANGE): vetha7d2e7e: link becomes ready
Apr 14 11:04:27 d30 gnome-shell[2177]: Removing a network device that was not added
Apr 14 11:04:27 d30 gnome-shell[1501]: Removing a network device that was not added
Apr 14 11:04:27 d30 avahi-daemon[1044]: Joining mDNS multicast group on interface br-e58f095f66da.IPv6 with address fe80::42:1aff:fe50:ef8c.
Apr 14 11:04:27 d30 avahi-daemon[1044]: New relevant interface br-e58f095f66da.IPv6 for mDNS.
Apr 14 11:04:27 d30 avahi-daemon[1044]: Registering new address record for fe80::42:1aff:fe50:ef8c on br-e58f095f66da.*.
Apr 14 11:04:28 d30 avahi-daemon[1044]: Joining mDNS multicast group on interface vethb64f481.IPv6 with address fe80::a0c5:f1ff:fedc:e862.
Apr 14 11:04:28 d30 avahi-daemon[1044]: New relevant interface vethb64f481.IPv6 for mDNS.
Apr 14 11:04:28 d30 avahi-daemon[1044]: Registering new address record for fe80::a0c5:f1ff:fedc:e862 on vethb64f481.*.
Apr 14 11:04:28 d30 avahi-daemon[1044]: Joining mDNS multicast group on interface veth8fdac26.IPv6 with address fe80::a89e:36ff:fe6a:4b9e.
Apr 14 11:04:28 d30 avahi-daemon[1044]: New relevant interface veth8fdac26.IPv6 for mDNS.
Apr 14 11:04:28 d30 avahi-daemon[1044]: Registering new address record for fe80::a89e:36ff:fe6a:4b9e on veth8fdac26.*.
Apr 14 11:04:29 d30 avahi-daemon[1044]: Joining mDNS multicast group on interface vetha7d2e7e.IPv6 with address fe80::9cb4:b9ff:fe49:53fb.
Apr 14 11:04:29 d30 avahi-daemon[1044]: New relevant interface vetha7d2e7e.IPv6 for mDNS.
Apr 14 11:04:29 d30 avahi-daemon[1044]: Registering new address record for fe80::9cb4:b9ff:fe49:53fb on vetha7d2e7e.*.
Apr 14 11:04:36 d30 systemd[1]: NetworkManager-dispatcher.service: Succeeded.
Apr 14 11:04:41 d30 gnome-shell[2177]: Window manager warning: Buggy client sent a _NET_ACTIVE_WINDOW message with a timestamp of 0 for 0x4200005
Apr 14 11:05:10 d30 kernel: [ 2385.357905] NVRM: Xid (PCI:0000:01:00): 86, pid=65987, name=python3, CCMDs 00000048 0000c9fa
Apr 14 11:05:11 d30 kernel: [ 2386.570884] python3[65987]: segfault at b8 ip 00007f06c3b74fc4 sp 00007ffed3a98c90 error 4 in libpython3.7m.so.1.0[7f06c3a2a000+608000]
Apr 14 11:05:11 d30 kernel: [ 2386.570893] Code: 84 00 00 00 00 00 41 55 41 54 55 53 48 89 fb 48 83 ec 08 4c 8b 6f 10 e8 8a 3a f6 ff 48 8b 05 bb 2c 6f 00 48 8b a8 c8 05 00 00 <8b> 85 b8 00 00 00 83 f8 31 7f 49 83 c0 01 4d 85 ed 89 85 b8 00 00
Apr 14 11:04:41 d30 gnome-shell[2177]: Window manager warning: Buggy client sent a _NET_ACTIVE_WINDOW message with a timestamp of 0 for 0x4200005
david2611 commented 1 year ago

Sorry for the delay on my end. How long did the window pop up for? The first time you run there will be some delay before the window contents are loaded and rendered. This should be much quicker for subsequent runs as long as the main Isaac Sim window is not closed. By the sounds of it, if the window popped up then disappeared during your install process, it should all be fine.

Have you tried a simpler command (e.g. benchbot_run -r carter_omni -t semantic_slam:passive:ground_truth -e miniroom:1) rather than SCD that needs multiple rooms? Shouldn't make any difference but miniroom is the simplest test room that we use internally. If something breaks there it breaks everywhere.

david2611 commented 1 year ago

Pinging this issue again. Have you managed to solve your problem or are there still issues being encountered?

motonarikambara commented 1 year ago

I apologize for the delayed response. I've tried the following, but the issue remains unresolved:

However, I was able to run it successfully on a machine equipped with an RTX3090, which leads me to believe that the problem may lie with the GPU.

david2611 commented 1 year ago

Hmmm this does seem to potentially be a GPU + driver combo. Your situation seems to match https://github.com/qcr/benchbot/issues/99. While not ideal would you be able to follow their "hack" and see if that works for you at least temporarily?

david2611 commented 1 year ago

I have sadly been unavailable for development on this lately for which I apologize. I have attempted a fix to the driver install issue. If anyone with hardware that has been causing issues can try a fresh benchbot_install and see if the problem has now been resolved it would be much appreciated