ERROR: Supervisor failed on processing connection 'image_depth' with error: JSONDecodeError('Expecting value: line 1 column 1 (char 0)')

[Okayu-i]

Hi, thanks for your great simulator!

I executed the following commands: benchbot_run --robot carter_omni --env office:2:3 --task scd:active:ground_truth and benchbot_submit --example hello_scd

Then I received the following logs:

################################################################################
################# CHECKING FOR BENCHBOT SOFTWARE STACK UPDATES #################
################################################################################

Checking BenchBot version ...               Up-to-date.
Checking BenchBot API version ...           Up-to-date.
Checking BenchBot Eval version ...          Up-to-date.
Checking BenchBot Supervisor version ...        Up-to-date.
Checking installed BenchBot add-ons are up-to-date ...  Up-to-date.

BenchBot is up-to-date.

################################################################################
###################### CLEANING UP ALL BENCHBOT REMNANTS #######################
################################################################################

Deleted the following containers:
Total reclaimed space: 0B

Finished cleaning! (use 'benchbot_run -k' for a full clean)

################################################################################
##################### STARTING THE BENCHBOT SOFTWARE STACK #####################
################################################################################

Running the BenchBot system with the following settings:

    Selected task:        scd:active:ground_truth
    Task results format:  object_map_with_states
    Selected robot:       carter_omni
    Selected environment: office:2:3
    Scene/s:              office:2, starting @ pose [0.0, 0.0, 0.0, 1.0, 1.0, 5.5, 0.3]
                          (map_path = '.sim_data/office_2.usd')
                          office:3, starting @ pose [0.7154, -0.0029, 0.0028, 0.6987, 0.0, -1.5, 0.3]
                          (map_path = '.sim_data/office_3.usd')
    Simulator required:   Yes (sim_omni)

Creating shared network 'benchbot_network':
7aa788ffaf42c300c98b5392d01ba53f62104959d5be9d6641c38f7bb16863df

Starting persistent container for ROS core:
f3d3876f0cefbab469d839baeca84b19578bd18489d5e677119c2faedd53ec03

Starting persistent container for BenchBot Robot Controller (sim_omni):
027f74e8a387c9863887388238d21005c9a678698351b1092167460fe173f126

Starting container for BenchBot Supervisor:
f4b9e02070b2baee1f1544d4d2a01eca23cd8224fc6becbf4e7f148fac2557ca

Starting container for BenchBot Debugging:
4446c70bce62c49874aa251f926c26f0b492d19915cf69edf940214da1ed54ff

################################################################################
################### BENCHBOT IS RUNNING (Ctrl^C to exit) ... ###################
################################################################################

Initialising supervisor...

Configuring the supervisor...
Starting a supervisor with the following configuration:

{'environments': [{'_file_path': '/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_challenge_sim_omni/environments/office_2.yaml',
                   'description': 'Large office environment with object '
                                  'variations under day lighting\n',
                   'map_path': '.sim_data/office_2.usd',
                   'name': 'office',
                   'object_labels': [...],
                   'robots': [...],
                   'start_pose': [...],
                   'trajectory_poses': [...],
                   'type': 'sim_omni',
                   'variant': 2},
                  {'_file_path': '/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_challenge_sim_omni/environments/office_3.yaml',
                   'description': 'Large office environment with object '
                                  'variations under night lighting\n',
                   'map_path': '.sim_data/office_3.usd',
                   'name': 'office',
                   'object_labels': [...],
                   'robots': [...],
                   'start_pose': [...],
                   'trajectory_poses': [...],
                   'type': 'sim_omni',
                   'variant': 3}],
 'results': {'_file_path': '/benchbot/addons/benchbot_addons/benchbot-addons/formats_object_map/formats/object_map_with_states.yaml',
             'description': 'The same as an "object map", but each object also '
                            'has field specifying a\n'
                            "probability distribution for the object's state. "
                            'Valid states are defined by \n'
                            "the top level 'state_list' field.\n",
             'functions': {'create': 'object_map_states.create_empty',
                           'create_object': 'object_map_states.create_empty_object',
                           'validate': 'object_map_states.validate'},
             'name': 'object_map_with_states'},
 'robot': {'_file_path': '/benchbot/addons/benchbot_addons/benchbot-addons/robots_sim_omni/robots/carter_omni.yaml',
           'address': 'http://benchbot_robot:10000',
           'connections': {'image_depth': {...},
                           'image_depth_info': {...},
                           'image_rgb': {...},
                           'image_rgb_info': {...},
                           'laser': {...},
                           'move_angle': {...},
                           'move_distance': {...},
                           'move_next': {...},
                           'poses': {...}},
           'global_frame': 'map',
           'name': 'carter_omni',
           'persistent_cmds': ['/benchbot/benchbot_simulator/run -P 10001 & '
                               'x=$! && sleep 10 && curl -X POST '
                               'http://localhost:10001/start && wait $x\n',
                               'rosrun benchbot_robot_controller noisify_odom '
                               '\\\n'
                               '  noise_linear:=0.2 noise_angular:=0.1\n'],
           'persistent_status': 'curl -s localhost:10001/started | grep -q '
                                "'true'\n",
           'poses': ['base_link', 'initial_pose', 'camera_left', 'lidar'],
           'robot_frame': 'base_link',
           'run_cmd': 'rostopic pub -1 /odom_start_pose std_msgs/String "data: '
                      '\'$START_POSE\'" && curl -s -o /dev/null '
                      'localhost:10001/open_environment \\\n'
                      '  -H "Content-Type: application/json" \\\n'
                      '  -d \'{"environment": "$ENVS_PATH/$MAP_PATH"}\' &&\n'
                      'curl -s -o /dev/null localhost:10001/place_robot \\\n'
                      '  -H "Content-Type: application/json" \\\n'
                      '  -d \'{"robot": "$ROBOT_PATH/.robot_data/carter.usd", '
                      '"start_pose": "$START_POSE"}\'\n',
           'stop_cmd': 'curl -s -o /dev/null -X POST '
                       'localhost:10001/stop_sim\n',
           'type': 'sim_omni'},
 'task': {'_file_path': '/benchbot/addons/benchbot_addons/benchbot-addons/tasks_ssu/tasks/scd_agt.yaml',
          'actions': ['move_angle', 'move_distance'],
          'description': 'Detect changes between two scenes of an environment. '
                         'An object map describes each changed object with a '
                         'probabilistic label suggestion, spatial location, '
                         'and probabilistic state change suggestion (added, '
                         'removed, changed). This task provides active robot '
                         'control, and sensor observations with ground truth '
                         'robot pose.\n',
          'localisation': 'ground_truth',
          'name': 'scd:active:ground_truth',
          'observations': ['image_depth',
                           'image_depth_info',
                           'image_rgb',
                           'image_rgb_info',
                           'laser',
                           'poses'],
          'results_format': 'object_map_with_states',
          'scene_count': 2,
          'type': 'sim_unreal'}}

Supervisor is now available @ 'http://0.0.0.0:10000' ...

Waiting until a robot controller is found @ 'http://benchbot_robot:10000' ... 
    Found
Sending environment data & robot config to controller ... 
    Ready
Starting the robot controller ... 
    Ready

172.20.0.254 - - [2023-04-10 22:39:23] "GET // HTTP/1.1" 200 146 0.000742
172.20.0.254 - - [2023-04-10 22:39:26] "GET /robot/is_running HTTP/1.1" 200 128 0.103774
172.20.0.254 - - [2023-04-10 22:39:26] "GET /config/robot HTTP/1.1" 200 2607 0.000420
172.20.0.254 - - [2023-04-10 22:39:26] "GET /robot/selected_environment HTTP/1.1" 200 149 0.002495
172.20.0.254 - - [2023-04-10 22:39:26] "GET /robot/is_dirty HTTP/1.1" 200 127 0.001974
172.20.0.254 - - [2023-04-10 22:39:26] "GET /robot/is_collided HTTP/1.1" 200 130 0.001900
172.20.0.254 - - [2023-04-10 22:39:26] "GET /robot/is_finished HTTP/1.1" 200 130 0.001604
172.20.0.254 - - [2023-04-10 22:39:26] "GET /config/task/observations HTTP/1.1" 200 188 0.000224
ERROR: Supervisor failed on processing connection 'image_depth' with error:
JSONDecodeError('Expecting value: line 1 column 1 (char 0)')
172.20.0.254 - - [2023-04-10 22:39:26] "GET /connections/image_depth HTTP/1.1" 500 401 0.004947

and

Submitting to the BenchBot system with the following settings:

    Submission mode:             Native (with example 'hello_eval_scd')
    Perform evaluation:          No

    Command to execute:          python3 .hello_eval_scd/hello_eval_scd 
    Command directory:           /home/yui/23_lab/benchbot/addons/benchbot_addons/benchbot-addons/examples_ssu/examples

################################################################################
##################### Running submission in 'Native' mode ######################
################################################################################

Running submission natively via command:

    'python3 .hello_eval_scd/hello_eval_scd '

In directory:

    '/home/yui/23_lab/benchbot/addons/benchbot_addons/benchbot-addons/examples_ssu/examples'

NOTE: this example doesn't actually use the task, and will work in any task. Normally though, it would be important to run this with an SCD task. 
Waiting to establish connection to a running supervisor ... 
Connected!
Waiting to establish connection to a running robot ... 
Connected!

Traceback (most recent call last):
  File "/home/yui/23_lab/benchbot/api/benchbot_api/benchbot.py", line 150, in _query
    raise _UnexpectedResponseError(resp.status_code)
benchbot_api.benchbot._UnexpectedResponseError: Received an unexpected response from BenchBot supervisor (HTTP status code: 500)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".hello_eval_scd/hello_eval_scd", line 13, in <module>
    BenchBot(agent=EvalScdAgent()).run()
  File "/home/yui/23_lab/benchbot/api/benchbot_api/benchbot.py", line 85, in __init__
    self.start()
  File "/home/yui/23_lab/benchbot/api/benchbot_api/benchbot.py", line 401, in start
    self.reset()
  File "/home/yui/23_lab/benchbot/api/benchbot_api/benchbot.py", line 295, in reset
    return self.step(None)
  File "/home/yui/23_lab/benchbot/api/benchbot_api/benchbot.py", line 451, in step
    raw_os = {
  File "/home/yui/23_lab/benchbot/api/benchbot_api/benchbot.py", line 452, in <dictcomp>
    o: self._query(o, BenchBot.RouteType.CONNECTION)
  File "/home/yui/23_lab/benchbot/api/benchbot_api/benchbot.py", line 154, in _query
    raise requests.ConnectionError(
requests.exceptions.ConnectionError: Communication to BenchBot supervisor failed using the route:
    http://benchbot_supervisor:10000/connections/image_depth

Submission failed with result error code: 1

I use an environment:

• Ubuntu 20.04
• NVIDIA RTX 3090

Can someone please guide me on how to resolve this issue?

[david2611]

Thanks for reaching out.

First off can I check if your Omniverse window is popping up at all? Does it stay after the crash? Does this happen for you under all settings (not just scd but semantic_slam, etc.)?

This sort of thing has been known to come up the first time you run a benchbot_run command due to potentially a timing issue within BenchBot as it waits for the ROS messages to be available (may be due a tweak in timings).

If you run your command and wait for the omniverse environment to be fully "online" and kill your benchbot_run command in the terminal (without running benchbot_run -k) the Omniverse window should stick around. If you then run your benchbot_run command again, I have found that this issue should disappear when you then do your benchbot_submit.

[Okayu-i]

Thank you for your prompt and helpful response. Following your instructions, I was able to resolve the error and confirm that the issue is resolved. Once again, thank you for your time and assistance.

[david2611]

No worries. Glad it is working :+1: