Executing benchbot_submit results in no scene change, even when it involves an SCD task.

[yutojubako]

I executed benchbot_run --robot carter_omni --env house:2:3 --task scd:active:ground_truth and observed the following log.

benchbot_run --robot carter_omni --env house:2:3 --task scd:active:ground_truth

################################################################################
################# CHECKING FOR BENCHBOT SOFTWARE STACK UPDATES #################
################################################################################

Checking BenchBot version ...               Up-to-date.
Checking BenchBot API version ...           Up-to-date.
Checking BenchBot Eval version ...          Up-to-date.
Checking BenchBot Supervisor version ...        Up-to-date.
Checking installed BenchBot add-ons are up-to-date ...  Up-to-date.

BenchBot is up-to-date.

################################################################################
###################### CLEANING UP ALL BENCHBOT REMNANTS #######################
################################################################################

Sending stop request to running controller:
{"stop_success":true}

Deleted the following containers:
Total reclaimed space: 0B

Finished cleaning! (use 'benchbot_run -k' for a full clean)

################################################################################
##################### STARTING THE BENCHBOT SOFTWARE STACK #####################
################################################################################

Running the BenchBot system with the following settings:

    Selected task:        scd:active:ground_truth
    Task results format:  object_map_with_states
    Selected robot:       carter_omni
    Selected environment: house:2:3
    Scene/s:              house:2, starting @ pose [0.781, 0, 0, 0.625, 3.2, -0.5, 0.3]
                          (map_path = '.sim_data/house_2.usd')
                          house:3, starting @ pose [0.781, 0, 0, 0.625, -3.2, 0.5, 0.3]
                          (map_path = '.sim_data/house_3.usd')
    Simulator required:   Yes (sim_omni)

Creating shared network 'benchbot_network':

Starting persistent container for ROS core:
Skipping (already running)

Starting persistent container for BenchBot Robot Controller (sim_omni):
Skipping (already running)

Starting container for BenchBot Supervisor:
52c72ec1bb7548691594e17263b3a5fe0bd7afe5327a3d63eff7c00368215978

Starting container for BenchBot Debugging:
f9bd8c43fe35631638cb97188d17c6c29e07b878294c28ea611eb9c81a992143

################################################################################
################### BENCHBOT IS RUNNING (Ctrl^C to exit) ... ###################
################################################################################

Initialising supervisor...

Configuring the supervisor...
Starting a supervisor with the following configuration:

{'environments': [{'_file_path': '/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/house_2.yaml',
                   'description': 'Large house environment with object '
                                  'variations under day lighting\n',
                   'map_path': '.sim_data/house_2.usd',
                   'name': 'house',
                   'object_labels': [...],
                   'robots': [...],
                   'start_pose': [...],
                   'trajectory_poses': [...],
                   'type': 'sim_omni',
                   'variant': 2},
                  {'_file_path': '/benchbot/addons/benchbot_addons/benchbot-addons/envs_bear_develop_sim_omni/environments/house_3.yaml',
                   'description': 'Large house environment with object '
                                  'variations under night lighting\n',
                   'map_path': '.sim_data/house_3.usd',
                   'name': 'house',
                   'object_labels': [...],
                   'robots': [...],
                   'start_pose': [...],
                   'trajectory_poses': [...],
                   'type': 'sim_omni',
                   'variant': 3}],
 'results': {'_file_path': '/benchbot/addons/benchbot_addons/benchbot-addons/formats_object_map/formats/object_map_with_states.yaml',
             'description': 'The same as an "object map", but each object also '
                            'has field specifying a\n'
                            "probability distribution for the object's state. "
                            'Valid states are defined by \n'
                            "the top level 'state_list' field.\n",
             'functions': {'create': 'object_map_states.create_empty',
                           'create_object': 'object_map_states.create_empty_object',
                           'validate': 'object_map_states.validate'},
             'name': 'object_map_with_states'},
 'robot': {'_file_path': '/benchbot/addons/benchbot_addons/benchbot-addons/robots_sim_omni/robots/carter_omni.yaml',
           'address': 'http://benchbot_robot:10000',
           'connections': {'image_depth': {...},
                           'image_depth_info': {...},
                           'image_rgb': {...},
                           'image_rgb_info': {...},
                           'laser': {...},
                           'move_angle': {...},
                           'move_distance': {...},
                           'move_next': {...},
                           'poses': {...}},
           'global_frame': 'map',
           'name': 'carter_omni',
           'persistent_cmds': ['/benchbot/benchbot_simulator/run -P 10001 & '
                               'x=$! && sleep 10 && curl -X POST '
                               'http://localhost:10001/start && wait $x\n',
                               'rosrun benchbot_robot_controller noisify_odom '
                               '\\\n'
                               '  noise_linear:=0.2 noise_angular:=0.1\n'],
           'persistent_status': 'curl -s localhost:10001/started | grep -q '
                                "'true'\n",
           'poses': ['base_link', 'initial_pose', 'camera_left', 'lidar'],
           'robot_frame': 'base_link',
           'run_cmd': 'rostopic pub -1 /odom_start_pose std_msgs/String "data: '
                      '\'$START_POSE\'" && curl -s -o /dev/null '
                      'localhost:10001/open_environment \\\n'
                      '  -H "Content-Type: application/json" \\\n'
                      '  -d \'{"environment": "$ENVS_PATH/$MAP_PATH"}\' &&\n'
                      'curl -s -o /dev/null localhost:10001/place_robot \\\n'
                      '  -H "Content-Type: application/json" \\\n'
                      '  -d \'{"robot": "$ROBOT_PATH/.robot_data/carter.usd", '
                      '"start_pose": "$START_POSE"}\'\n',
           'stop_cmd': 'curl -s -o /dev/null -X POST '
                       'localhost:10001/stop_sim\n',
           'type': 'sim_omni'},
 'task': {'_file_path': '/benchbot/addons/benchbot_addons/benchbot-addons/tasks_ssu/tasks/scd_agt.yaml',
          'actions': ['move_angle', 'move_distance'],
          'description': 'Detect changes between two scenes of an environment. '
                         'An object map describes each changed object with a '
                         'probabilistic label suggestion, spatial location, '
                         'and probabilistic state change suggestion (added, '
                         'removed, changed). This task provides active robot '
                         'control, and sensor observations with ground truth '
                         'robot pose.\n',
          'localisation': 'ground_truth',
          'name': 'scd:active:ground_truth',
          'observations': ['image_depth',
                           'image_depth_info',
                           'image_rgb',
                           'image_rgb_info',
                           'laser',
                           'poses'],
          'results_format': 'object_map_with_states',
          'scene_count': 2,
          'type': 'sim_unreal'}}

Supervisor is now available @ 'http://0.0.0.0:10000' ...

Waiting until a robot controller is found @ 'http://benchbot_robot:10000' ... 
    Found
Sending environment data & robot config to controller ... 
    Ready
Starting the robot controller ... 
    Ready

Next, I executedbencbot_submit --native python3 my_solution.py. After finishing the patrol in house2, the robot started moving in a reinitialized house2 instead of switching to house3. During this, no errors were outputted, and the message Moving to next scene ... Done.was displayed. Here is my_solution.py

class AiGymAgent(Agent):
   def is_done(self, action_result):
       # Finish when we are out of poses or we collide
       return (action_result == ActionResult.FINISHED
               or action_result == ActionResult.COLLISION)

   def pick_action(self, observations, action_list):
       # Always move to next, but also show scene number is available
       print("Running in scene number: %s" % observations['scene_number'])
       return 'move_next', {}

   def save_result(self, filename, empty_results, results_format_fns):
       # We have no results, lets save the empty results with a message
       empty_results['aigym'] = 'Generated by aigym_scd'
       with open(filename, 'w') as f:
           json.dump(empty_results, f)

if __name__ == '__main__':
   # Start BenchBot, & initialise our agent
   bb = BenchBot()
   a = AiGymAgent()

   # Use our Agent to run through the first scene until completion
   observations, action_result = bb.reset()
   print("camera matrix: %s" % observations['image_depth_info'])

   # Ask BenchBot to move us to the next scene
   bb.next_scene()

   # Run through the second scene until completion
   observations, action_result = bb.reset()
   print("observations: %s" % observations['image_depth_info'])

   # When we are finished save our results
   # NOTE: you must save to the default RESULT_LOCATION as show below for your
   # code to work with the '-e' and '-r|--results-location' flags of the
   # 'benchbot_submit' script
   a.save_result(RESULT_LOCATION, bb.empty_results(), bb.results_functions())

Please contact me if you know anything about this.

[yutojubako]

I have encountered the same issue in every SCD task (active/passive, ground_truth/dead_reckoning) and all possible environment pairings.

[david2611]

Sorry for the delay in this. Have just updated the benchbot robot controller to try and fix this problem. It has passed tests on my end. Can you just run a benchbot_install on your end and confirm this also fixes the problem for you?

[david2611]

Pinging this issue again. Can I please receive confirmation that SCD is now working for you? I shall close this issue in 3 working days if I hear of no further issues on your end.

[yutojubako]

Thank you for your reply. Sorry for the delay in replying. By running benchbot_install again, we were able to get SCD to work properly in the environment of apartment1:4 for now. This problem has been successfully resolved, but we have not yet tried it in all other environments and will contact you again if we encounter this type of trouble.