Gazebo often fails to connect to master

[sskorol]

Hi @noeperez @roberottt,

I played with default launch configs a bit and noticed a very high startup failure rate due to the inability of a Gazebo client to connect to the master. In this case, the whole set of launched nodes just quits and Gazebo client hangs in the background.

Any thoughts on how to stabilize it?

[noeperez]

We have experienced Gazebo failures at startup only sometimes. Also, we have Gazebo problems with other simulations not using the HuNavSim. So, actually, I do not know the problem, and I can not help you at the moment unfortunately. It seems a Gazebo problem more than an issue related to the wrapper, but we will look into it.

[sskorol]

Well, I've never faced such issues before with Gazebo. I have a pet project with an AWS small_house world that is running via its native launch script. And it always correctly spawns Gazebo. I'm looking at their launch script, and they use the following approach:

def generate_launch_description():
    world_file_name = 'small_house.world'
    package_dir = get_package_share_directory('aws_robomaker_small_house_world')
    gazebo_ros = get_package_share_directory('gazebo_ros')

    gazebo_client = launch.actions.IncludeLaunchDescription(
    launch.launch_description_sources.PythonLaunchDescriptionSource(
            os.path.join(gazebo_ros, 'launch', 'gzclient.launch.py')),
        condition=launch.conditions.IfCondition(launch.substitutions.LaunchConfiguration('gui'))
     )
    gazebo_server = launch.actions.IncludeLaunchDescription(
        launch.launch_description_sources.PythonLaunchDescriptionSource(
            os.path.join(gazebo_ros, 'launch', 'gzserver.launch.py'))
    )

    return LaunchDescription([
        DeclareLaunchArgument(
          'world',
          default_value=[os.path.join(package_dir, 'worlds', world_file_name), ''],
          description='SDF world file'),
        DeclareLaunchArgument(
            name='gui',
            default_value='false'
        ),
        DeclareLaunchArgument(
            name='use_sim_time',
            default_value='true'
        ),
        DeclareLaunchArgument('state',
            default_value='true',
            description='Set "true" to load "libgazebo_ros_state.so"'),
        gazebo_server,
        gazebo_client,
    ])

if __name__ == '__main__':
    generate_launch_description()

hunav_gazebo_wrapper uses a bit different approach via the process executor with a delayed start:

    gzserver_cmd = [
        'gzserver ',
         world_path, 
        _boolean_command('verbose'), '',
        '-s ', 'libgazebo_ros_init.so',
        '-s ', 'libgazebo_ros_factory.so',
        '--ros-args',
        '--params-file', config_file,
    ]

    gzclient_cmd = [
        'gzclient',
        _boolean_command('verbose'), ' ',
    ]

    gzserver_process = ExecuteProcess(
        cmd=gzserver_cmd,
        output='screen',
        shell=True,
        on_exit=Shutdown()
    )

    gzclient_process = ExecuteProcess(
        cmd=gzclient_cmd,
        output='screen',
        shell=True,
        on_exit=Shutdown()
    )

    ordered_launch_event2 = RegisterEventHandler(
        OnProcessStart(
            target_action=hunav_gazebo_worldgen_node,
            on_start=[
                LogInfo(msg='GenerateWorld started, launching Gazebo after 2 seconds...'),
                TimerAction(
                    period=2.0,
                    actions=[gzserver_process, gzclient_process],
                )
            ]
        )

So chances are the reason is in such process augmentation. I'll try to double-check if it'd work in a classic way.

[noeperez]

Our case is different, we need to first generate the world file with the agents and plugins, and then to launch that world in Gazebo. I think that is not the problem, but you could just try to increase the timer period (seconds) just in case Gazebo is launched before the world is created.

[sskorol]

I believe increasing the timer won't help as the issue is between Gazebo server and client, not between world generator and Gazebo. As far as I can see from the recent launch logic, you spawn both server and client at the same time. A blind guess is that we need to wait for server startup before spawning a client. Will probably test this theory.