ros2 / rviz

ROS 3D Robot Visualizer
BSD 3-Clause Clear License
293 stars 211 forks source link

rviz2 randomly crashes while running with nav2 stack #703

Closed AlexeyMerzlyakov closed 2 years ago

AlexeyMerzlyakov commented 3 years ago

Basic info

RViz2 from frequently crashes during the use while running with Nav2 stack. The problem appears in random places even if all Nav2-related stuff was disabled (see minimal configuration file in reproduction section below). Crashes were noted on Crystal and still appearing on latest Rolling.

More info

Crashes appears irregularly in different places. All types of messages and backtraces were detected and sorted-out by their probability of occurrence:

  1. Invalid Argument
    terminate called after throwing an instance of 'std::system_error'
    what():  Invalid argument
  2. Segfault
    Thread 16 "rviz2" received signal SIGSEGV, Segmentation fault.
  3. Assertion Robust
    rviz2: ../nptl/pthread_mutex_lock.c:428: __pthread_mutex_lock_full: Assertion `e != ESRCH || !robust' failed.
    --Type <RET> for more, q to quit, c to continue without paging--
  4. Deadlock: RViz2 hangs during the use
  5. Assertion Prio
    [rviz2-4] rviz2: tpp.c:82: __pthread_tpp_change_priority: Assertion `new_prio == -1 || (new_prio >= fifo_min_prio && new_prio <= fifo_max_prio)' failed.
    [ERROR] [rviz2-4]: process has died [pid 174985, exit code -6, cmd '/home/leha/ros2_rolling/install/rviz2/lib/rviz2/rviz2 -d /home/leha/nav2_ws/install/nav2_bringup/share/nav2_bringup/rviz/nav2_default_view.rviz --ros-args'].

For the types No.1-4 GDB backtraces were collected and attached in the archive: backtraces.zip For type No.5 backtrace was not obtained (it was detected once). All of the problems above were detected by using the same scenario. Thus maybe (but not proven) all issue types have the same root cause.

Steps to reproduce

Despite the problem is irregular and there is no 100% algorithm how to reproduce it, here are the steps allowing with more or less probability to get the problem:

  1. Replace navigation2/nav2_bringup/bringup/rviz/nav2_default_view.rviz file with minimal repro-configuraiton attached to current issue: nav2_default_view.rviz.zip
  2. Run Nav2 simulation as usual:
    ros2 launch nav2_bringup tb3_simulation_launch.py
  3. In appeared RViz screen set robot pose by "2D Pose Estimate"
  4. enable-disable many times the "Controller" checkbox as shown in the picture below: reproduction
  5. If problem won't appear within ~30-40 seconds, close RViz and repeat starting from step No.2.

Environment

Qt version is not printed by rviz:

$ qmake --version
Using Qt version 5.12.8 in /usr/lib/x86_64-linux-gnu
clalancette commented 3 years ago

So I spent a bit of time looking at this yesterday.

First, thanks to @AlexeyMerzlyakov for providing easy instructions for a reproducer. Following the instructions in the initial comment, I was able to reproduce after a few tries.

I then rebuilt parts of the stack with debugging enabled. After adding in some debugging here and there, my initial thought is that this is actually a bug in tf2_ros that is getting exposed by repeatedly enabling and disabling the Map and/or the Polygon in the UI.

In particular, it seems like what is happening is that every time the Map or Polygon is getting enabled, it is creating a new MessageFilter object. Every time they are disabled, that MessageFilter object is destroyed. One of the things that can happen while the MessageFilter object is alive is that it can start waiting for a transform: https://github.com/ros2/geometry2/blob/c36999b17cbf1e64a7937232133db79898bfa608/tf2_ros/include/tf2_ros/message_filter.h#L426 . However, the kicker is that if the MessageFilter object is destroyed before that transform comes in, it never unregisters its interest in that transform. Thus, the MessageFilter object may be destroyed, and sometime later the underlying Buffer object may get the transform and call transformReadyCallback for that transform. But that is now pointing to random memory, so arbitrary things may happen.

I played around a bit with trying to unregister the wait when the MessageFilter object was destroyed, but I wasn't really successful yet. I'm going to have to put this down for now, but I think this is the direction we need to explore; make sure that the MessageFilter unregisters any waitForTransforms it did while it was active.

peredwardsson commented 3 years ago

Any update on this topic? Crashing many times per day for me, typically when posting a goal position. Can fairly reliably generate crashes, happy to receive instructions if you need more information.

peredwardsson commented 3 years ago

Reliable crashes this morning. Happens just when the local costmap appears first. Frame below is when it crashes.

rviz_crash

Full log and backtrace:

$ ROS_DOMAIN_ID=2 gdb -ex run --args rviz2 -l
GNU gdb (Ubuntu 8.1.1-0ubuntu1) 8.1.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from rviz2...(no debugging symbols found)...done.
Starting program: /opt/ros/eloquent/bin/rviz2 -l
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffe06e3700 (LWP 13237)]
[New Thread 0x7fffd65c7700 (LWP 13238)]
[New Thread 0x7fffd5dc6700 (LWP 13239)]
[New Thread 0x7fffcffff700 (LWP 13240)]
[New Thread 0x7fffcf7fe700 (LWP 13242)]
[New Thread 0x7fffceffd700 (LWP 13243)]
[New Thread 0x7fffce7fc700 (LWP 13244)]
[New Thread 0x7fffcdffb700 (LWP 13245)]
[New Thread 0x7fffcd7fa700 (LWP 13246)]
[New Thread 0x7fffccff9700 (LWP 13247)]
[New Thread 0x7fffb3fff700 (LWP 13248)]
[INFO] [rviz2]: Creating resource group General
[INFO] [rviz2]: Creating resource group OgreInternal
[INFO] [rviz2]: Creating resource group OgreAutodetect
[INFO] [rviz2]: SceneManagerFactory for type 'DefaultSceneManager' registered.
[INFO] [rviz2]: Registering ResourceManager for type Material
[INFO] [rviz2]: Registering ResourceManager for type Mesh
[INFO] [rviz2]: Registering ResourceManager for type Skeleton
[INFO] [rviz2]: MovableObjectFactory for type 'ParticleSystem' registered.
[INFO] [rviz2]: ArchiveFactory for archive type FileSystem registered.
[INFO] [rviz2]: DDS codec registering
[INFO] [rviz2]: ETC codec registering
[INFO] [rviz2]: Registering ResourceManager for type HighLevelGpuProgram
[INFO] [rviz2]: Registering ResourceManager for type Compositor
[INFO] [rviz2]: MovableObjectFactory for type 'Entity' registered.
[INFO] [rviz2]: MovableObjectFactory for type 'Light' registered.
[INFO] [rviz2]: MovableObjectFactory for type 'BillboardSet' registered.
[INFO] [rviz2]: MovableObjectFactory for type 'ManualObject' registered.
[INFO] [rviz2]: MovableObjectFactory for type 'BillboardChain' registered.
[INFO] [rviz2]: MovableObjectFactory for type 'RibbonTrail' registered.
[INFO] [rviz2]: *-*-* OGRE Initialising
[INFO] [rviz2]: *-*-* Version 1.12.1 (Rhagorthua)
[INFO] [rviz2]: OverlayElementFactory for type Panel registered.
[INFO] [rviz2]: OverlayElementFactory for type BorderPanel registered.
[INFO] [rviz2]: OverlayElementFactory for type TextArea registered.
[INFO] [rviz2]: Registering ResourceManager for type Font
[INFO] [rviz2]: Loading library /opt/ros/eloquent/opt/rviz_ogre_vendor/lib/OGRE/RenderSystem_GL.so.1.12.1
[INFO] [rviz2]: Installing plugin: GL RenderSystem
[INFO] [rviz2]: OpenGL Rendering Subsystem created.
[INFO] [rviz2]: Plugin successfully installed
[INFO] [rviz2]: Loading library /opt/ros/eloquent/opt/rviz_ogre_vendor/lib/OGRE/Codec_STBI.so.1.12.1
[INFO] [rviz2]: stb_image - v2.19 - public domain JPEG/PNG reader
[INFO] [rviz2]: Supported formats: jpeg,jpg,png,bmp,psd,tga,gif,pic,ppm,pgm,hdr
[INFO] [rviz2]: CPU Identifier & Features
[INFO] [rviz2]: -------------------------
[INFO] [rviz2]:  *   CPU ID: AuthenticAMD: AMD Ryzen 7 5800HS with Radeon Graphics
[INFO] [rviz2]:  *          SSE: yes
[INFO] [rviz2]:  *         SSE2: yes
[INFO] [rviz2]:  *         SSE3: yes
[INFO] [rviz2]:  *        SSE41: no
[INFO] [rviz2]:  *        SSE42: no
[INFO] [rviz2]:  *          MMX: yes
[INFO] [rviz2]:  *       MMXEXT: yes
[INFO] [rviz2]:  *        3DNOW: no
[INFO] [rviz2]:  *     3DNOWEXT: no
[INFO] [rviz2]:  *         CMOV: yes
[INFO] [rviz2]:  *          TSC: yes
[INFO] [rviz2]:  *INVARIANT TSC: yes
[INFO] [rviz2]:  *          FPU: yes
[INFO] [rviz2]:  *          PRO: yes
[INFO] [rviz2]:  *           HT: no
[INFO] [rviz2]: -------------------------
[INFO] [rviz2]: ******************************
*** Starting GLX Subsystem ***
******************************
[INFO] [rviz2]: GLX_VERSION = 1.4
[INFO] [rviz2]: GLX_EXTENSIONS = GLX_ARB_get_proc_address GLX_ARB_multisample GLX_EXT_visual_info GLX_EXT_visual_rating GLX_EXT_import_context GLX_SGI_video_sync GLX_NV_swap_group GLX_SGIX_fbconfig GLX_SGIX_pbuffer GLX_SGI_swap_control GLX_EXT_swap_control GLX_EXT_swap_control_tear GLX_EXT_buffer_age GLX_ARB_create_context GLX_ARB_create_context_profile GLX_NV_float_buffer GLX_ARB_fbconfig_float GLX_EXT_fbconfig_packed_float GLX_EXT_texture_from_pixmap GLX_EXT_framebuffer_sRGB GLX_NV_copy_image GLX_NV_copy_buffer GLX_NV_multisample_coverage GLX_EXT_create_context_es_profile GLX_EXT_create_context_es2_profile GLX_ARB_create_context_no_error GLX_ARB_create_context_robustness GLX_NV_delay_before_swap GLX_EXT_stereo_tree GLX_ARB_context_flush_control GLX_NV_robustness_video_memory_purge GLX_NV_multigpu_context 
[INFO] [rviz2]: Registering ResourceManager for type Texture
[INFO] [rviz2]: GLRenderSystem::_createRenderWindow "OgreWindow(0)", 1x1 windowed  miscParams: FSAA=4 border=none contentScalingFactor=1,000000 currentGLContext=false externalWindowHandle=92274689 left=0 parentWindowHandle=92274689 top=0 
[INFO] [rviz2]: Actual frame buffer FSAA: 4, gamma: 0, colourBufferSize: 32
[INFO] [rviz2]: Created GL 3.0 context
[INFO] [rviz2]: GLXWindow::create used FBConfigID = 211
[INFO] [rviz2]: GL_VERSION = 4.6.0.0
[INFO] [rviz2]: GL_VENDOR = NVIDIA Corporation
[INFO] [rviz2]: GL_RENDERER = GeForce RTX 3060 Laptop GPU/PCIe/SSE2
[INFO] [rviz2]: GL_EXTENSIONS = GL_AMD_multi_draw_indirect GL_AMD_seamless_cubemap_per_texture GL_AMD_vertex_shader_viewport_index GL_AMD_vertex_shader_layer GL_ARB_arrays_of_arrays GL_ARB_base_instance GL_ARB_bindless_texture GL_ARB_blend_func_extended GL_ARB_buffer_storage GL_ARB_clear_buffer_object GL_ARB_clear_texture GL_ARB_clip_control GL_ARB_color_buffer_float GL_ARB_compatibility GL_ARB_compressed_texture_pixel_storage GL_ARB_conservative_depth GL_ARB_compute_shader GL_ARB_compute_variable_group_size GL_ARB_conditional_render_inverted GL_ARB_copy_buffer GL_ARB_copy_image GL_ARB_cull_distance GL_ARB_debug_output GL_ARB_depth_buffer_float GL_ARB_depth_clamp GL_ARB_depth_texture GL_ARB_derivative_control GL_ARB_direct_state_access GL_ARB_draw_buffers GL_ARB_draw_buffers_blend GL_ARB_draw_indirect GL_ARB_draw_elements_base_vertex GL_ARB_draw_instanced GL_ARB_enhanced_layouts GL_ARB_ES2_compatibility GL_ARB_ES3_compatibility GL_ARB_ES3_1_compatibility GL_ARB_ES3_2_compatibility GL_ARB_explicit_attrib_location GL_ARB_explicit_uniform_location GL_ARB_fragment_coord_conventions GL_ARB_fragment_layer_viewport GL_ARB_fragment_program GL_ARB_fragment_program_shadow GL_ARB_fragment_shader GL_ARB_fragment_shader_interlock GL_ARB_framebuffer_no_attachments GL_ARB_framebuffer_object GL_ARB_framebuffer_sRGB GL_ARB_geometry_shader4 GL_ARB_get_program_binary GL_ARB_get_texture_sub_image GL_ARB_gl_spirv GL_ARB_gpu_shader5 GL_ARB_gpu_shader_fp64 GL_ARB_gpu_shader_int64 GL_ARB_half_float_pixel GL_ARB_half_float_vertex GL_ARB_imaging GL_ARB_indirect_parameters GL_ARB_instanced_arrays GL_ARB_internalformat_query GL_ARB_internalformat_query2 GL_ARB_invalidate_subdata GL_ARB_map_buffer_alignment GL_ARB_map_buffer_range GL_ARB_multi_bind GL_ARB_multi_draw_indirect GL_ARB_multisample GL_ARB_multitexture GL_ARB_occlusion_query GL_ARB_occlusion_query2 GL_ARB_parallel_shader_compile GL_ARB_pipeline_statistics_query GL_ARB_pixel_buffer_object GL_ARB_point_parameters GL_ARB_point_sprite GL_ARB_polygon_offset_clamp GL_ARB_post_depth_coverage GL_ARB_program_interface_query GL_ARB_provoking_vertex GL_ARB_query_buffer_object GL_ARB_robust_buffer_access_behavior GL_ARB_robustness GL_ARB_sample_locations GL_ARB_sample_shading GL_ARB_sampler_objects GL_ARB_seamless_cube_map GL_ARB_seamless_cubemap_per_texture GL_ARB_separate_shader_objects GL_ARB_shader_atomic_counter_ops GL_ARB_shader_atomic_counters GL_ARB_shader_ballot GL_ARB_shader_bit_encoding GL_ARB_shader_clock GL_ARB_shader_draw_parameters GL_ARB_shader_group_vote GL_ARB_shader_image_load_store GL_ARB_shader_image_size GL_ARB_shader_objects GL_ARB_shader_precision GL_ARB_shader_storage_buffer_object GL_ARB_shader_subroutine GL_ARB_shader_texture_image_samples GL_ARB_shader_texture_lod GL_ARB_shading_language_100 GL_ARB_shader_viewport_layer_array GL_ARB_shading_language_420pack GL_ARB_shading_language_include GL_ARB_shading_language_packing GL_ARB_shadow GL_ARB_sparse_buffer GL_ARB_sparse_texture GL_ARB_sparse_texture2 GL_ARB_sparse_texture_clamp GL_ARB_spirv_extensions GL_ARB_stencil_texturing GL_ARB_sync GL_ARB_tessellation_shader GL_ARB_texture_barrier GL_ARB_texture_border_clamp GL_ARB_texture_buffer_object GL_ARB_texture_buffer_object_rgb32 GL_ARB_texture_buffer_range GL_ARB_texture_compression GL_ARB_texture_compression_bptc GL_ARB_texture_compression_rgtc GL_ARB_texture_cube_map GL_ARB_texture_cube_map_array GL_ARB_texture_env_add GL_ARB_texture_env_combine GL_ARB_texture_env_crossbar GL_ARB_texture_env_dot3 GL_ARB_texture_filter_anisotropic GL_ARB_texture_filter_minmax GL_ARB_texture_float GL_ARB_texture_gather GL_ARB_texture_mirror_clamp_to_edge GL_ARB_texture_mirrored_repeat GL_ARB_texture_multisample GL_ARB_texture_non_power_of_two GL_ARB_texture_query_levels GL_ARB_texture_query_lod GL_ARB_texture_rectangle GL_ARB_texture_rg GL_ARB_texture_rgb10_a2ui GL_ARB_texture_stencil8 GL_ARB_texture_storage GL_ARB_texture_storage_multisample GL_ARB_texture_swizzle GL_ARB_texture_view GL_ARB_timer_query GL_ARB_transform_feedback2 GL_ARB_transform_feedback3 GL_ARB_transform_feedback_instanced GL_ARB_transform_feedback_overflow_query GL_ARB_transpose_matrix GL_ARB_uniform_buffer_object GL_ARB_vertex_array_bgra GL_ARB_vertex_array_object GL_ARB_vertex_attrib_64bit GL_ARB_vertex_attrib_binding GL_ARB_vertex_buffer_object GL_ARB_vertex_program GL_ARB_vertex_shader GL_ARB_vertex_type_10f_11f_11f_rev GL_ARB_vertex_type_2_10_10_10_rev GL_ARB_viewport_array GL_ARB_window_pos GL_ATI_draw_buffers GL_ATI_texture_float GL_ATI_texture_mirror_once GL_S3_s3tc GL_EXT_texture_env_add GL_EXT_abgr GL_EXT_bgra GL_EXT_bindable_uniform GL_EXT_blend_color GL_EXT_blend_equation_separate GL_EXT_blend_func_separate GL_EXT_blend_minmax GL_EXT_blend_subtract GL_EXT_compiled_vertex_array GL_EXT_Cg_shader GL_EXT_depth_bounds_test GL_EXT_direct_state_access GL_EXT_draw_buffers2 GL_EXT_draw_instanced GL_EXT_draw_range_elements GL_EXT_fog_coord GL_EXT_framebuffer_blit GL_EXT_framebuffer_multisample GL_EXTX_framebuffer_mixed_formats GL_EXT_framebuffer_multisample_blit_scaled GL_EXT_framebuffer_object GL_EXT_framebuffer_sRGB GL_EXT_geometry_shader4 GL_EXT_gpu_program_parameters GL_EXT_gpu_shader4 GL_EXT_multi_draw_arrays GL_EXT_multiview_texture_multisample GL_EXT_multiview_timer_query GL_EXT_packed_depth_stencil GL_EXT_packed_float GL_EXT_packed_pixels GL_EXT_pixel_buffer_object GL_EXT_point_parameters GL_EXT_polygon_offset_clamp GL_EXT_post_depth_coverage GL_EXT_provoking_vertex GL_EXT_raster_multisample GL_EXT_rescale_normal GL_EXT_secondary_color GL_EXT_separate_shader_objects GL_EXT_separate_specular_color GL_EXT_shader_image_load_formatted GL_EXT_shader_image_load_store GL_EXT_shader_integer_mix GL_EXT_shadow_funcs GL_EXT_sparse_texture2 GL_EXT_stencil_two_side GL_EXT_stencil_wrap GL_EXT_texture3D GL_EXT_texture_array GL_EXT_texture_buffer_object GL_EXT_texture_compression_dxt1 GL_EXT_texture_compression_latc GL_EXT_texture_compression_rgtc GL_EXT_texture_compression_s3tc GL_EXT_texture_cube_map GL_EXT_texture_edge_clamp GL_EXT_texture_env_combine GL_EXT_texture_env_dot3 GL_EXT_texture_filter_anisotropic GL_EXT_texture_filter_minmax GL_EXT_texture_integer GL_EXT_texture_lod GL_EXT_texture_lod_bias GL_EXT_texture_mirror_clamp GL_EXT_texture_object GL_EXT_texture_shadow_lod GL_EXT_texture_shared_exponent GL_EXT_texture_sRGB GL_EXT_texture_sRGB_R8 GL_EXT_texture_sRGB_decode GL_EXT_texture_storage GL_EXT_texture_swizzle GL_EXT_timer_query GL_EXT_transform_feedback2 GL_EXT_vertex_array GL_EXT_vertex_array_bgra GL_EXT_vertex_attrib_64bit GL_EXT_window_rectangles GL_EXT_x11_sync_object GL_EXT_import_sync_object GL_NV_robustness_video_memory_purge GL_IBM_rasterpos_clip GL_IBM_texture_mirrored_repeat GL_KHR_context_flush_control GL_KHR_debug GL_EXT_memory_object GL_EXT_memory_object_fd GL_NV_memory_object_sparse GL_KHR_parallel_shader_compile GL_KHR_no_error GL_KHR_robust_buffer_access_behavior GL_KHR_robustness GL_EXT_semaphore GL_EXT_semaphore_fd GL_NV_timeline_semaphore GL_KHR_shader_subgroup GL_KTX_buffer_region GL_NV_alpha_to_coverage_dither_control GL_NV_bindless_multi_draw_indirect GL_NV_bindless_multi_draw_indirect_count GL_NV_bindless_texture GL_NV_blend_equation_advanced GL_NV_blend_equation_advanced_coherent GL_NVX_blend_equation_advanced_multi_draw_buffers GL_NV_blend_minmax_factor GL_NV_blend_square GL_NV_clip_space_w_scaling GL_NV_command_list GL_NV_compute_program5 GL_NV_compute_shader_derivatives GL_NV_conditional_render GL_NV_conservative_raster GL_NV_conservative_raster_dilate GL_NV_conservative_raster_pre_snap GL_NV_conservative_raster_pre_snap_triangles GL_NV_conservative_raster_underestimation GL_NV_copy_depth_to_color GL_NV_copy_image GL_NV_depth_buffer_float GL_NV_depth_clamp GL_NV_draw_texture GL_NV_draw_vulkan_image GL_NV_ES1_1_compatibility GL_NV_ES3_1_compatibility GL_NV_explicit_multisample GL_NV_feature_query GL_NV_fence GL_NV_fill_rectangle GL_NV_float_buffer GL_NV_fog_distance GL_NV_fragment_coverage_to_color GL_NV_fragment_program GL_NV_fragment_program_option GL_NV_fragment_program2 GL_NV_fragment_shader_barycentric GL_NV_fragment_shader_interlock GL_NV_framebuffer_mixed_samples GL_NV_framebuffer_multisample_coverage GL_NV_geometry_shader4 GL_NV_geometry_shader_passthrough GL_NV_gpu_program4 GL_NV_internalformat_sample_query GL_NV_gpu_program4_1 GL_NV_gpu_program5 GL_NV_gpu_program5_mem_extended GL_NV_gpu_program_fp64 GL_NV_gpu_shader5 GL_NV_half_float GL_NV_light_max_exponent GL_NV_memory_attachment GL_NV_mesh_shader GL_NV_multisample_coverage GL_NV_multisample_filter_hint GL_NV_occlusion_query GL_NV_packed_depth_stencil GL_NV_parameter_buffer_object GL_NV_parameter_buffer_object2 GL_NV_path_rendering GL_NV_path_rendering_shared_edge GL_NV_pixel_data_range GL_NV_point_sprite GL_NV_primitive_restart GL_NV_primitive_shading_rate GL_NV_query_resource GL_NV_query_resource_tag GL_NV_register_combiners GL_NV_register_combiners2 GL_NV_representative_fragment_test GL_NV_sample_locations GL_NV_sample_mask_override_coverage GL_NV_scissor_exclusive GL_NV_shader_atomic_counters GL_NV_shader_atomic_float GL_NV_shader_atomic_float64 GL_NV_shader_atomic_fp16_vector GL_NV_shader_atomic_int64 GL_NV_shader_buffer_load GL_NV_shader_storage_buffer_object GL_NV_shader_subgroup_partitioned GL_NV_shader_texture_footprint GL_NV_shading_rate_image GL_NV_stereo_view_rendering GL_NV_texgen_reflection GL_NV_texture_barrier GL_NV_texture_compression_vtc GL_NV_texture_dirty_tile_map GL_NV_texture_env_combine4 GL_NV_texture_multisample GL_NV_texture_rectangle GL_NV_texture_rectangle_compressed GL_NV_texture_shader GL_NV_texture_shader2 GL_NV_texture_shader3 GL_NV_transform_feedback GL_NV_transform_feedback2 GL_NV_uniform_buffer_unified_memory GL_NV_vdpau_interop GL_NV_vdpau_interop2 GL_NV_vertex_array_range GL_NV_vertex_array_range2 GL_NV_vertex_attrib_integer_64bit GL_NV_vertex_buffer_unified_memory GL_NV_vertex_program GL_NV_vertex_program1_1 GL_NV_vertex_program2 GL_NV_vertex_program2_option GL_NV_vertex_program3 GL_NV_viewport_array2 GL_NV_viewport_swizzle GL_NVX_conditional_render GL_NV_gpu_multicast GL_NVX_progress_fence GL_NVX_gpu_memory_info GL_NVX_nvenc_interop GL_NV_shader_thread_group GL_NV_shader_thread_shuffle GL_KHR_blend_equation_advanced GL_KHR_blend_equation_advanced_coherent GL_OVR_multiview GL_OVR_multiview2 GL_SGIS_generate_mipmap GL_SGIS_texture_lod GL_SGIX_depth_texture GL_SGIX_shadow GL_SUN_slice_accum 
[INFO] [rviz2]: ***************************
[INFO] [rviz2]: *** GL Renderer Started ***
[INFO] [rviz2]: ***************************
[INFO] [rviz2]: Registering ResourceManager for type GpuProgram
[INFO] [rviz2]: GLSL support detected
[INFO] [rviz2]: GL: Using GL_EXT_framebuffer_object for rendering to textures (best)
[INFO] [rviz2]: FBO PF_UNKNOWN depth/stencil support: D0S8 D16S0 D24S0 D32S0 Packed-D24S8
[INFO] [rviz2]: FBO PF_L8 depth/stencil support: D0S0 D0S8 D16S0 D24S0 D32S0 Packed-D24S8
[INFO] [rviz2]: FBO PF_A8 depth/stencil support: D0S0 D0S8 D16S0 D24S0 D32S0 Packed-D24S8
[INFO] [rviz2]: FBO PF_BYTE_LA depth/stencil support: D0S0 D0S8 D16S0 D24S0 D32S0 Packed-D24S8
[INFO] [rviz2]: FBO PF_R5G6B5 depth/stencil support: D0S0 D0S8 D16S0 D24S0 D32S0 Packed-D24S8
[INFO] [rviz2]: FBO PF_A4R4G4B4 depth/stencil support: D0S0 D0S8 D16S0 D24S0 D32S0 Packed-D24S8
[INFO] [rviz2]: FBO PF_A1R5G5B5 depth/stencil support: D0S0 D0S8 D16S0 D24S0 D32S0 Packed-D24S8
[INFO] [rviz2]: FBO PF_R8G8B8 depth/stencil support: D0S0 D0S8 D16S0 D24S0 D32S0 Packed-D24S8
[INFO] [rviz2]: FBO PF_B8G8R8 depth/stencil support: D0S0 D0S8 D16S0 D24S0 D32S0 Packed-D24S8
[INFO] [rviz2]: FBO PF_A8R8G8B8 depth/stencil support: D0S0 D0S8 D16S0 D24S0 D32S0 Packed-D24S8
[INFO] [rviz2]: FBO PF_A8B8G8R8 depth/stencil support: D0S0 D0S8 D16S0 D24S0 D32S0 Packed-D24S8
[INFO] [rviz2]: FBO PF_B8G8R8A8 depth/stencil support: D0S0 D0S8 D16S0 D24S0 D32S0 Packed-D24S8
[INFO] [rviz2]: FBO PF_FLOAT16_RGB depth/stencil support: D0S0 D0S8 D16S0 D24S0 D32S0 Packed-D24S8
[INFO] [rviz2]: FBO PF_FLOAT16_RGBA depth/stencil support: D0S0 D0S8 D16S0 D24S0 D32S0 Packed-D24S8
[INFO] [rviz2]: FBO PF_FLOAT32_RGB depth/stencil support: D0S0 D0S8 D16S0 D24S0 D32S0 Packed-D24S8
[INFO] [rviz2]: FBO PF_FLOAT32_RGBA depth/stencil support: D0S0 D0S8 D16S0 D24S0 D32S0 Packed-D24S8
[INFO] [rviz2]: FBO PF_X8R8G8B8 depth/stencil support: D0S0 D0S8 D16S0 D24S0 D32S0 Packed-D24S8
[INFO] [rviz2]: FBO PF_X8B8G8R8 depth/stencil support: D0S0 D0S8 D16S0 D24S0 D32S0 Packed-D24S8
[INFO] [rviz2]: FBO PF_R8G8B8A8 depth/stencil support: D0S0 D0S8 D16S0 D24S0 D32S0 Packed-D24S8
[INFO] [rviz2]: FBO PF_DEPTH16 depth/stencil support: D0S0 D16S0 D24S0 D32S0 Packed-D24S8
[INFO] [rviz2]: FBO PF_SHORT_RGBA depth/stencil support: D0S0 D0S8 D16S0 D24S0 D32S0 Packed-D24S8
[INFO] [rviz2]: FBO PF_R3G3B2 depth/stencil support: D0S0 D0S8 D16S0 D24S0 D32S0 Packed-D24S8
[INFO] [rviz2]: FBO PF_FLOAT16_R depth/stencil support: D0S0 D0S8 D16S0 D24S0 D32S0 Packed-D24S8
[INFO] [rviz2]: FBO PF_FLOAT32_R depth/stencil support: D0S0 D0S8 D16S0 D24S0 D32S0 Packed-D24S8
[INFO] [rviz2]: FBO PF_FLOAT16_GR depth/stencil support: D0S0 D0S8 D16S0 D24S0 D32S0 Packed-D24S8
[INFO] [rviz2]: FBO PF_FLOAT32_GR depth/stencil support: D0S0 D0S8 D16S0 D24S0 D32S0 Packed-D24S8
[INFO] [rviz2]: FBO PF_SHORT_RGB depth/stencil support: D0S0 D0S8 D16S0 D24S0 D32S0 Packed-D24S8
[INFO] [rviz2]: [GL] : Valid FBO targets PF_UNKNOWN PF_L8 PF_A8 PF_BYTE_LA PF_R5G6B5 PF_A4R4G4B4 PF_A1R5G5B5 PF_R8G8B8 PF_B8G8R8 PF_A8R8G8B8 PF_A8B8G8R8 PF_B8G8R8A8 PF_FLOAT16_RGB PF_FLOAT16_RGBA PF_FLOAT32_RGB PF_FLOAT32_RGBA PF_X8R8G8B8 PF_X8B8G8R8 PF_R8G8B8A8 PF_DEPTH16 PF_SHORT_RGBA PF_R3G3B2 PF_FLOAT16_R PF_FLOAT32_R PF_FLOAT16_GR PF_FLOAT32_GR PF_SHORT_RGB 
[INFO] [rviz2]: RenderSystem capabilities
[INFO] [rviz2]: -------------------------
[INFO] [rviz2]: RenderSystem Name: OpenGL Rendering Subsystem
[INFO] [rviz2]: GPU Vendor: nvidia
[INFO] [rviz2]: Device Name: GeForce RTX 3060 Laptop GPU/PCIe/SSE2
[INFO] [rviz2]: Driver Version: 4.6.0.0
[INFO] [rviz2]:  * Fixed function pipeline: yes
[INFO] [rviz2]:  * Anisotropic texture filtering: yes
[INFO] [rviz2]:  * Hardware stencil buffer: yes
[INFO] [rviz2]:    - Stencil depth: 8
[INFO] [rviz2]:    - Two sided stencil support: yes
[INFO] [rviz2]:    - Wrap stencil values: yes
[INFO] [rviz2]:  * 32-bit index buffers: yes
[INFO] [rviz2]:  * Vertex programs: yes
[INFO] [rviz2]:  * Number of floating-point constants for vertex programs: 1024
[INFO] [rviz2]:  * Number of integer constants for vertex programs: 0
[INFO] [rviz2]:  * Number of boolean constants for vertex programs: 0
[INFO] [rviz2]:  * Fragment programs: yes
[INFO] [rviz2]:  * Number of floating-point constants for fragment programs: 1024
[INFO] [rviz2]:  * Number of integer constants for fragment programs: 0
[INFO] [rviz2]:  * Number of boolean constants for fragment programs: 0
[INFO] [rviz2]:  * Geometry programs: yes
[INFO] [rviz2]:  * Number of floating-point constants for geometry programs: 2048
[INFO] [rviz2]:  * Number of integer constants for geometry programs: 0
[INFO] [rviz2]:  * Number of boolean constants for geometry programs: 0
[INFO] [rviz2]:  * Tessellation Hull programs: no
[INFO] [rviz2]:  * Number of floating-point constants for tessellation hull programs: 8016
[INFO] [rviz2]:  * Number of integer constants for tessellation hull programs: 21892
[INFO] [rviz2]:  * Number of boolean constants for tessellation hull programs: 21845
[INFO] [rviz2]:  * Tessellation Domain programs: no
[INFO] [rviz2]:  * Number of floating-point constants for tessellation domain programs: 0
[INFO] [rviz2]:  * Number of integer constants for tessellation domain programs: 57456
[INFO] [rviz2]:  * Number of boolean constants for tessellation domain programs: 21972
[INFO] [rviz2]:  * Compute programs: no
[INFO] [rviz2]:  * Number of floating-point constants for compute programs: 21845
[INFO] [rviz2]:  * Number of integer constants for compute programs: 0
[INFO] [rviz2]:  * Number of boolean constants for compute programs: 240
[INFO] [rviz2]:  * Supported Shader Profiles: arbfp1 arbvp1 fp20 fp30 fp40 glsl glsl100 glsl110 glsl120 gp4fp gp4gp gp4vp gpu_fp gpu_gp gpu_vp nvgp4 vp30 vp40
[INFO] [rviz2]:  * Texture Compression: yes
[INFO] [rviz2]:    - DXT: yes
[INFO] [rviz2]:    - VTC: yes
[INFO] [rviz2]:    - PVRTC: no
[INFO] [rviz2]:    - ATC: no
[INFO] [rviz2]:    - ETC1: no
[INFO] [rviz2]:    - ETC2: no
[INFO] [rviz2]:    - BC4/BC5: no
[INFO] [rviz2]:    - BC6H/BC7: no
[INFO] [rviz2]:    - ASTC: no
[INFO] [rviz2]:    - Mipmaps for compressed formats: yes
[INFO] [rviz2]:  * Hardware Occlusion Query: yes
[INFO] [rviz2]:  * User clip planes: yes
[INFO] [rviz2]:  * VET_UBYTE4 vertex element type: yes
[INFO] [rviz2]:  * Infinite far plane projection: yes
[INFO] [rviz2]:  * Hardware render-to-texture: yes
[INFO] [rviz2]:  * Floating point textures: yes
[INFO] [rviz2]:  * Non-power-of-two textures: yes
[INFO] [rviz2]:  * 1d textures: yes
[INFO] [rviz2]:  * Volume textures: yes
[INFO] [rviz2]:  * Multiple Render Targets: 8
[INFO] [rviz2]:    - With different bit depths: yes
[INFO] [rviz2]:  * Point Sprites: yes
[INFO] [rviz2]:  * Wide Lines: yes
[INFO] [rviz2]:  * Hardware Gamma: yes
[INFO] [rviz2]:  * Extended point parameters: yes
[INFO] [rviz2]:  * Max Point Size: 2047
[INFO] [rviz2]:  * Vertex texture fetch: yes
[INFO] [rviz2]:  * Number of texture units: 16
[INFO] [rviz2]:  * Number of vertex attributes: 16
[INFO] [rviz2]:  * Stencil buffer depth: 8
[INFO] [rviz2]:  * Number of vertex blend matrices: 0
[INFO] [rviz2]:    - Max vertex textures: 32
[INFO] [rviz2]:    - Vertex textures shared: yes
[INFO] [rviz2]:  * Render to Vertex Buffer : yes
[INFO] [rviz2]:  * Hardware Atomic Counters: no
[INFO] [rviz2]:  * PBuffer support: yes
[INFO] [rviz2]:  * Vertex Array Objects: no
[INFO] [rviz2]:  * Separate shader objects: no
[INFO] [rviz2]:  * GLSL SSO redeclare interface block: no
[INFO] [rviz2]:  * Debugging/ profiling events: no
[INFO] [rviz2]:  * Map buffer storage: yes
[INFO] [rviz2]: Using FSAA from GL_ARB_multisample extension.
[INFO] [rviz2]: DefaultWorkQueue('Root') initialising on thread main.
[INFO] [rviz2]: Particle Renderer Type 'billboard' registered
[INFO] [rviz2]: Stereo is NOT SUPPORTED
[INFO] [rviz2]: OpenGl version: 4.6 (GLSL 4.6)
[INFO] [rviz2]: Creating resource group rviz_rendering
[INFO] [rviz2]: Added resource location '/opt/ros/eloquent/share/rviz_rendering/ogre_media' of type 'FileSystem' to resource group 'rviz_rendering'
[INFO] [rviz2]: Added resource location '/opt/ros/eloquent/share/rviz_rendering/ogre_media/textures' of type 'FileSystem' to resource group 'rviz_rendering'
[INFO] [rviz2]: Added resource location '/opt/ros/eloquent/share/rviz_rendering/ogre_media/fonts' of type 'FileSystem' to resource group 'rviz_rendering'
[INFO] [rviz2]: Added resource location '/opt/ros/eloquent/share/rviz_rendering/ogre_media/fonts/liberation-sans' of type 'FileSystem' to resource group 'rviz_rendering'
[INFO] [rviz2]: Added resource location '/opt/ros/eloquent/share/rviz_rendering/ogre_media/models' of type 'FileSystem' to resource group 'rviz_rendering'
[INFO] [rviz2]: Added resource location '/opt/ros/eloquent/share/rviz_rendering/ogre_media/materials' of type 'FileSystem' to resource group 'rviz_rendering'
[INFO] [rviz2]: Added resource location '/opt/ros/eloquent/share/rviz_rendering/ogre_media/materials/scripts' of type 'FileSystem' to resource group 'rviz_rendering'
[INFO] [rviz2]: Added resource location '/opt/ros/eloquent/share/rviz_rendering/ogre_media/materials/glsl120' of type 'FileSystem' to resource group 'rviz_rendering'
[INFO] [rviz2]: Added resource location '/opt/ros/eloquent/share/rviz_rendering/ogre_media/materials/glsl120/include' of type 'FileSystem' to resource group 'rviz_rendering'
[INFO] [rviz2]: Added resource location '/opt/ros/eloquent/share/rviz_rendering/ogre_media/materials/glsl120/nogp' of type 'FileSystem' to resource group 'rviz_rendering'
[INFO] [rviz2]: Added resource location '/opt/ros/eloquent/share/rviz_rendering/ogre_media/materials/scripts120' of type 'FileSystem' to resource group 'rviz_rendering'
[INFO] [rviz2]: Parsing scripts for resource group General
[INFO] [rviz2]: Finished parsing scripts for resource group General
[INFO] [rviz2]: Creating resources for group General
[INFO] [rviz2]: All done
[INFO] [rviz2]: Parsing scripts for resource group OgreAutodetect
[INFO] [rviz2]: Finished parsing scripts for resource group OgreAutodetect
[INFO] [rviz2]: Creating resources for group OgreAutodetect
[INFO] [rviz2]: All done
[INFO] [rviz2]: Parsing scripts for resource group OgreInternal
[INFO] [rviz2]: Finished parsing scripts for resource group OgreInternal
[INFO] [rviz2]: Creating resources for group OgreInternal
[INFO] [rviz2]: All done
[INFO] [rviz2]: Parsing scripts for resource group rviz_rendering
[INFO] [rviz2]: Parsing script glsl120.program
[INFO] [rviz2]: Parsing script nogp.program
[INFO] [rviz2]: Parsing script indexed_8bit_image.material
[INFO] [rviz2]: Parsing script point_cloud_tile.material
[INFO] [rviz2]: Parsing script default_pick_and_depth.material
[INFO] [rviz2]: Parsing script point_cloud_point.material
[INFO] [rviz2]: Parsing script point_cloud_sphere.material
[INFO] [rviz2]: Parsing script point_cloud_square.material
[INFO] [rviz2]: Parsing script point_cloud_box.material
[INFO] [rviz2]: Parsing script point_cloud_flat_square.material
[INFO] [rviz2]: Parsing script liberation_sans.fontdef
[INFO] [rviz2]: Finished parsing scripts for resource group rviz_rendering
[INFO] [rviz2]: Creating resources for group rviz_rendering
[INFO] [rviz2]: All done
[INFO] [rviz2]: GLRenderSystem::_createRenderWindow "OgreWindow(1)", 100x30 windowed  miscParams: FSAA=4 contentScalingFactor=1,000000 currentGLContext=false externalWindowHandle=88080401 left=0 parentWindowHandle=88080401 top=0 
[INFO] [rviz2]: Actual frame buffer FSAA: 4, gamma: 0, colourBufferSize: 32
[INFO] [rviz2]: Created GL 3.0 context
[INFO] [rviz2]: GLXWindow::create used FBConfigID = 211
[INFO] [rviz2]: Stereo is NOT SUPPORTED
[New Thread 0x7fffa3185700 (LWP 13249)]
[INFO] [rviz2]: Mesh: Loading rviz_sphere.mesh.
[INFO] [rviz2]: Texture 'SelectionRect0Texture': Loading 1 faces(PF_R8G8B8A8,1x1x1) Internal format is PF_R8G8B8A8,1x1x1.
[INFO] [rviz2]: Texture 'ROSImageTexture0': Loading 1 faces(PF_A8B8G8R8,420x300x1) Internal format is PF_A8B8G8R8,420x300x1.
[INFO] [rviz2]: Texture 'ROSImageTexture0': Loading 1 faces(PF_A8B8G8R8,420x300x1) Internal format is PF_A8B8G8R8,420x300x1.
[INFO] [rviz2]: Texture 'MapPaletteTexture0': Loading 1 faces(PF_A8B8G8R8,256x1x1) Internal format is PF_A8B8G8R8,256x1x1.
[INFO] [rviz2]: Texture 'MapPaletteTexture1': Loading 1 faces(PF_A8B8G8R8,256x1x1) Internal format is PF_A8B8G8R8,256x1x1.
[INFO] [rviz2]: Texture 'MapPaletteTexture2': Loading 1 faces(PF_A8B8G8R8,256x1x1) Internal format is PF_A8B8G8R8,256x1x1.
[INFO] [rviz2]: Texture 'MapPaletteTexture3': Loading 1 faces(PF_A8B8G8R8,256x1x1) Internal format is PF_A8B8G8R8,256x1x1.
[INFO] [rviz2]: Texture 'MapPaletteTexture4': Loading 1 faces(PF_A8B8G8R8,256x1x1) Internal format is PF_A8B8G8R8,256x1x1.
[INFO] [rviz2]: Texture 'MapPaletteTexture5': Loading 1 faces(PF_A8B8G8R8,256x1x1) Internal format is PF_A8B8G8R8,256x1x1.
[INFO] [rviz2]: Texture 'MapPaletteTexture6': Loading 1 faces(PF_A8B8G8R8,256x1x1) Internal format is PF_A8B8G8R8,256x1x1.
[INFO] [rviz2]: Texture 'MapPaletteTexture7': Loading 1 faces(PF_A8B8G8R8,256x1x1) Internal format is PF_A8B8G8R8,256x1x1.
[INFO] [rviz2]: Texture 'MapPaletteTexture8': Loading 1 faces(PF_A8B8G8R8,256x1x1) Internal format is PF_A8B8G8R8,256x1x1.
[INFO] [rviz2]: Mesh: Loading rviz_cylinder.mesh.
[INFO] [rviz2]: Mesh: Loading rviz_cone.mesh.
[New Thread 0x7fffa0c8c700 (LWP 13250)]
[Thread 0x7fffa3185700 (LWP 13249) exited]
[INFO] [rviz2]: Texture 'ROSImageTexture0': Loading 1 faces(PF_A8B8G8R8,420x300x1) Internal format is PF_A8B8G8R8,420x300x1.
[WARN] [rcl.logging_rosout]: Publisher already registered for provided node name. If this is due to multiple nodes with the same name then all logs for that logger name will go out over the existing publisher. As soon as any node with that name is destructed it will unregister the publisher, preventing any further logs for that name from being published on the rosout topic.
[WARN] [rcl.logging_rosout]: Publisher already registered for provided node name. If this is due to multiple nodes with the same name then all logs for that logger name will go out over the existing publisher. As soon as any node with that name is destructed it will unregister the publisher, preventing any further logs for that name from being published on the rosout topic.
[New Thread 0x7fffa3185700 (LWP 13251)]
[INFO] [lifecycle_manager_client_service_client]: Waiting for the lifecycle_manager/is_active service...
[New Thread 0x7fff9bbe6700 (LWP 13252)]
[INFO] [lifecycle_manager_client_service_client]: Waiting for the lifecycle_manager/is_active service...
[New Thread 0x7fff9b3e5700 (LWP 13253)]
[INFO] [rviz2]: Using FSAA from GL_ARB_multisample extension.
[INFO] [lifecycle_manager_client_service_client]: Waiting for the lifecycle_manager/is_active service...
[INFO] [lifecycle_manager_client_service_client]: Waiting for the lifecycle_manager/is_active service...
[INFO] [lifecycle_manager_client_service_client]: Waiting for the lifecycle_manager/is_active service...
[INFO] [lifecycle_manager_client_service_client]: Waiting for the lifecycle_manager/is_active service...
[INFO] [lifecycle_manager_client_service_client]: Waiting for the lifecycle_manager/is_active service...
[INFO] [lifecycle_manager_client_service_client]: Waiting for the lifecycle_manager/is_active service...
[INFO] [lifecycle_manager_client_service_client]: Waiting for the lifecycle_manager/is_active service...
[INFO] [lifecycle_manager_client_service_client]: Waiting for the lifecycle_manager/is_active service...
[INFO] [lifecycle_manager_client_service_client]: Waiting for the lifecycle_manager/is_active service...
[INFO] [lifecycle_manager_client_service_client]: Waiting for the lifecycle_manager/is_active service...
[INFO] [lifecycle_manager_client_service_client]: Waiting for the lifecycle_manager/is_active service...
[INFO] [lifecycle_manager_client_service_client]: Waiting for the lifecycle_manager/is_active service...
[INFO] [lifecycle_manager_client_service_client]: Waiting for the lifecycle_manager/is_active service...
[INFO] [lifecycle_manager_client_service_client]: Waiting for the lifecycle_manager/is_active service...
[INFO] [lifecycle_manager_client_service_client]: Waiting for the lifecycle_manager/is_active service...
[INFO] [lifecycle_manager_client_service_client]: Waiting for the lifecycle_manager/is_active service...
[INFO] [lifecycle_manager_client_service_client]: Waiting for the lifecycle_manager/is_active service...
[INFO] [lifecycle_manager_client_service_client]: Waiting for the lifecycle_manager/is_active service...
[INFO] [rviz2]: Font Liberation Sans using texture size 512x256
[INFO] [rviz2]: Texture 'Liberation SansTexture': Loading 1 faces(PF_BYTE_LA,512x256x1) Internal format is PF_BYTE_LA,512x256x1.
[INFO] [rviz2]: Trying to create a map of size 60 x 60 using 1 swatches
[INFO] [rviz2]: Texture 'MapTexture0': Loading 1 faces(PF_L8,60x60x1) Internal format is PF_L8,60x60x1.
[INFO] [rviz2]: Texture 'MapTexture1': Loading 1 faces(PF_L8,60x60x1) Internal format is PF_L8,60x60x1.

Thread 14 "rviz2" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffa0c8c700 (LWP 13250)]
0x00007fffebd3a1c9 in ?? () from /usr/lib/x86_64-linux-gnu/libGLdispatch.so.0
(gdb) bt
#0  0x00007fffebd3a1c9 in  () at /usr/lib/x86_64-linux-gnu/libGLdispatch.so.0
#1  0x00007fffa6ebc88a in Ogre::GLHardwareVertexBuffer::GLHardwareVertexBuffer(Ogre::HardwareBufferManagerBase*, unsigned long, unsigned long, Ogre::HardwareBuffer::Usage, bool) ()
    at /opt/ros/eloquent/opt/rviz_ogre_vendor/lib/OGRE/RenderSystem_GL.so.1.12.1
#2  0x00007fffa6eb3cd3 in Ogre::GLHardwareBufferManager::createVertexBuffer(unsigned long, unsigned long, Ogre::HardwareBuffer::Usage, bool) ()
    at /opt/ros/eloquent/opt/rviz_ogre_vendor/lib/OGRE/RenderSystem_GL.so.1.12.1
#3  0x00007ffff5489dd9 in Ogre::ManualObject::end() ()
    at /opt/ros/eloquent/opt/rviz_ogre_vendor/lib/libOgreMain.so.1.12.1
#4  0x00007fffa6601b6a in rviz_default_plugins::displays::PolygonDisplay::processMessage(std::shared_ptr<geometry_msgs::msg::PolygonStamped_<std::allocator<void> > const>) ()
    at /opt/ros/eloquent/lib/librviz_default_plugins.so
#5  0x00007fffa66032f4 in rviz_common::MessageFilterDisplay<geometry_msgs::msg::PolygonStamped_<std::allocator<void> > >::incomingMessage(std::shared_ptr<geometry_msgs::msg::PolygonStamped_<std::allocator<void> > const>) () at /opt/ros/eloquent/lib/librviz_default_plugins.so
#6  0x00007fffa66034b8 in std::_Function_handler<void (std::shared_ptr<geometry_msgs::msg::PolygonStamped_<std::allocator<void> > const> const&), std::_Bind<void (rviz_common::MessageFilterDisplay<geometry_msgs::msg::PolygonStamped_<std::allocator<void> > >::*(rviz_common::MessageFilterDisplay<geometry_msgs::msg::PolygonStamped_<std::allocator<void> > >*, std::_Placeholder<1>))(std::shared_ptr<geometry_msgs::msg::PolygonStamped_<std::allocator<void> > const>)> >::_M_invoke(std::_Any_data const&, std::shared_ptr<geometry_msgs::msg::PolygonStamped_<std::allocator<void> > const> const&) ()
    at /opt/ros/eloquent/lib/librviz_default_plugins.so
#7  0x00007fffa660bd68 in message_filters::CallbackHelper1T<std::shared_ptr<geometry_msgs::msg::PolygonStamped_<std::allocator<void> > const> const&, geometry_msgs::msg::PolygonStamped_<std::allocator<void> > >::call(message_filters::MessageEvent<geometry_msgs::msg::PolygonStamped_<std::allocator<void> > const> const&, bool) () at /opt/ros/eloquent/lib/librviz_default_plugins.so
---Type <return> to continue, or q <return> to quit---
#8  0x00007fffa660779f in message_filters::Signal1<geometry_msgs::msg::PolygonStamped_<std::allocator<void> > >::call(message_filters::MessageEvent<geometry_msgs::msg::PolygonStamped_<std::allocator<void> > const> const&) () at /opt/ros/eloquent/lib/librviz_default_plugins.so
#9  0x00007fffa660e791 in tf2_ros::MessageFilter<geometry_msgs::msg::PolygonStamped_<std::allocator<void> >, rviz_common::transformation::FrameTransformer>::transformReadyCallback(std::shared_future<geometry_msgs::msg::TransformStamped_<std::allocator<void> > > const&, unsigned long) ()
    at /opt/ros/eloquent/lib/librviz_default_plugins.so
#10 0x00007fffa529df9f in std::function<void (std::shared_future<geometry_msgs::msg::TransformStamped_<std::allocator<void> > > const&)>::operator()(std::shared_future<geometry_msgs::msg::TransformStamped_<std::allocator<void> > > const&) const (__args#0=..., this=0x555556130708)
    at /usr/include/c++/7/bits/std_function.h:706
#11 0x00007fffa529df9f in tf2_ros::Buffer::<lambda(tf2::TransformableRequestHandle, const string&, const string&, tf2::TimePoint, tf2::TransformableResult)>::operator() (result=<optimized out>, time=..., source_frame=..., target_frame=..., request_handle=<optimized out>, __closure=0x5555561306f0)
    at ./src/buffer.cpp:237
#12 0x00007fffa529df9f in std::_Function_handler<void(long unsigned int, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long int, std::ratio<1, 1000000000> > >, tf2::TransformableResult), tf2_ros::Buffer::waitForTransform(const string&, const string&, const TimePoint&, const Duration&, tf2_ros::TransformReadyCallback)::<lambda(tf2::TransformableRequestHandle, const string&, const string&, tf2::TimePoint, tf2::TransformableResult)> >::_M_invoke(const std::_Any_data &, unsigned long &&, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > &, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > &, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1, 1000000000> > > &&, tf2::TransformableResult &&) (__functor=..., __args#0=<optimized out>, __args#1=..., __args#2=..., __args#3=..., __arg---Type <return> to continue, or q <return> to quit---
s#4=<optimized out>) at /usr/include/c++/7/bits/std_function.h:316
#13 0x00007fffa4c5efd4 in std::function<void (unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tf2::TransformableResult)>::operator()(unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, tf2::TransformableResult) const (__args#4=<optimized out>, __args#3=..., __args#2="odom", __args#1=..., __args#0=<optimized out>, this=0x555556128f80) at /usr/include/c++/7/bits/std_function.h:706
#14 0x00007fffa4c5efd4 in tf2::BufferCore::testTransformableRequests() (this=this@entry=0x555555a16c20) at ./src/buffer_core.cpp:1493
#15 0x00007fffa4c601a2 in tf2::BufferCore::setTransformImpl(tf2::Transform const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) (this=this@entry=0x555555a16c20, transform_in=..., frame_id="odom", child_frame_id="base_link", stamp=..., authority="Authority undetectable", is_static=false) at ./src/buffer_core.cpp:337
#16 0x00007fffa4c60597 in tf2::BufferCore::setTransform(geometry_msgs::msg::TransformStamped_<std::allocator<void> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) (this=0x555555a16c20, transform=..., authority="Authority undetectable", is_static=is_static@entry=false) at ./src/buffer_core.cpp:235
#17 0x00007fffa52a44ae in tf2_ros::TransformListener::subscription_callback(std::shared_ptr<tf2_msgs::msg::TFMessage_<std::allocator<void> > >, bool) (this=0x555555dad770, msg=..., is_static=<optimized out>) at ./src/transform_listener.cpp:102
---Type <return> to continue, or q <return> to quit---
#18 0x00007fffa66cbc88 in std::_Function_handler<void (std::shared_ptr<tf2_msgs::msg::TFMessage_<std::allocator<void> > >), std::_Bind<void (tf2_ros::TransformListener::*(tf2_ros::TransformListener*, std::_Placeholder<1>, bool))(std::shared_ptr<tf2_msgs::msg::TFMessage_<std::allocator<void> > >, bool)> >::_M_invoke(std::_Any_data const&, std::shared_ptr<tf2_msgs::msg::TFMessage_<std::allocator<void> > >&&) () at /opt/ros/eloquent/lib/librviz_default_plugins.so
#19 0x00007fffa66d14fd in rclcpp::AnySubscriptionCallback<tf2_msgs::msg::TFMessage_<std::allocator<void> >, std::allocator<void> >::dispatch(std::shared_ptr<tf2_msgs::msg::TFMessage_<std::allocator<void> > >, rmw_message_info_t const&) () at /opt/ros/eloquent/lib/librviz_default_plugins.so
#20 0x00007fffa66d1960 in rclcpp::Subscription<tf2_msgs::msg::TFMessage_<std::allocator<void> >, std::allocator<void>, rclcpp::message_memory_strategy::MessageMemoryStrategy<tf2_msgs::msg::TFMessage_<std::allocator<void> >, std::allocator<void> > >::handle_message(std::shared_ptr<void>&, rmw_message_info_t const&) () at /opt/ros/eloquent/lib/librviz_default_plugins.so
#21 0x00007ffff658d49f in rclcpp::executor::Executor::execute_subscription(std::shared_ptr<rclcpp::SubscriptionBase>) () at /opt/ros/eloquent/lib/librclcpp.so
#22 0x00007ffff658e7c5 in rclcpp::executor::Executor::execute_any_executable(rclcpp::executor::AnyExecutable&) () at /opt/ros/eloquent/lib/librclcpp.so
#23 0x00007ffff6594fbf in rclcpp::executors::SingleThreadedExecutor::spin() ()
    at /opt/ros/eloquent/lib/librclcpp.so
#24 0x00007fffa52a494c in tf2_ros::TransformListener::<lambda(rclcpp::node_interfaces::NodeBaseInterface::SharedPtr)>::operator() (node_base_interface=..., __closure=<optimized out>)
    at ./src/transform_listener.cpp:79
#25 0x00007fffa52a494c in std::__invoke_impl<void, tf2_ros::TransformListener::initThread(rclcpp::node_interfaces::NodeBaseInterface::SharedPtr)::<lambda(rclcpp::node_interfaces::NodeBaseInterface::SharedPtr)>, std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface> > (__f=...)
    at /usr/include/c++/7/bits/invoke.h:60
#26 0x00007fffa52a494c in std::__invoke<tf2_ros::TransformListener::initThread(rclcpp::node_interface---Type <return> to continue, or q <return> to quit---
s::NodeBaseInterface::SharedPtr)::<lambda(rclcpp::node_interfaces::NodeBaseInterface::SharedPtr)>, std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface> > (__fn=...)
    at /usr/include/c++/7/bits/invoke.h:95
#27 0x00007fffa52a494c in std::thread::_Invoker<std::tuple<tf2_ros::TransformListener::initThread(rclcpp::node_interfaces::NodeBaseInterface::SharedPtr)::<lambda(rclcpp::node_interfaces::NodeBaseInterface::SharedPtr)>, std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface> > >::_M_invoke<0, 1> (this=<optimized out>) at /usr/include/c++/7/thread:234
#28 0x00007fffa52a494c in std::thread::_Invoker<std::tuple<tf2_ros::TransformListener::initThread(rclcpp::node_interfaces::NodeBaseInterface::SharedPtr)::<lambda(rclcpp::node_interfaces::NodeBaseInterface::SharedPtr)>, std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface> > >::operator() (this=<optimized out>) at /usr/include/c++/7/thread:243
#29 0x00007fffa52a494c in std::thread::_State_impl<std::thread::_Invoker<std::tuple<tf2_ros::TransformListener::initThread(rclcpp::node_interfaces::NodeBaseInterface::SharedPtr)::<lambda(rclcpp::node_interfaces::NodeBaseInterface::SharedPtr)>, std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface> > > >::_M_run(void) (this=0x5555558a6640) at /usr/include/c++/7/thread:186
#30 0x00007ffff62276df in  () at /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#31 0x00007ffff3fcc6db in start_thread (arg=0x7fffa0c8c700) at pthread_create.c:463
#32 0x00007ffff5c8271f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) 
(gdb) 
ros-discourse commented 2 years ago

This issue has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/ros-2-tsc-meeting-minutes-2021-10-28/22947/1

ros-discourse commented 2 years ago

This issue has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/ros-2-tsc-meeting-minutes-2021-11-18/23209/1

iuhilnehc-ynos commented 2 years ago

However, the kicker is that if the MessageFilter object is destroyed before that transform comes in, it never unregisters its interest in that transform. Thus, the MessageFilter object may be destroyed, and sometime later the underlying Buffer object may get the transform and call transformReadyCallback for that transform. But that is now pointing to random memory, so arbitrary things may happen.

I agree with what @clalancette analyzed about the root cause for the issue. I'd like to share something about my thought. If MessageFilter is destroyed, I think we can do

patches for some repositories

- https://github.com/ros2/geometry2.git ```patch diff --git a/tf2_ros/include/tf2_ros/message_filter.h b/tf2_ros/include/tf2_ros/message_filter.h index cb61843..87c90e7 100644 --- a/tf2_ros/include/tf2_ros/message_filter.h +++ b/tf2_ros/include/tf2_ros/message_filter.h @@ -144,7 +144,8 @@ public: \endverbatim */ template -class MessageFilter : public MessageFilterBase, public message_filters::SimpleFilter +class MessageFilter : public MessageFilterBase, public message_filters::SimpleFilter, + public std::enable_shared_from_this> { public: using MConstPtr = std::shared_ptr; @@ -428,7 +429,10 @@ public: frame_id, stamp, buffer_timeout_, - std::bind(&MessageFilter::transformReadyCallback, this, std::placeholders::_1, handle)); + std::bind(&MessageFilter::transformReadyCallback, + this, std::placeholders::_1, handle, + // use this->weak_from_this() if C++17 is enable + std::weak_ptr>(this->shared_from_this()))); } } @@ -478,95 +482,101 @@ private: expected_success_count_ = 1; } - void transformReadyCallback(const tf2_ros::TransformStampedFuture & future, const uint64_t handle) + void transformReadyCallback( + const tf2_ros::TransformStampedFuture & future, + const uint64_t handle, + std::weak_ptr> weak_this) { namespace mt = message_filters::message_traits; MEvent saved_event; bool event_found = false; - { - // We will be accessing and mutating messages now, require unique lock - std::unique_lock lock(messages_mutex_); - - // find the message this request is associated with - typename L_MessageInfo::iterator msg_it = messages_.begin(); - typename L_MessageInfo::iterator msg_end = messages_.end(); - - for (; msg_it != msg_end; ++msg_it) { - MessageInfo & info = *msg_it; - auto handle_it = std::find(info.handles.begin(), info.handles.end(), handle); - if (handle_it != info.handles.end()) { - // found msg_it - ++info.success_count; - if (info.success_count >= expected_success_count_) { - saved_event = msg_it->event; - messages_.erase(msg_it); - event_found = true; + auto pthis = weak_this.lock(); + if (pthis) { + { + // We will be accessing and mutating messages now, require unique lock + std::unique_lock lock(messages_mutex_); + + // find the message this request is associated with + typename L_MessageInfo::iterator msg_it = messages_.begin(); + typename L_MessageInfo::iterator msg_end = messages_.end(); + + for (; msg_it != msg_end; ++msg_it) { + MessageInfo & info = *msg_it; + auto handle_it = std::find(info.handles.begin(), info.handles.end(), handle); + if (handle_it != info.handles.end()) { + // found msg_it + ++info.success_count; + if (info.success_count >= expected_success_count_) { + saved_event = msg_it->event; + messages_.erase(msg_it); + event_found = true; + } + break; } - break; } } - } - - if (!event_found) { - return; - } - bool can_transform = true; - const MConstPtr & message = saved_event.getMessage(); - std::string frame_id = stripSlash(mt::FrameId::value(*message)); - rclcpp::Time stamp = mt::TimeStamp::value(*message); - - bool transform_available = true; - FilterFailureReason error = filter_failure_reasons::Unknown; - try { - future.get(); - } catch (...) { - transform_available = false; - error = filter_failure_reasons::OutTheBack; - } + if (!event_found) { + return; + } - if (transform_available) { - std::unique_lock frames_lock(target_frames_mutex_); - // make sure we can still perform all the necessary transforms - typename V_string::iterator it = target_frames_.begin(); - typename V_string::iterator end = target_frames_.end(); - for (; it != end; ++it) { - const std::string & target = *it; - if (!buffer_.canTransform(target, frame_id, tf2::timeFromSec(stamp.seconds()), NULL)) { - can_transform = false; - break; - } + bool can_transform = true; + const MConstPtr & message = saved_event.getMessage(); + std::string frame_id = stripSlash(mt::FrameId::value(*message)); + rclcpp::Time stamp = mt::TimeStamp::value(*message); + + bool transform_available = true; + FilterFailureReason error = filter_failure_reasons::Unknown; + try { + future.get(); + } catch (...) { + transform_available = false; + error = filter_failure_reasons::OutTheBack; + } - if (time_tolerance_.nanoseconds()) { - if (!buffer_.canTransform( - target, frame_id, - tf2::timeFromSec((stamp + time_tolerance_).seconds()), NULL)) - { + if (transform_available) { + std::unique_lock frames_lock(target_frames_mutex_); + // make sure we can still perform all the necessary transforms + typename V_string::iterator it = target_frames_.begin(); + typename V_string::iterator end = target_frames_.end(); + for (; it != end; ++it) { + const std::string & target = *it; + if (!buffer_.canTransform(target, frame_id, tf2::timeFromSec(stamp.seconds()), NULL)) { can_transform = false; break; } + + if (time_tolerance_.nanoseconds()) { + if (!buffer_.canTransform( + target, frame_id, + tf2::timeFromSec((stamp + time_tolerance_).seconds()), NULL)) + { + can_transform = false; + break; + } + } } + } else { + can_transform = false; } - } else { - can_transform = false; - } - if (can_transform) { - TF2_ROS_MESSAGEFILTER_DEBUG( - "Message ready in frame %s at time %.3f, count now %d", - frame_id.c_str(), stamp.seconds(), messages_.size()); + if (can_transform) { + TF2_ROS_MESSAGEFILTER_DEBUG( + "Message ready in frame %s at time %.3f, count now %d", + frame_id.c_str(), stamp.seconds(), messages_.size()); - ++successful_transform_count_; - messageReady(saved_event); - } else { - ++dropped_message_count_; + ++successful_transform_count_; + messageReady(saved_event); + } else { + ++dropped_message_count_; - TF2_ROS_MESSAGEFILTER_DEBUG( - "Discarding message in frame %s at time %.3f, count now %d", - frame_id.c_str(), stamp.seconds(), messages_.size()); - messageDropped(saved_event, error); + TF2_ROS_MESSAGEFILTER_DEBUG( + "Discarding message in frame %s at time %.3f, count now %d", + frame_id.c_str(), stamp.seconds(), messages_.size()); + messageDropped(saved_event, error); + } } } ``` - https://github.com/ros-planning/navigation2.git ```patch diff --git a/nav2_amcl/include/nav2_amcl/amcl_node.hpp b/nav2_amcl/include/nav2_amcl/amcl_node.hpp index e2446e0c..179cedb4 100644 --- a/nav2_amcl/include/nav2_amcl/amcl_node.hpp +++ b/nav2_amcl/include/nav2_amcl/amcl_node.hpp @@ -156,7 +156,7 @@ protected: */ void initMessageFilters(); std::unique_ptr> laser_scan_sub_; - std::unique_ptr> laser_scan_filter_; + std::shared_ptr> laser_scan_filter_; message_filters::Connection laser_scan_connection_; // Publishers and subscribers diff --git a/nav2_amcl/src/amcl_node.cpp b/nav2_amcl/src/amcl_node.cpp index 293bbdb4..723dd6c7 100644 --- a/nav2_amcl/src/amcl_node.cpp +++ b/nav2_amcl/src/amcl_node.cpp @@ -1229,7 +1229,7 @@ AmclNode::initMessageFilters() laser_scan_sub_ = std::make_unique>( rclcpp_node_.get(), scan_topic_, rmw_qos_profile_sensor_data); - laser_scan_filter_ = std::make_unique>( + laser_scan_filter_ = std::make_shared>( *laser_scan_sub_, *tf_buffer_, odom_frame_id_, 10, rclcpp_node_, transform_tolerance_); laser_scan_connection_ = laser_scan_filter_->registerCallback( ``` - https://github.com/SteveMacenski/slam_toolbox.git (optional) ```patch diff --git a/include/slam_toolbox/slam_toolbox_common.hpp b/include/slam_toolbox/slam_toolbox_common.hpp index 8e43a6e..be4dd6c 100644 --- a/include/slam_toolbox/slam_toolbox_common.hpp +++ b/include/slam_toolbox/slam_toolbox_common.hpp @@ -127,7 +127,7 @@ protected: std::unique_ptr tfL_; std::unique_ptr tfB_; std::unique_ptr> scan_filter_sub_; - std::unique_ptr> scan_filter_; + std::shared_ptr> scan_filter_; std::shared_ptr> sst_; std::shared_ptr> sstm_; std::shared_ptr> ssMap_; diff --git a/src/slam_toolbox_common.cpp b/src/slam_toolbox_common.cpp index 8d01022..a27bec0 100644 --- a/src/slam_toolbox_common.cpp +++ b/src/slam_toolbox_common.cpp @@ -208,7 +208,7 @@ void SlamToolbox::setROSInterfaces() std::make_unique>( shared_from_this().get(), scan_topic_, rmw_qos_profile_sensor_data); scan_filter_ = - std::make_unique>( + std::make_shared>( *scan_filter_sub_, *tf_, odom_frame_, 1, shared_from_this(), tf2::durationFromSec(transform_timeout_.seconds())); scan_filter_->registerCallback( ```

This patch does not extend the lifecycle of MessageFilter but has a limitation. (The MessageFilter must be created by shared_ptr).

clalancette commented 2 years ago

@iuhilnehc-ynos Thanks so much for taking a look at this!

To test your code, I first recompiled all of the ROS 2 core from source, and navigation2 from source. With that configuration (and the reproducer provided in the first comment), I was able to reproduce the error.

I then applied your code to geometry2 and to navigation2 (I wasn't using slam_toolbox, so I didn't test that code). I then recompiled both the ROS 2 core and navigation2. With that code in place, I was not able to reproduce the crash after trying 10 times locally. This doesn't guarantee that it is fixed, but it is a really good indication.

@iuhilnehc-ynos Do you mind opening PRs for these?

iuhilnehc-ynos commented 2 years ago

@clalancette

With that code in place, I was not able to reproduce the crash after trying 10 times locally. This doesn't guarantee that it is fixed, but it is a really good indication.

Thank you for your confirmation. Before opening PRs, I need to try to figure out the first method which seems ideal. (The geometry2 implementation in ROS2 is a little different from ROS1)

iuhilnehc-ynos commented 2 years ago

I have done the first method. Even if it seems to work well, I need to check it more. The basic idea is as follows,

NOTE: not to use std::pair<TransformStampedFuture, tf2::TransformableRequestHandle>

from

using TransformStampedFuture = std::shared_future<geometry_msgs::msg::TransformStamped>;

to

class TransformStampedFuture : public std::shared_future<geometry_msgs::msg::TransformStamped> {
 ...
 private:
  tf2::TransformableRequestHandle handle_;
}
  virtual void
  cancel(TransformStampedFuture& ts_future) = 0;
iuhilnehc-ynos commented 2 years ago

@clalancette

Could you help review these two PRs above?