imxv4l2videosrc: possible threading problem

CPU: IMX6 DualLite Used camera: OV7740 Similar results when using ION and IPU allocator

Kernel version: 5.4.70-F+S gstreamer (and plugins) version: 1.16.2.imx-r0 gstreamer1.0-plugins-imx version: 2.1.0 (rev d14c9b4dce2062a58246740e7399d423a28a34a4) libimxvpuapi version: 2.2.2 libimxdmabuffer version: 1.1.2

All build with yocto 3.0.4 (Zeus)

Consider the following pipeline (called from a bash script), which splits the source into three sinks: (Some parameters, e.g. $mjpg_framerate, are passed to the shell script).

gst-launch-1.0 --gst-debug-level=3 imxv4l2videosrc device=/dev/video0  ! videorate ! video/x-raw,framerate=15/1 ! videoconvert ! video/x-raw,format=YUY2 ! tee name=tp \
    tp. ! queue ! v4l2sink device=/dev/video1 \
    tp. ! queue ! videoscale ! video/x-raw,width=$mjpg_width,height=$mjpg_height ! videorate ! video/x-raw,framerate=$mjpg_framerate ! v4l2sink device=/dev/video2 \
    tp. ! queue ! videoscale ! video/x-raw,width=$rtsp_width,height=$rtsp_height ! videorate ! video/x-raw,framerate=$rtsp_framerate ! v4l2sink device=/dev/video3 &

After a few frames (varying in amount; between 1 and typically 30) , this pipeline crashes with a SEGFAULT.

The line where it crashes is in video-frame.c in the gstreamer1.0-plugins base, where the source frame is copied to the destination:

    for (j = 0; j < h; j++) {
      memcpy (dp, sp, w); // This line crashes
      dp += ds;
      sp += ss;
    }

The corresponding backtrace can be found here:

#0  memcpy () at ../sysdeps/arm/armv7/multiarch/memcpy_impl.S:473
No locals.
#1  0x769c9366 in gst_video_frame_copy_plane (dest=0x75afdd9c, src=0x75afdc2c, plane=0)
    at ../../../../git/gst-libs/gst/video/video-frame.c:386
        j = 0
        sinfo = 0x75afdc2c
        dinfo = 0x75afdd9c
        finfo = 0x76a2214c <formats+756>
        sp = 0x72ca5000 <error: Cannot access memory at address 0x72ca5000>
        dp = 0x7353c000 "\020|\016\200\v~\n\200\t\177\t\200\t\177\t\200\t\200\t\200\t\200\t\200\t\200\b\200\b\200\b\200\b\200\b\200\b\200\t\200\t\200\t\200\t\200\t\177\t\200\t\200\t\200\t\200\b\200\b\200\t\177\b\200\b\177\t\200\t\177\t\200\t\177\t\200\n\200\n\200\n\200\n\200\n\200\v\200\v\200\v\177\v\200\f~\f\200\f}\f\200\f~\f\200\r\177\r\200\r\177\016\200\017\177\020~\023\200\026{\032\200 w%\200+s2\200\071q?\200FnM\200Un]\200_nd}lnu~xj{~\177d\203\200\203a\205\200\210a\210\200\207a\214\200\214a\215\200\216a\215\200\220b\222\200\222c\223\177\223b\226\177\226c\226\200\227c\231\200"...
        w = 1280
        h = 480
        ss = 1280
        ds = 1280
        __func__ = "gst_video_frame_copy_plane"
        __PRETTY_FUNCTION__ = "gst_video_frame_copy_plane"
#2  0x769c9482 in gst_video_frame_copy (dest=0x75afdd9c, src=0x75afdc2c)
    at ../../../../git/gst-libs/gst/video/video-frame.c:424
        i = 0
        n_planes = 1
        sinfo = 0x75afdc2c
        dinfo = 0x75afdd9c
        __func__ = "gst_video_frame_copy"
#3  0x76765632 in gst_v4l2_buffer_pool_copy_buffer (pool=0x74708600, dest=0x749060a8, src=0x432e90)
    at ../../../git/sys/v4l2/gstv4l2bufferpool.c:134
        src_frame = {info = {finfo = 0x76a2214c <formats+756>, interlace_mode = GST_VIDEO_INTERLACE_MODE_PROGRESSIVE,
            flags = GST_VIDEO_FLAG_NONE, width = 640, height = 480, size = 614400, views = 1,
            chroma_site = GST_VIDEO_CHROMA_SITE_UNKNOWN, colorimetry = {range = GST_VIDEO_COLOR_RANGE_16_235,
              matrix = GST_VIDEO_COLOR_MATRIX_BT601, transfer = GST_VIDEO_TRANSFER_SRGB,
              primaries = GST_VIDEO_COLOR_PRIMARIES_BT709}, par_n = 1, par_d = 1, fps_n = 15, fps_d = 2, offset = {0, 0, 0,
              0}, stride = {1280, 0, 0, 0}, ABI = {abi = {multiview_mode = GST_VIDEO_MULTIVIEW_MODE_NONE,
                multiview_flags = GST_VIDEO_MULTIVIEW_FLAGS_NONE, field_order = GST_VIDEO_FIELD_ORDER_UNKNOWN},
              _gst_reserved = {0xffffffff, 0x0, 0x0, 0x0}}}, flags = GST_VIDEO_FRAME_FLAG_NONE, buffer = 0x432e90,
          meta = 0x74906950, id = 0, data = {0x72ca5000, 0x1, 0x5c4110, 0x41ede8}, map = {{memory = 0x747020b8,
              flags = GST_MAP_READ, data = 0x72ca5000 <error: Cannot access memory at address 0x72ca5000>, size = 614400,
              maxsize = 614400, user_data = {0x50e248, 0x76807870, 0x75afdcf8,
                0x76a9b311 <gst_base_transform_src_eventfunc+196>}, _gst_reserved = {0x50b9a39d, 0x0, 0x74700010,
                0x76ccfa85 <__GI___libc_realloc+456>}}, {memory = 0x3, flags = (unknown: 128),
              data = 0x1 <error: Cannot access memory at address 0x1>, size = 0, maxsize = 4179808010, user_data = {0x49d,
                0x75afdd20, 0x76f6b25f <gst_util_uint64_scale+42>, 0x1}, _gst_reserved = {0x0, 0x0, 0x0, 0x1}}, {
              memory = 0x0, flags = (GST_MAP_WRITE | unknown: 4179808008),
              data = 0x49d <error: Cannot access memory at address 0x49d>, size = 1974459712, maxsize = 1995447699,
              user_data = {0x1, 0x0, 0xf922cf0a, 0x49d}, _gst_reserved = {0x5c2c00, 0x5cb138, 0xf922cf0a, 0x49d}}, {
              memory = 0x75afdd80, flags = (GST_MAP_READ | GST_MAP_WRITE | unknown: 1995447920), data = 0x0, size = 0,
              maxsize = 0, user_data = {0x0, 0x1, 0x0, 0x1}, _gst_reserved = {0x0, 0xf922cf0a, 0x49d, 0x0}}},
          _gst_reserved = {0x5cb138, 0x76a9b209 <gst_base_transform_src_event>, 0x5cb0b8, 0x0}}
        dest_frame = {info = {finfo = 0x76a2214c <formats+756>, interlace_mode = GST_VIDEO_INTERLACE_MODE_PROGRESSIVE,
            flags = GST_VIDEO_FLAG_NONE, width = 640, height = 480, size = 614400, views = 1,
            chroma_site = GST_VIDEO_CHROMA_SITE_UNKNOWN, colorimetry = {range = GST_VIDEO_COLOR_RANGE_16_235,
              matrix = GST_VIDEO_COLOR_MATRIX_BT601, transfer = GST_VIDEO_TRANSFER_SRGB,
              primaries = GST_VIDEO_COLOR_PRIMARIES_BT709}, par_n = 1, par_d = 1, fps_n = 15, fps_d = 2, offset = {0, 0, 0,
              0}, stride = {1280, 0, 0, 0}, ABI = {abi = {multiview_mode = GST_VIDEO_MULTIVIEW_MODE_NONE,
                multiview_flags = GST_VIDEO_MULTIVIEW_FLAGS_NONE, field_order = GST_VIDEO_FIELD_ORDER_UNKNOWN},
              _gst_reserved = {0xffffffff, 0x0, 0x0, 0x0}}}, flags = GST_VIDEO_FRAME_FLAG_NONE, buffer = 0x749060a8,
          meta = 0x75b04650, id = 0, data = {0x7353c000, 0x58ac24f2, 0x0, 0x3}, map = {{memory = 0x74906c70,
              flags = GST_MAP_WRITE

,
              data = 0x7353c000 "\020|\016\200\v~\n\200\t\177\t\200\t\177\t\200\t\200\t\200\t\200\t\200\t\200\b\200\b\200\b\200\b\200\b\200\b\200\t\200\t\200\t\200\t\200\t\177\t\200\t\200\t\200\t\200\b\200\b\200\t\177\b\200\b\177\t\200\t\177\t\200\t\177\t\200\n\200\n\200\n\200\n\200\n\200\v\200\v\200\v\177\v\200\f~\f\200\f}\f\200\f~\f\200\r\177\r\200\r\177\016\200\017\177\020~\023\200\026{\032\200 w%\200+s2\200\071q?\200FnM\200Un]\200_nd}lnu~xj{~\177d\203\200\203a\205\200\210a\210\200\207a\214\200\214a\215\200\216a\215\200\220b\222\200\222c\223\177\223b\226\177\226c\226\200\227c\231\200"..., size = 614400, maxsize = 614400,
              user_data = {0x0, 0x0, 0x0, 0x0}, _gst_reserved = {0x0, 0x3ff00000, 0x75afde80,
                0x76f5175f <gst_segment_to_stream_time+310>}}, {memory = 0x75afde90,
              flags = (GST_MAP_READ | GST_MAP_FLAG_LAST | unknown: 1995471196)
,
              data = 0x5f <error: Cannot access memory at address 0x5f>, size = 60, maxsize = 1, user_data = {0x75afe038,
                0x74906cc0, 0x5d4e98, 0x58ac24f2}, _gst_reserved = {0x1, 0x74902400, 0x76fd351c, 0x75afde98}}, {
              memory = 0x76f42a79 <gst_poll_read_control+76>, flags = (GST_MAP_READ | GST_MAP_WRITE | unknown: 12),
              data = 0x5d4e98 "", size = 4090566769, maxsize = 1, user_data = {0x75afdec0,
                0x76ef71ed <default_acquire_buffer+96>, 0x17, 0x76e8541b <g_object_unref+50>}, _gst_reserved = {0x76fd351c,
                0x76fd351c, 0x50e248, 0x75afdfb0}}, {memory = 0x75afdf7c, flags = (unknown: 1953531392), data = 0x0,
              size = 1953531296, maxsize = 0, user_data = {0x76fd351c, 0x74708600, 0x76e852b7 <g_object_ref+102>,
                0x76fd351c}, _gst_reserved = {0x749060a8, 0x50e248, 0x76ee4a9d <gst_object_ref+108>, 0x75afdf18}}},
          _gst_reserved = {0x75afdfb0, 0x75afdf7c, 0x74708600, 0x0}}
        finfo = 0x76a2214c <formats+756>
        __func__ = "gst_v4l2_buffer_pool_copy_buffer"

#4  0x76765ff0 in gst_v4l2_buffer_pool_prepare_buffer (pool=0x74708600, dest=0x749060a8, src=0x432e90)
    at ../../../git/sys/v4l2/gstv4l2bufferpool.c:399
        ret = GST_FLOW_OK
        own_src = 0
        __func__ = "gst_v4l2_buffer_pool_prepare_buffer"

#5  0x767697f0 in gst_v4l2_buffer_pool_process (pool=0x74708600, buf=0x75afe010)
    at ../../../git/sys/v4l2/gstv4l2bufferpool.c:2067
        params = {format = GST_FORMAT_UNDEFINED, start = 0, stop = 0, flags = GST_BUFFER_POOL_ACQUIRE_FLAG_DONTWAIT,
          _gst_reserved = {0x0, 0x0, 0x0, 0x0}}
        to_queue = 0x749060a8
        buffer = 0x50b9a39d
        group = 0x3
        index = 5998624
        ret = GST_FLOW_OK
        bpool = 0x74708600
        obj = 0x58f200
        __func__ = "gst_v4l2_buffer_pool_process"

#6  0x7676b4c2 in gst_v4l2sink_show_frame (vsink=0x5b89e8, buf=0x432e90) at ../../../git/sys/v4l2/gstv4l2sink.c:609
        ret = GST_FLOW_OK
        v4l2sink = 0x5b89e8
        obj = 0x58f200
        bpool = 0x74708600
        __func__ = "gst_v4l2sink_show_frame"

#7  0x769d38dc in gst_video_sink_show_frame (bsink=0x5b89e8, buf=0x432e90)
    at ../../../../git/gst-libs/gst/video/gstvideosink.c:229
        klass = 0x53ae00
        __PRETTY_FUNCTION__ = "gst_video_sink_show_frame"

#8  0x76a89968 in gst_base_sink_chain_unlocked (basesink=0x5b89e8, pad=0x5c00e0, obj=0x432e90, is_list=0)
    at ../../../../git/libs/gst/base/gstbasesink.c:3649
        bclass = 0x53ae00
        priv = 0x5b8820
        ret = GST_FLOW_OK
        start = 1354343325
        end = 1487676658
        segment = 0x5b8ae0
        sync_buf = 0x432e90
        late = 0
        step_end = 0
        prepared = 0
        __func__ = "gst_base_sink_chain_unlocked"
        __PRETTY_FUNCTION__ = "gst_base_sink_chain_unlocked"

#9  0x76a8a33a in gst_base_sink_chain_main (basesink=0x5b89e8, pad=0x5c00e0, obj=0x432e90, is_list=0)
    at ../../../../git/libs/gst/base/gstbasesink.c:3775
        result = 1996305692
        __PRETTY_FUNCTION__ = "gst_base_sink_chain_main"

#10 0x76a8a440 in gst_base_sink_chain (pad=0x5c00e0, parent=0x5b89e8, buf=0x432e90)
    at ../../../../git/libs/gst/base/gstbasesink.c:3804
        basesink = 0x5b89e8
#11 0x76f31d9a in gst_pad_chain_data_unchecked () from target:/usr/lib/libgstreamer-1.0.so.0
No symbol table info available.

#12 0x76f32644 in gst_pad_push_data () from target:/usr/lib/libgstreamer-1.0.so.0
No symbol table info available.

#13 0x76f32aba in gst_pad_push () from target:/usr/lib/libgstreamer-1.0.so.0
No symbol table info available.

#14 0x76a9c310 in gst_base_transform_chain (pad=0x5c4110, parent=0x5c2c00, buffer=0x432e90)
    at ../../../../git/libs/gst/base/gstbasetransform.c:2330
        position_out = 666666666
        trans = 0x5c2c00
        klass = 0x52f400
        priv = 0x5c2b70
        ret = GST_FLOW_OK
        position = 666666666
        timestamp = 533333333
        duration = 133333333
        outbuf = 0x432e90
        __PRETTY_FUNCTION__ = "gst_base_transform_chain"
#15 0x76f31d9a in gst_pad_chain_data_unchecked () from target:/usr/lib/libgstreamer-1.0.so.0
No symbol table info available.

#16 0x76f32644 in gst_pad_push_data () from target:/usr/lib/libgstreamer-1.0.so.0
No symbol table info available.

#17 0x76f32aba in gst_pad_push () from target:/usr/lib/libgstreamer-1.0.so.0
No symbol table info available.

#18 0x76821740 in gst_video_rate_push_buffer (videorate=0x5aa868, outbuf=0x432e90, duplicate=1, next_intime=1400000000)
    at ../../../git/gst/videorate/gstvideorate.c:719
        res = GST_FLOW_OK
        push_ts = 533333333
        __PRETTY_FUNCTION__ = "gst_video_rate_push_buffer"

#19 0x768217dc in gst_video_rate_flush_prev (videorate=0x5aa868, duplicate=1, next_intime=1400000000)
    at ../../../git/gst/videorate/gstvideorate.c:738
        outbuf = 0x432e90
        __PRETTY_FUNCTION__ = "gst_video_rate_flush_prev"

#20 0x76824976 in gst_video_rate_transform_ip (trans=0x5aa868, buffer=0x74906348)
    at ../../../git/gst/videorate/gstvideorate.c:1597
        r = GST_FLOW_OK
        next_ts = 533333333
        prevtime = 200000000
        count = 3
        diff1 = 333333333
        diff2 = 866666667
        videorate = 0x5aa868
        res = GST_FLOW_CUSTOM_SUCCESS
        intime = 1400000000
        in_ts = 1400000000
        in_dur = 66666666
        last_ts = 266666666
        avg_period = 0
        skip = 0
        __PRETTY_FUNCTION__ = "gst_video_rate_transform_ip"
        __func__ = "gst_video_rate_transform_ip"

#21 0x76a9bbca in default_generate_output (trans=0x5aa868, outbuf=0x75afe78c)
    at ../../../../git/libs/gst/base/gstbasetransform.c:2136
        bclass = 0x5a0c00
        priv = 0x5aa7d8
        ret = GST_FLOW_OK
        inbuf = 0x433150
        want_in_place = 1
        __PRETTY_FUNCTION__ = "default_generate_output"

#22 0x76a9c19e in gst_base_transform_chain (pad=0x5bacd0, parent=0x5aa868, buffer=0x433150)
    at ../../../../git/libs/gst/base/gstbasetransform.c:2294
        trans = 0x5aa868
        klass = 0x5a0c00
        priv = 0x5aa7d8
        ret = GST_FLOW_OK
        position = 1466666666
        timestamp = 1400000000
        duration = 66666666
        outbuf = 0x74906348
        __PRETTY_FUNCTION__ = "gst_base_transform_chain"
#23 0x76f31d9a in gst_pad_chain_data_unchecked () from target:/usr/lib/libgstreamer-1.0.so.0
No symbol table info available.

#24 0x76f32644 in gst_pad_push_data () from target:/usr/lib/libgstreamer-1.0.so.0
No symbol table info available.

#25 0x76f32aba in gst_pad_push () from target:/usr/lib/libgstreamer-1.0.so.0
No symbol table info available.

#26 0x76a9c310 in gst_base_transform_chain (pad=0x5c0cf8, parent=0x5c29d0, buffer=0x433150)
    at ../../../../git/libs/gst/base/gstbasetransform.c:2330
        position_out = 1466666666
        trans = 0x5c29d0
        klass = 0x52f400
        priv = 0x5c2940
        ret = GST_FLOW_OK
        position = 1466666666
        timestamp = 1400000000
        duration = 66666666
        outbuf = 0x433150

        __PRETTY_FUNCTION__ = "gst_base_transform_chain"
#27 0x76f31d9a in gst_pad_chain_data_unchecked () from target:/usr/lib/libgstreamer-1.0.so.0
No symbol table info available.

#28 0x76f32644 in gst_pad_push_data () from target:/usr/lib/libgstreamer-1.0.so.0
No symbol table info available.

#29 0x76f32aba in gst_pad_push () from target:/usr/lib/libgstreamer-1.0.so.0
No symbol table info available.

#30 0x76a9c310 in gst_base_transform_chain (pad=0x5baa20, parent=0x5be568, buffer=0x433150)
    at ../../../../git/libs/gst/base/gstbasetransform.c:2330

        position_out = 1466666666
        trans = 0x5be568
        klass = 0x434850
        priv = 0x5be4d8
        ret = GST_FLOW_OK
        position = 1466666666
        timestamp = 1400000000
        duration = 66666666
        outbuf = 0x433150
        __PRETTY_FUNCTION__ = "gst_base_transform_chain"
#31 0x76f31d9a in gst_pad_chain_data_unchecked () from target:/usr/lib/libgstreamer-1.0.so.0
No symbol table info available.

#32 0x76f32644 in gst_pad_push_data () from target:/usr/lib/libgstreamer-1.0.so.0
No symbol table info available.

#33 0x76f32aba in gst_pad_push () from target:/usr/lib/libgstreamer-1.0.so.0
No symbol table info available.

#34 0x767d1b6a in gst_queue_push_one (queue=0x5b24e0) at ../../../git/plugins/elements/gstqueue.c:1384
        buffer = 0x433150
        result = GST_FLOW_OK
        data = 0x433150
        is_list = 0
        __PRETTY_FUNCTION__ = "gst_queue_push_one"

#35 0x767d261c in gst_queue_loop (pad=0x5ba8c8) at ../../../git/plugins/elements/gstqueue.c:1537
        queue = 0x5b24e0
        ret = GST_FLOW_OK
        __PRETTY_FUNCTION__ = "gst_queue_loop"
#36 0x76f61b22 in gst_task_func () from target:/usr/lib/libgstreamer-1.0.so.0
No symbol table info available.

#37 0x76f628e0 in default_func () from target:/usr/lib/libgstreamer-1.0.so.0
No symbol table info available.

Very probable cause for this crash is a threading problem when mapping / unmapping the buffer:

Take as example the imx_dma_buffer_ion_allocator_unmap in imxdmabuffer_ion_allocator.c (all part of libimxdmabuffer). Without locking, imx_ion_buffer->mapping_refcount--; could be called by two or more threads at the same time, resulting in a never unmapped buffer. This would cause the allocator to never be unallocated, resulting in an ever shrinking available CMA memory space. This was observed with our pipeline.

Likewise, imx_ion_buffer->mapping_refcount-- in imxdmabuffer_ion_allocator.c and imx_ion_buffer->mapping_refcount++ could conflict in a similar way, resulting in an unmapped buffer before the frame was copied, resulting in the SEGFAULT.

As a test (real patch can be found further below), we fixed this issue by adding a lock around imx_dma_buffer_unmap and imx_dma_buffer_map in gstimxdmabufallocator.c:

in gst_imx_dmabuf_allocator_mem_map_full:

    pthread_mutex_lock(&lock);
    mapped_virtual_address = imx_dma_buffer_map(imx_dma_buffer, flags, &error);
    pthread_mutex_unlock(&lock);

in gst_imx_dmabuf_allocator_mem_unmap_full:

    pthread_mutex_lock(&lock);
    imx_dma_buffer_unmap(imx_dma_buffer);
    pthread_mutex_unlock(&lock);

Other allocators would require locks in other places, e.g. in gstimxdefaultallocator.c. Alternatively, the issue could be fixed in the libimxdmabuffer repo.

This solved our issue: All three video streams runs smoothly without SEGFAULT.

At the moment, we solved our issue by applying the patch below to libimxdmabuffer:

0002-make-threadsafe.patch

I'm not sure if it would be better to either fix this issue in the gstreamer1.0-plugin-imx, or in the libimxdmabuffer. Either way, a lock resolved our issues.

The reason I'm posting this: Is this an issue that requires resolving by the maintainers of either this or the libimxdmabuffer repo, or are we doing something weird / wrong, resulting in this threading problem?

Freescale / gstreamer-imx

imxv4l2videosrc: possible threading problem #315