Closed ryanppeters closed 1 year ago
Hi @ryanppeters
thank you for the pull request. I quickly analyzed the modifications and everything seems well integrated.
I must ask you for more information about the frame_id
field of the metadata and how you initialize it.
Regarding instead the ROI, I saw that you used a basic rectangle to define it. This can be good for a first implementation, but in the future, we could change it into a more generic polygon like we already did in the ROS2 wrapper:
Setting: https://github.com/stereolabs/zed-ros2-wrapper/blob/master/zed_wrapper/config/common.yaml#L22-L25 ROI creation: https://github.com/stereolabs/zed-ros2-wrapper/blob/master/zed_components/src/tools/src/sl_tools.cpp#L420-L478
I must ask you for more information about the
frame_id
field of the metadata and how you initialize it.
frame_id
is set from the return value of GST_BUFFER_OFFSET(buf)
which is "a media specific offset for the buffer data. For video frames, this is the frame number of this buffer..." (source). We see this holds true for ZedSrc
at GstZedSrc Lines2509-210. The GST_BUFFER_OFFSET(buf)
call occurs at GstZedSrc.cpp Lines2513-2514 with offset
being the value for the frame_id
to be set in the gst_buffer_add_zed_src_meta(...)
. We choose to keep calculating the frame_id
in GstZedSrc
rather than GstZedSrcMeta
for clarity relating to (1) passing all GstZedSrcMeta
fields at once and (2) getting the offset just after it was updated inside ZedSrc
by GstZedSrc Lines2509-210.
Unfortunately, some NVidia Elements (we believe to be either NvStreamMux
or NvInfer
) are modifying the offset
member in GstBuffer
such that we cannot reliably use GST_BUFFER_OFFSET(...)
to track GstBuffer
s through the pipeline, hence the need to add this in GstZedSrcMeta
instead.
@ryanppeters fixed and merged into master
.
Thank you for the great PR :+1:
Thanks for merging and glad we can help!
I did some additional testing with Deepstream and determined that it is NvStreamMux
that changes the GstBuffer
offset (sets it to GST_BUFFER_OFFSET_NONE
). This makes sense since it is batching together 1+ buffers into a new GstBuffer
. Although we only use one source (ZED2i) in our use case, NvStreamMux
still batches it causing the offset to change.
I will add that after doing more research and testing, I discovered that the NvDsFrameMeta
from the output NvDsBatchMeta
of NvStreamMux
has a field frame_num
that could be potentially used to track the buffers without needing the extra frame_id
in GstZedSrcMeta
. The GstZedSrcMeta.frame_id
and NvDsFrameMeta.frame_num
solutions both work well for single source pipelines with Deepstream, but tend to break down if multiple sources (specifically multiple ZED cameras) are used, which is a problem for the future.
The frame_id
field is almost hidden for most of the users and if you think that it's better to remove it please feel free to submit a new PR.
This is a feature for advanced users and only real tests on the field can justify its presence or not.
Updates the plugin to add the new features from the the ZED SDK v3.8 release, namely:
Camera::setRegionOfInterest
function as thezedsrc
propertiesroi
,roi-x
,roi-y
,roi-w
, androi-h
ObjectDetectionParameters::allow_reduced_precision_inference
as thezedsrc
propertyod-allow-reduced-precision-inference
PositionalTrackingParameter::set_floor_as_origin
as thezedsrc
propertyset-gravity-as-origin
PositionalTrackingParameters::depth_min_range
as thezedsrc
propertypos-depth-min-range
as the
zedsrcproperty
od-prediction-timeout-s`Adds a
frame_id
field toGstZedSrcMeta
in order to track the meta/buffer throughout the GStreamer pipeline (when working with source code).Fixes a bug in the initial world transform that input roll, pitch, yaw in the same order regardless of the coordinate system being used
Notes:
zedsrc
parameters from the API reference but some of them may still not be the bestframe_id
could probably be improved and welcome any redesign of it, but we find it necessary for our code to have the ability to track buffers throughout the pipeline. We work with NVidia Deepstream via CPP and found that some of Nvidia's GStreamer elements "mess-up" the information in the base GstBuffer which prevents us from using that to track the Buffer throughout the pipeline