How to add jpegenc (to generate snapshot) and splitmuxsink (to generate video) together in a single pipeline?

[dhaval-zala-aivid]

@nnshah1

How to add jpegenc (to generate snapshot) and splitmuxsink (to generate video) together in a single pipeline? I am able to add jpegenc or splitmuxsink separately in pipeline and its running properly. But if I add both together in a single pipeline Its generate snapshot only and video with 0 byte in size. I have tried multiple combination for this issue. But not able to solve.

Here is the pipeline:

{
    "type": "GStreamer",
    "template": ["urisourcebin name=source ! decodebin ! video/x-raw ",
                    " ! videoconvert name=videoconvert",
                    " ! tee name=t ! queue",
                    " ! gvadetect model={models[object_detection][coco_yolov5_tiny_608to416_FP32][network]} name=detection",
                    " ! gvametaconvert name=metaconvert",
                    " ! jpegenc ! gvapython name=gvapython_n module=/home/pipeline-server/server/pplcount.py class=ImageCapture",
                    " ! appsink name=appsink",
                    " t. ! videoconvert ! x264enc ! splitmuxsink muxer=avimux location=\"/tmp/temp-%d.mp4\" max-size-time=30000000000"
                    ],
    "description": "Object detection pipeline extended to add frame count to meta-data and save frames to disk",
    "parameters": {
        "type": "object",
        "properties": {
            "detection-device": {
                "element": "detection",
                "type": "string"
            },
            "inference-interval": {
                "element": "detection",
                "type": "integer"
            },
            "add-empty-results": {
                "element": "metaconvert",
                "type": "boolean",
                "default": true
            },
            "max-files": {
                "element": "filesink",
                "type": "integer",
                "default": 1000
            },
            "recording_prefix": {
                "type": "string",
                "element": {
                    "name": "splitmuxsink",
                    "property": "location"
                },
                "default": "/home/pipeline-server"
            }
        }
    }
}

[mikhail-nikolskiy]

You can probably use jpegenc and multifilesink to save .jpg images, ex

! jpegenc ! multifilesink location=img_%06d.jpg

[dhaval-zala-aivid]

@mikhail-nikolskiy

I want to save jpg based on post-processing. I don't want to save each frames that's why I have used ImageCapture class in pipeline to save the particular snapshots only. What you have suggested will capture the image every frame is not required in my case.

[nnshah1]

There seems to be an interaction between the x264enc and the queue that stalls the pipeline (in my local experiments). Using vaapi elements I was able to get the output correctly. You may try putting a queue after the second t branch. The other thing to note is if you name the splitmuxsink element as splitmuxsink - pipeline server will set the location automatically to include the timestamp for the recording.

attaching the pipeline.json file and save_jpeg.py file for reference.

{
    "type": "GStreamer",
    "template": ["{auto_source} ! decodebin",
                        " ! tee name=t ! queue",
                " ! gvadetect model={models[object_detection][person_vehicle_bike][network]} name=detection",
                " ! gvametaconvert name=metaconvert ! jpegenc ! gvapython module=/home/pipeline-server/save_jpeg.py",
                        " ! appsink name=appsink",
                        " t. ! queue ! vaapipostproc ! vaapih264enc ! splitmuxsink name=splitmuxsink muxer=avimux max-size-time=30000" 
            ],
    "description": "Person Vehicle Bike Detection based on person-vehicle-bike-detection-crossroad-0078",
    "parameters": {
        "type": "object",
        "properties": {
            "detection-properties": {
                "element": {
                    "name": "detection",
                    "format": "element-properties"
                }
            },
            "detection-device": {
                "element": {
                    "name": "detection",
                    "property": "device"
                },
                "type": "string",
                "default": "{env[DETECTION_DEVICE]}"
            },
            "detection-model-instance-id": {
                "element": {
                    "name": "detection",
                    "property": "model-instance-id"
                },
                "type": "string"
            },
            "inference-interval": {
                "element": "detection",
                "type": "integer"
            },
            "threshold": {
                "element": "detection",
                "type": "number"
            },
            "recording_prefix": {
            "type":"string",
            "default": "/home/pipeline-server"
            }
        }
    }
}

from gstgva.util import gst_buffer_data
import gi
gi.require_version("Gst", "1.0")
# pylint: disable=wrong-import-position

from gi.repository import Gst

count = 0
def process_frame(frame):
    global count
    count +=1
    buffer = frame._VideoFrame__buffer
    with gst_buffer_data(buffer,Gst.MapFlags.READ) as data:
        filename = "frame-{}.jpeg".format(count%10)
        with open(filename,"wb",0) as output:
            output.write(data)
    return True