dusty-nv / jetson-utils

C++/CUDA/Python multimedia utilities for NVIDIA Jetson
MIT License
737 stars 289 forks source link

VideoOutput.Render() to webrtc failure using appsink frame #188

Closed Hongfeng-mtec closed 11 months ago

Hongfeng-mtec commented 11 months ago

I am trying to use appsink to get a video frame, which I convert to cudaImage, then try to stream it to webrtc. The code is largely like this:

#pipeline: ... ! videoconvert ! video/x-raw,format=RGB ! appsink

self.output = videoOutput('webrtc://@:8554/live')

def __on_new_sample(self,app_sink):
        sample = app_sink.emit("pull-sample") 
        caps = sample.get_caps()
        height = caps.get_structure(0).get_value("height")
        width = caps.get_structure(0).get_value("width")
        buffer = sample.get_buffer()
        numpy_frame = np.ndarray(
            shape=(height, width, 3),
            dtype=np.uint8,
            buffer=buffer.extract_dup(0, buffer.get_size()))
        cuda_img= cudaFromNumpy(numpy_frame)
        print(cuda_img)
        self.output.Render(cuda_img)

Now the cudaImage object looks legit, but I am get following error on the render function.

<cudaImage object>
   -- ptr:      0x102360000
   -- size:     2764800
   -- width:    1280
   -- height:   720
   -- channels: 3
   -- format:   rgb8
   -- mapped:   true
   -- freeOnDelete: true
   -- timestamp:    0.000000

TypeError: can't convert return value to desired type

If I output the video to v4l2sink, then run video_viewer using input from the loopback device, it works fine. L4T: 32.7.4, Hardware: TX2

Hongfeng-mtec commented 11 months ago

another data point: if I just modify python/examples/cuda_from_numpy.py, and append

output = videoOutput('webrtc://@:8555/live')  
while True:
    output.Render(cuda_mem)
    time.sleep(0.1)

that script works fine, I am wondering what's the difference here.

Hongfeng-mtec commented 11 months ago

it turns out nothing related to videoOutput Render(), if I add "return 0" to the new-sample callback, the error message goes away. sorry for the false alarm.