search

openframeworks / openFrameworks

openFrameworks is a community-developed cross platform toolkit for creative coding in C++.
http://openframeworks.cc
Other
9.94k stars 2.55k forks source link

ofGstVideoPlayer doesn't handle HLS video stream size changes, and sometimes crashes on change. #5308

Open bakercp opened 7 years ago

bakercp commented 7 years ago

I'm currently streaming from this HLS playlist.

http://iphone-streaming.ustream.tv/uhls/17074538/streams/live/iphone/playlist.m3u8

grabber.setPlayer(std::make_shared<ofGstVideoPlayer>());
grabber.load("http://iphone-streaming.ustream.tv/uhls/17074538/streams/live/iphone/playlist.m3u8");
grabber.play();

It's an HLS playlist. Initially it loads the first listed stream (RESOLUTION=854x480) and after a few seconds it (appears) to jump to the higher res stream (RESOLUTION=1280x720). But the pixels / texture do not appear to resize. The result is the 1280x720 video cropped to 854x480.

In some cases, which I have not been able to reproduce reliably (but I believe it has something to do with this resizing behavior), I will get back access exceptions in the buffer_cb when gst is trying to write an aligned pixels buffer to ofPixels.

There are plenty of interesting unhandled messages coming from ofGstUtils::gstHandleMessage() including GST_MESSAGE_STREAM_STATUS, GST_MESSAGE_ELEMENT, GST_MESSAGE_STREAM_START and others.

I'm happy to dig into this, but, wondering where to dig into this one. It seems to me that when a new buffer is processed, we should be able to simply inspect the meta-data and reallocate the player's ofPixels / ofTexture to match, but I'm not sure. Any thoughts are appreciated.

bakercp commented 7 years ago

By the way, my current work around is parse the m3u8 stream to get the link to the HD stream ... but this seems a little brittle since this link can change.

arturoc commented 7 years ago

i dont' think the current implementation of ofGstVideo* or even ofVideoGrabber and Player support resizing while the movie is playing, it always assumes the video will remain the same size for the whole lifetime of the object or until it's closed.

as you point out a way to implement the change in the ofGstVideo* side would be to watch the incomming messages https://github.com/openframeworks/openFrameworks/blob/master/libs/openFrameworks/video/ofGstUtils.cpp#L1255 for a size change and change the size of the ofPixels accordingly but it might be easier to just do so in the function that processes the incoming buffers https://github.com/openframeworks/openFrameworks/blob/master/libs/openFrameworks/video/ofGstUtils.cpp#L1255 or in update in case the pipeline is set to frame by frame mode.

you'll need to take into account that the size of the real buffer might be different than the size of the video because of padding so just comparing the size with the already allocated pixels might not be enough.

probably a combination of both ideas is the safest, when a message arrives signaling a change in size, enable a flag to reallocate the pixels in the next frame

then in the ofVideoGrabber/Player size probably just checking that the size of the current pixels and texture is the same would and reallocating if not be enough