VideoStreaming Buffer issue

[leochan2009]

Hi all, I found that the current circular buffer is not suitable for video steaming purpose, as it is "last come first serve". For example the server encodes frame 1,2,3,4,5...etc and send to the client, the client might have only decoded the frame 1,3,5, which resulted in distorted image frames. It become even worse if key frame is missed.

To achieve successful decoding and only display the latest video frame. The connector should process the message buffer in this way:

1. The size of circular buffer should be larger(current size is 3), so no key frame message is washed away.
2. The circular buffer should be able to perform "the first come first serve".
3. Once the PeriodicProcess() function in connector executes, it should decode all the video messages in the circular buffer and display the latest frame only.

Any recommendation? @drouin-simon , i remember a college of you Etienne Léger successfully decoded the video on the iPad, what kind of mechanism he used? it would be great to learn from his experience.

Best, Longquan

[lassoan]

The size of circular buffer should be larger

I agree. For continuous real-time applications, we typically need minimum latency, so we only need the latest frame and for that we need to keep the latest P frame and all I frames since then. Any time when a new P frame arrives we can wipe all earlier frames from the buffer. It should not be difficult to have dynamic buffer size to keep these frames.

The circular buffer should be able to perform "the first come first serve".

I'm not sure if it is a priority or needed at all. I would say lossless bulk (non-real-time) data transfer is out of our scope. For that, you can use common file transfer protocols (sftp, ...) or simply files (if on the same computer).

Once the PeriodicProcess() function in connector executes, it should decode all the video messages in the circular buffer and display the latest frame only.

I'm not sure if you really mean all - if there are undecoded frames before the latest P frame then it should be safe to ignore those.

[leochan2009]

Hi Andras, @lassoan

I agree with you about the necessary of first come first server mode.

Regarding the point 3, the decoder needs to decode all these frames from the latest Key frame to the latest current frame. The frame before the latest Key frame could be ignored.

however, the current buffering mechanism, we don't unpack the body part of the message when receiving the data, only the header is unpacked, the message body is pushed to the buffer. So we could only get the frame type after the message gets pull out from the buffer and unpacked in the "igtlmessage to VTK" converting process. Also in most cases, when the converting process is pulling from the buffer, the buffer contains no key frame. As the decoder is very fast(up to 30 frame per second for Full HD video under lossless mode and 200 frame per second for near lossless mode), I would suggest to decode all the frames that is available.

Best, Longquan

[drouin-simon]

Hi Longquan,

I believe Etienne used H.264 on the ipad. Don't know if he gets hardware acceleration.

Andras, you remember Etienne? he was in Italy this summer. His application is really cool. It grabs the video from the ipad camera and optionally sends it to the Ibis workstation (through OpenIGTLink). We have a tracker attached to the ipad so a plugin in Ibis grabs the tracking info, computes the augmented image and sends it back to the ipad. Then the ipad app composites the augmentation with the video image and display the AR on the ipad screen.

The app is written in Qt, so it could easily run on Android devices as well (I believe it did at some point but hasn't been maintained).

We should add Etienne to the IGSIO organization so that he can take part in the conversation. I don't seem to have the permissions to do that. Andras, can you add him (@errollgarner) or upgrade my permissions?

I'm trying to convince Etienne's supervisor (Marta Kersten) to send him to PW again. She just started as a prof, so it is always a big investment for her. He would be a good asset to have at PW.

[leochan2009]

Hi Simon, @drouin-simon Thanks for the information. I will contact him about more details I would be pleased to see Etienne and his application in the PW, please do convince his supervisor:)

Best, Longquan