IGSIO / OpenIGTLinkIO

Library for interfacing to openigtlink/OpenIGTLink, dependent on VTK and Qt. Based on openigtlink/OpenIGTLinkIF
Apache License 2.0
13 stars 24 forks source link

VIdeo Streaming Merge #35

Open leochan2009 opened 6 years ago

leochan2009 commented 6 years ago

Dear OpenIGTLinkIO developers,

With the ever-increasing demands for video streaming, i think we should work on this issue again. A lot of discussions have been going on: OpenIGTLink SuperBuild Video Save and Reload To achieve the video save and reload in 3DSlicer, we need to update OpenIGTLinkIF module which depends on IO library. I made some commits in IO before, but was reverted because the OpenIGTLink library was not ready. The openigtlink library was improved in the last months regarding the Superbuild of video streaming libraries. Though openigtlink library still needs some improvements, i think it is a good time to dig into this issue before the project week.

Best, Longquan

drouin-simon commented 6 years ago

Hi Longquan,

Thanks for bringing this issue, it is very interesting. Compressed loading and saving of video sequence is a feature I've been wanting for a while. It also seems to me that it is useful in so many places that it should be in a library of its own (or perhaps in IO). That way it could be used in Slicer modules as well as in the Plus Toolkit. I would also use it in IBIS for a bunch of things. It can definitely be an important tool for the interoperability that we wanted when we created this group.

I don't know what functionality you had in mind. On my part, I would like a module that could stream the images to/from disk in a way that is transparent to the client application. You could both record a sequence directly to disk, play it back or scroll back and forth in the sequence without having to keep everything in the main memory. The library would handle the buffering. From the application's point of view, you would simply put in frames or recover the frame you want and, internally, the lib would handle buffering and encoding. Anyways, I'd like to know more about your goals.

I see you have a project about this in PW. I've created a project too called "Tracked US Standardization 5", thinking to continue the work we have been doing since the last couple of PW. It seems there is some overlap with yours. Maybe we can discuss how to merge these projects or make them collaborate since it is probably of interest to the same people.

Let me know what you think.

Cheers.

s.

leochan2009 commented 6 years ago

Hi Simon,

Thanks for your comments, Great that you have interests in video loading and saving. Yes, i think this part is very helpful in many scenario. We could probably create another library for this functionality. Currently a module called "Sequence" in Slicer is used for recording and replaying videos and tracking information, etc. Our original idea is just to incorporate the video compression into the module to reduce file sizes. I did some preliminary work in my forked branch and was able to reduce the video file size to only 2% of the original file(lossy compression), or around 5% to 10% for lossless compression. The challenge i came across is the replay part. The decoding of a single frame requires the information from other frames. for example, to replay frame Number 100, it will require the decode of frame number 98 and 99. We could find the best solution for this issue in the project week. I find it a great idea of loading the file into client application directly from disk without touching the main memory, and it is possible if we store the compressed bit stream in a way that is well structured. I have no objection to merge the project page to the Tracked US standardization 5 page, but let me confirm with other investigator.

Best, Longquan

leochan2009 commented 6 years ago

@lassoan , @Sunderlandkyl, @tokjun What do you think about merging our project page with the Tracked US standardization 5 ?

drouin-simon commented 6 years ago

Quick question: I was just looking at the code in Plus to read/write tracked video sequences to/from mha files and I was wondering whether it would be relevant to share some code between Plus and the Sequence extension. I guess it is useful to be able to load data recorded with Plus directly into Slicer through the Sequence extension. It might be already possible, but how do you make sure that compatibility is maintained between the 2?

leochan2009 commented 6 years ago

As long as the file is saved into a .mha file format, you should be able to load the file directly into Slicer, however the time information is lost. We could still compose a sequence node using the manual-set timestamps and replay the sequence. I haven't looked into the video sequence implementation in Plus. But i think as long as the time stamp is saved, we will be able to recover the data in Slicer, and vice versa.

adamrankin commented 6 years ago

I think Slicer's reading of MHA knows nothing about timestamps of frames. That's a header field that Plus has added.

leochan2009 commented 6 years ago

yes, in Slicer we use the mrmlscene to store the time stamps. if we could interpret the header field from Plus, then we should able to replay plus-saved video sequence in slicer.

lassoan commented 6 years ago

It's true that mha and nrrd reading/writing is duplicated between Plus and Slicer. There haven't been any library that was shared between Plus and Slicer that could contain such shared code. But maybe we could have a Plus core or some new shared library that could contain such things. Not just data I/O, but there is also some other useful, quite generic classes, such as transform repository, tracked frame list, etc.

drouin-simon commented 6 years ago

This was the original purpose of OpenIGTLinkIO. We started by extracting the code from OpenIGTLinkIF because it was already compatible with Plus, but the code is still essentially duplicated between IO and Plus (not the same code, but doing the same thing). The next logical step would be to merge the code of IO and Plus. Have a single package that reads, write, transmits and receives, which ensures all programs remain compatible whenever we change something.

My understanding is that at the moment, Plus and IO are quite far appart, so it may not be possible to merge them in the short term, but we should keep that long-term goal in mind and slowly move all functionality and definitions that may be interesting to share from Plus to the IO library.

I'm not familiar with the Sequence extension code, so my vision may be biased here.