search

PlusToolkit / PlusLib

Software library for data acquisition, pre-processing, and calibration for navigated image-guided interventions.
http://www.plustoolkit.org
Other
133 stars 102 forks source link

Unable to figure out how to use Video Streaming #464

Closed jamesobutler closed 5 years ago

jamesobutler commented 5 years ago

I was looking into compressed video streaming as shown in this PerkLab video. Video streaming for image messages was added in https://github.com/PlusToolkit/PlusLib/commit/af8d5fd8562a7b49a694a84ea72754e7712905bb.

I've tried adding something like the following with EncodingType="VP90" in my configuration with a general mmf webcam for testing, but I don't notice any change. I'm getting the same FPS as reported by NodeModifiedStatistics Slicer module and same performance as viewed in Windows Task Manager. Am I doing something wrong? Can video streaming be enabled for any stream sending image messages to Slicer? Is there a documentation page that I've missed?

  <PlusOpenIGTLinkServer 
    MaxNumberOfIgtlMessagesToSend="1" 
    MaxTimeSpentWithProcessingMs="50" 
    ListeningPort="18944" 
    SendValidTransformsOnly="true" 
    OutputChannelId="VideoStream" > 
    <DefaultClientInfo> 
      <MessageTypes> 
        <Message Type="IMAGE" />
      </MessageTypes>
      <ImageNames>
        <Image Name="Image" EmbeddedTransformToFrame="Reference" EncodingType="VP90"/>
      </ImageNames>
    </DefaultClientInfo>
  </PlusOpenIGTLinkServer>
adamrankin commented 5 years ago

Would encoding the image stream change FPS? It shouldn't in theory, it would just change the bandwidth used, no?

jamesobutler commented 5 years ago

From the description of the video:

Example: Without compression, HD RGB video frame rate was 12fps, using ~95MBS bandwidth; with compression we reached the maximum refresh rate (30fps) using less than 1MBPS bandwidth.

adamrankin commented 5 years ago

Interesting. Ok, good to know. @Sunderlandkyl is the video streaming wizard.

Sunderlandkyl commented 5 years ago

I think that it's referring to the FPS within Slicer, but I'm not sure.

I didn't realize that video had been uploaded to YouTube, but the encoding format has changed (see http://perk-software.cs.queensu.ca/plus/doc/nightly/user/ApplicationPlusServer.html).

Ex:

<Image Name="Image" EmbeddedTransformToFrame="Reference">
  <Encoding
    FourCC="VP90"
    Lossless="FALSE"
    Speed="8"
    DeadlineMOde="REALTIME"
    RateControl="Q" />
</Image>
jamesobutler commented 5 years ago

I used the new entries above but with "VP90", I kept getting an error, so I changed it to "I420" and it started successfully. I made sure in my build of Plus to check "PLUS_USE_VP9" in the configuration in addition to "PLUS_ENABLE_VIDEOSTREAMING".

|ERROR|002.861000| Could not create encoder for image stream WebcamStream of type VP90| in C:\P\PR17_x32-Dec\PlusLib\src\PlusOpenIGTLink\vtkPlusIgtlMessageFactory.cxx(543)

I guess what I was thinking it could do, is not actually going to help the problem I'm trying to solve. I have a couple streams going on into Slicer(4.10.0) which has started to max out my computer. But I guess that has more to do with the processing of rendering the images in the slice window. It of course improves when I'm not actively showing the stream in my Slicer window.

Sunderlandkyl commented 5 years ago

Are you building with VS2017?

Currently, the VP9 codec used in Plus is leveraging the one from OpenIGTLink, which uses prebuilt dll on Windows and only has support for VS2013 and VS2015 (https://github.com/openigtlink/CodecLibrariesFile/tree/master/VP9).

In the future, I'd like to set up Plus to use it's own build of VP9, independent from OpenIGTLink.

jamesobutler commented 5 years ago

Ah, yes. I usually always build Plus with VS2017 and haven't had any issues up to this point.

adamrankin commented 5 years ago

Discussion added here for source of VP9 VS2017

https://github.com/openigtlink/OpenIGTLink/issues/202

leochan2009 commented 5 years ago

Thanks for the report, yes, we could either use the build library from this repo, @adamrankin thanks for finding the repository with vs2017 build . https://github.com/ShiftMediaProject/libvpx/releases The library in ShiftMediaProject seems very well organized, I will test if the linking with openigtlink is fine. or @Sunderlandkyl , it is great if we could have the VP9 library maintained in Plus. As the current VP9 library wasn't updated frequently in (https://github.com/openigtlink/CodecLibrariesFile/tree/master/VP9).