Streaming collision between Capistrano and MMF Video

[jamesobutler]

I'm running a built version of Plus(32-bit) using Capistrano Video device(a wobbler probe) with its own SDK and streaming it into Slicer using openIGTLink port 18944. I'm also using a downloaded nightly package of Plus(64-bit) to run an MMF webcam device which streams into Slicer using openIGTLink port 18946.

I start the Capistrano Video device and then start the Webcam streaming device. MMF resolution is set to 1280x720 which my Microsoft LifeCam HD-3000 supports. If I leave out "AcquisitionRate" in the MMF webcam video device xml and let it try to get the best frame rate, I will get streaming errors for the Capistrano Video. This occurs as soon as I connect the client in Slicer's OpenIGTLinkIF module. The streaming errors that I get are erratic line mess ups across the wobbler image. See below:

I can prevent this streaming collision issue by setting my MMF resolution down to 640x360 and specifying AcquisitionRate="10".

Both the Capistrano imaging engine and the Microsoft LifeCam HD-3000 are both USB2.0 devices. They are plugged into unique usb ports in the back of my desktop tower.

Do I have to lookout for how much combined data I'm streaming using OpenIGTLink where connecting another device would hurt the stream quality of another device on another port?

[jamesobutler]

So I've been trying to debug this lately and have narrowed it down to a specific order of operations. I'm not sure if this is something with how PLUS is handling openigtlink, or some issue with OpenIGTLinkIF within Slicer.

Current Setup: CViewUSB (capistrano labs program for viewing ultrasound stream directly; seen in images below) Slicer 4.8.0 stable PlusApp 2.6.0 stable (used to stream the webcam into Slicer through openigtlink) Capistrano Ultrasound USB Probe connected directly to computer by USB2.0 Microsoft LifeCam HD-3000 connected directly to computer by USB2.0 (1280x720; YUY2; AcquisitionRate="10")

No Problems:

I don't get any ultrasound streaming issues within CView if I FIRST, create a Slicer IGTLconnector node for the webcam stream and make it active so it has a WAIT status and THEN start PlusServer. Something about the node going from WAIT->ON prevents ultrasound streaming problems, but if the node goes immediately from OFF->ON when setting it to active, then there are two different image problems.

Problems:

1. I get image shift problems in the ultrasound stream within Cview. This happens if I'm already streaming ultrasound in Cview and then create a Slicer IGTLConnector node for the webcam and set it to active in which it goes OFF->ON immediately because the PlusServer is already running. It doesn't matter when the PlusServer was started relative to streaming ultrasound in Cview, but it does matter when Slicer IGTLConnector node is set to active relative to starting the PlusServer.
2. I get erratically changing dark bands in the ultrasound stream if I FIRST start PlusServer, then connect the Slicer IGTLConnector node for the webcam and set it to active in which it goes from OFF->ON immediately. Then SECOND, I start streaming ultrasound in Cview.

[lassoan]

What is unclear for me is how OpenIGTLink communication is related to scanning or scan conversion errors in CView.

• It may be just a CPU load issue - if CView, Plus, Slicer, etc. are all running on the same computer then maybe CView sometimes does not get enough CPU time and that causes errors in the digitization. Setting CView process priority to higher may help.
• There may be also a bug in CView that causes memory corruption when OpenIGTLink communication is set up a certain way.
• If network transfer in CView does not use a background thread but it blocks the main thread then any slowdown or congestion of network transfer may interfere with real-time data acquisition and processing.

Does CView use OpenIGTLink directly or it uses Plus? Do you know if CView is configured to use OpenIGTLink v2 or v3? Are all processes running on the same computer?

[jamesobutler]

Does CView use OpenIGTLink directly or it uses Plus?

CView is a C# GUI based program made by Capistrano Labs. It doesn't use OpenIGTLink. It calls the two Dlls directly (usbProbe.dll and BmodeUSB.dll).

Are all processes running on the same computer?

All the process are being run on the same computer.

It may be just a CPU load issue -...

I don't believe it to be a CPU load issue. With all those programs running and streaming, I'm at most 30% load.

More Information

Current Setup PlusApp 2.6.0 stable (64 bit) for mmf webcam streaming into Slicer (Port 18946) Plus-2.7.0 recent build (32 bit) for Capistrano ultrasound streaming into Slicer (Port 18944) Slicer 4.8.0 Stable

I don't think it is CView, the program, that is the issue because I've tried streaming ultrasound with Capistrano using a built version of Plus and still have streaming issues due to certain order of operations. Capistrano Video source in Plus also calls the Dlls similar to CView.

With the above setup I was able to recreate streaming issues if I started Plus64-bit for the webcam and Plus32-bit for Capistrano first and then create the Slicer IGTLink clients second where their status goes from OFF->ON immediately upon setting status to active since the servers are already started.

Is there something different when the client side goes OFF->ON versus WAIT->ON (Asynchronous/synchronous)?

If I do, the following below there are no streaming issues.

1. Slicer client Plus32bit OFF->WAIT
2. Slicer client Plus64bit OFF-->WAIT
3. Start Plus64bit (Slicer client goes WAIT->ON)
4. Start Plus 32bit (Slicer client goes WAIT->ON)

Also no issues with the below (a few times it led to stream shift problems, but works most of the time).

1. Slicer client Plus64bit OFF-->WAIT
2. Slicer client Plus32bit OFF->WAIT
3. Start Plus 32bit (Slicer client goes WAIT->ON)
4. Start Plus64bit (Slicer client goes WAIT->ON)

[lassoan]

Does CView communicates with any other software through OpenIGTLink? If not, then how it is possible that the image in CView gets corrupted, depending on how two other applications (PlusServer and Slicer) are communicating with each other?

There should be absolutely no difference between the client connecting to an already running server (OFF->ON) versus client is waiting for a server to start and then connect (WAIT->ON). When connector in Slicer is in WAIT state then it tries to connect once in every second exactly the same way as it connects when you enable connection.

Could you check if you don't send the images through OpenIGTLink but save to file directly in PlusServer then you can still see artifacts? (just to know if the error is in the acquisition or OpenIGTLink streaming part)

[jamesobutler]

Thanks for the responses @lassoan. Yeah not sure why a Slicer and Plus communication would affect streaming that is going on through CView which is not using openigtlink.

1. I start Plus64-bit for the webcam where the xml specifies an openigtlink server but doesn't include any data to go across it.
2. I then start Plus32-bit for capistrano streaming where the xml specifies an openigtlink server but doesn't include any data to go across it.
3. I then start frame recording for the capistrano and move the probe around to get unique frames.
4. I stop frame recording and view the MHD within Slicer. There is no image problems observed in any of the frames.

I repeated the same thing above, but now the server is trying to send the image frames, though I'm still not setting up Slicer to receive them at all. This also results in no image problems in any of the recorded frames.

Now, this is when it messes up:

1. Start Plus64-bit for the webcam
2. Start Plus32-bit for the capistrano stream.
3. Then create the Slicer Plus32bit client which goes from OFF->ON (I'm not viewing the stream in the slice view, but it is connected).
4. Then create the Slicer Plus64bit client which goes from OFF->ON (Also not viewing the stream in the slice view, but it is connected).
5. Start frame recording for the capistrano ultrasound.
6. Move the probe around to get unique frames and then stop the recording.
7. Open the MHD in Slicer and I observe the frame shift image problems.

If I do the same thing as above, but create the Slicer Clients first so they are in a WAIT status and then start the servers, then doing frame recording results in no image shift problems.

[jamesobutler]

The image shift problems have been solved by fixing the 3rd party imaging engine drivers installer. Driver files with incompatibilities were causing havoc. Updating the drivers installer cleared up this issue.

Also from the SDK developer regarding the image shift

The fundamental issue with the image shift is that the probe is trying to push data on a bus that's not requesting it (USB is host based). We don't have much buffering so that's a problem. We lose vectors in these cases. This is exacerbated by high memory (more so than high CPU) loads on the system. So if a lot of different threads/processes are all banging on the USB and memory, that will cause this. This is one area that is slightly improved in the 2016 DLLs [...]

I'm closing this issue since I no longer consider this to be a PLUS issue.