How can the Clients are synchronized?

ChienDuong commented 6 years ago

Dear Marek, As far as I understand, The Clients will acquire Frame simultaneously., then send to Sever. The Sever wait until all Client captured, then receive the information. Now, I run the system with two Kinect, one in Release mode (up to 30fps), one in Debug (5fps) Thus, the Sever, will receive the latest frame of faster Client (Release), and the only one frame from slower Kinect (Debug).
How can I make sure that two Kinect frames are quite similar? The observed results between two case: 2 Release Kinect, and 1 Release and 1 Debug are not much different. Both of case, the system are quite synchronization. The only assumption, I guess is the Debug camera captured the image 5fps, but process it very fast. Thus,the latest frame captured from Release Camera is almost the same as the the frame captured by Debug camera. Am I right?

Thank you, Mark

MarekKowalski commented 6 years ago

Hi,

There is currently not synchronization mechanism employed in the app. Whenever a client receives a request for a frame it sends the last frame that was obtained from the sensor. Because of that even if one client has a lower framerate (for example due to the fact that it runs in debug) the sever should still receive at a high framerate (unless it is limited by bandwidth).

Does that answer your question?

Marek

ChienDuong commented 6 years ago

Hi Marek, Thank you for your answer. I would like to point out some of my thoughts below.

Sever sends the request to the Client, then it will receive the information whenever the Flag "allGathered=true" ( Sever finished to receive all Client information). The reason that the sever doesnt reduce frame rate to the slowest one is:

Client: Capture the image and Send the information are independent. Client always acquired new image. Whenever it received the request from the Sever, it just sendd the latest Frame. Thus, the faster Client may send the newer frame while the slower frame still sendd the same frame (the slower has not captured the new frame at the time it received request). The disadvantage is: the two frames from two Kinects showing in the sever are not captured at the same moment.

Second, If I want to do the simple synchronization (I am tracking the skeleton of human, so I want to make sure that two skeleton come from the same frame as much as possible), the most simple way I can do is: somehow, the faster Client captures the image only when the slower Client captures the image. Dont let them do it independent, then the results in Sever is at slowest frame rate.

I attached the current results, I combine another information into one Client, it make this Client is much slower than another Client. And without synchronization, two skeleton is differents ezgif com-video-to-gif Do I understand correctly?

Thank you, Mark

ChienDuong commented 6 years ago

Hi Marek, I think my thoughts above is correct. I already can synchronize as shown in the result below . 3synchronizeresults However, Now I have another questions. Kinect SDK2.0 provided the The MultiSouceFrameReader, which can read many stream data at the same time such as: Color, Depth, Body, Inferred. As far as I know, the multisouceFrame reader somehow fuse all the stream together and we can get the slowest fps (e.g. if: Color 15fps and depth, inferred: 30fps -> Multisouces =15fps). But it is not correct 100%, sometimes, when Multisoucesreader is successful, all the specific types inside are still not successful. Thus, If we want to take out the specific stream (GetBody, GetColor,GetDepth), we still need to double check it right?.

Thank you, Mark

MarekKowalski commented 6 years ago

Hi ChienDuong,

Your resoning in the first post is correct, this is exactly how the system works. As for ways to synchornize the two Kinects: the method you proposed should work ok for two devices. I guess a more robust method would be to synchronize the clocks of the clients and use timestamps for each frame. This way you might even be able to interpolate between frames to achieve better alignment.

As for the MultSourceFrameReader, it's been a while since I looked at the Kinect SDK and I am not sure how this particular method works, sorry :(

Let me know if you have any other questions I can help with!

Thanks,

Marek

ChienDuong commented 6 years ago

Hi Marek,

I really appreciate your suggestions. I will spend time to improve my result.

Thank you so much,

Mark

percramer commented 4 years ago

Hi ChienDuong,

the skeleton synchronization is that something you added to your source? Or is it in the general source also? And does it handle 2 Kinects or multiple?

Regards,

Per

ChienDuong commented 4 years ago

Hi Per,

Long-time ago, I may forget something obvious. As far as I remember, The skeleton is already implemented in the code, Kinect SDK also supports skeleton detection. You just need to show it out. In order to combine the view from multiple views (multiple Clients) into one, Marek already implemented it. For synchronization the timestamp, the original code did not implement it, the result I showed above is also the results without synchronization. The results have slightly different between cameras, the client (camera) sends their latest frame when received the request from the server.

Hope it can help, Mark

MarekKowalski / LiveScan3D

How can the Clients are synchronized? #26