Added Temporal Synchronisation

[ChristopherRemde]

The Kinect for Azure provides temporal synchronisation between devices via its 3.5mm sync ports. When using multiple Kinects, this should improve the quality of the depth scan, aswell as syncing the device captures temporally.

Our team decided to add this functionality to LiveScan3D, which includes these features:

• Automatic detection of the role of the device (Master, Subordinate or Standalone)
• Master is being started after all subordinates have started, as recommended by the device manufacturer
• Subordinates have an offset of 160us between each device (Recommended minimum)
• The role of each device is displayed in the server list with a [MASTER] or [SUBORDINATE] tag, if temp sync is enabled
• Timestamp in the recording.bin file is now the actual Kinect device Timestamp in microseconds, not the PC timestamp
• Fast motions during recording now show less artifacts

Usage:

1: Connect all devices via sync cables, one camera should be the master, the others should be daisy-chained subordinates (Star-Formation is not supported) 2: Connect all LiveScanClients to the server 3: On the server, go to Settings and check the "Enable Temporal Sync" box 4: The clients should now restart the Kinect camera. During this process the client can temporarily freeze 5: When the restart is finished, you should now see the role tag of each client in the server list

Notice: When there is no role tag shown in the server list, the device is running as a standalone

Known Issues:

• When the Master client has not been connected last to the server, the delay between the client connected before the master and the one after the master is 320us. To avoid this issue, please connect the Master client last.
• The "Enable Temporal Sync" checkbox in the settings should only be activated after all clients have been connected to the server
• When the "Live View" windows is opened while toggeling the "Enable Temporal Sync" checkbox, LiveScan crashes

Notes: I tested a lot if the sync is working, and I'm confident that depth precision, when using multiple kinects, is enhanced. Temporal alignment, especially during fast motions, should also be more stable . However, dropped frames can still occur and the timestamp of the frames are not always aligned perfectly. I'm not totally sure what causes this, but there seems to be a lot of issues open regarding the sync feature stablity on the Azure-Kinect-SDK Github page. I'll monitor the discussions, and if there is a fix available, provide it as pull request.

I also tested this pull request in conjunction with #48 and didn't encounter any issues.

[MarekKowalski]

Hi Christopher,

Thanks a lot for posting this PR, it looks great. Also, sorry for taking so long to take a look at it. Within the next few weeks I hope to have some more time to take proper care of this repo, including your PR. Is there any chance you'd have some time to take a look at the bug that causes the app to crash if synchronisation is enabled with the window open?

Thanks,

Marek

[ChristopherRemde]

Hello Marek,

Don't worry, I'm very thankful that you are still maintaining this repo!

I'll try to find the issue which crashes the app, I also found another bug yesterday, where the server sends the message to start the master twice. Unfortunately I also won't have access to my Kinects until the start of september, but I'll try to push the fixes as soon as I can!

[ChristopherRemde]

Hi Marek,

I could successfully fix all known issues from this commit and also added a clearer UI which prohibits user errors. However, I'd still like to add a function which allows the user to set the Exposure manually, as this is a requirement for a good temporal sync, so I'm going to commit the next version in a week or a bit later, when I have the feature ready.

[ChristopherRemde]

Hi Marek,

So this took a good time longer than I expected, sorry for that!

I finally commited the manual Exposure feature, but also made the application a lot more robust. The UI for this feature is now a lot more robust against user errors, It only allows the user to enable/disable the temp sync feature when the devices are ready. Also when an error happens (Sync cables not configured probably, device doesn't open on restart), clear errors are printed for the user and the program tries to handle them to a certain extend (Deactives sync feature when cables are not correctly connected, tries to restart the device multiple times).

Here you can see the new UI. I also took the freedom to disable the Body Data settings, as these are currently not used in the K4A version and can lead to the program crashing.

Other changes to make this feature & the software more robust:

• Settings window can only be opened once
• When the client can't open a device, it tries multiple times
• Recording bins are now saved with a unique ID per device, to allow multiple clients to record on the same PC
• Synchronized & Unsynchronized devices can be both used in one server
• Included a workaround for a bug in the Azure Kinect SDK regarding exposure (https://github.com/microsoft/Azure-Kinect-Sensor-SDK/issues/1378)

I tested this commit throughly and there is no error I'm aware of. But there could always be something slipping through my net, so if an error related to this PR will be detected in the future, I'll of course try to fix it.

[MarekKowalski]

Hi Christopher, This looks great, thanks so much for making this PR and apologies for not replying to it sooner. It'd be great if you could add a sentence or two about how to enable the synchronisation to the README or alternatively add a separate piece of text about it in the docs directory. Otherwise, I am happy to accept this PR, though I can't try it myself as I do not have access to Kinects :(

Marek

[ChristopherRemde]

Hi Marek,

I added two commits, one with an updated Readme Text, and one with the information in a seperate document in the Docs folder. I figured that the text in the readme is a bit too long, so I made a seperat manual. But you can decide which one to keep!

If you want the Source-Docx for the new manual, I can send it to you!

Regarding the testing: You can also leave this PR unmerged, if you want to test it first, totally understandable!

[MarekKowalski]

Hi Christopher,

Thanks a lot for your amazing work. I am still not able to test this, but I see that people are using your branch and it works great, so I'll just merge this PR in :)

Once again, thanks so much!

Marek