google-research / libsoftwaresync

:camera: :camera: :camera: :camera: :camera: Wireless software synchronization of multiple distributed smartphone cameras.
https://arxiv.org/abs/1812.09366
Apache License 2.0
145 stars 34 forks source link

Sync Audio #10

Open sunwell1994 opened 2 years ago

sunwell1994 commented 2 years ago

Thanks for the interesting audio. The phase alignment method is really helpful and the results are impressive. One more question is whether we can record the audio streaming with perfect synchronization. Based on the reference latency, it seems that we may get a delay of less than 20 microseconds for the image and audio stream after applying alignment to both audio and image. Is it true?

Elucidation commented 2 years ago

You'll need to quantify the latency of initiating audio streaming/recording. I think when we were looking into using audio chrips for synchronization the variable latency of starting/stopping audio streaming was much greater than 20 microseconds.

Assuming you can resolve that, you could indeed use the network sync portion of software sync to trigger the audio streaming, and that part of it should be < 20 us.

The phase alignment sync portion is for the cameras currently, so it may not be relevant to audio streaming unless you're also recording video.

sunwell1994 commented 2 years ago

You'll need to quantify the latency of initiating audio streaming/recording. I think when we were looking into using audio chrips for synchronization the variable latency of starting/stopping audio streaming was much greater than 20 microseconds.

Assuming you can resolve that, you could indeed use the network sync portion of software sync to trigger the audio streaming, and that part of it should be < 20 us.

The phase alignment sync portion is for the cameras currently, so it may not be relevant to audio streaming unless you're also recording video.

Thanks for sharing your experience. It is really helpful to remind me that the starting/stopping audio streaming is really large. I would like to record the video but I find that the video is not perfectly synced with audio.

Is the reset sampling method possible to fix the start of the audio stream into the target period? Since its variance is large, it may require many more iterations.

Elucidation commented 2 years ago

Is the reset sampling method possible to fix the start of the audio stream into the target period? Since its variance is large, it may require many more iterations.

Rough guess is yes but this would be complicated: 1 - You'll need to look into how audio streaming starts/stops relative to the video, and whether it takes a long time to restart it, and if the variability of the error is random enough that reset sampling is valid. Might be too long to be useful. 2 - You need some way to measure the offset error of the audio from the video, this might be easy if the audio stream has a separate start time stamp, but I haven't looked into this.

sunwell1994 commented 2 years ago

Thanks so much. Will try to figure out these challenges.