Audio desynchronized with video.

[tfabris]

Currently the audio stream is desynchronized from the video stream. The audio is several seconds earlier than the video.

To do:

• Confirm the origins of the problem. Define if the issue is coming in from the camera itself, Synology, or the YouTube portion of the stream. Each is testable with the equivalent of a clapboard test. View on the Reolink app to see the Camera, and view in Surveillance Station timeline for the Synology. If both of those are good then it's YouTube streaming fault.
• See if reducing the stream back down to 1080p (instead of 1440p as it is currently) will fix the issue. It is possible the problem started when we went to 1440p.
• See if increasing the bytes bandwidth on the camera fixes the issue.

[tfabris]

Status:

• Added a feature to CrowCam.sh which bounces the stream at the top of each hour.
• This glitches the video stream for a couple of seconds per every hour, but hopefully this will result in a synched stream, since it usually takes a little while for the stream to become desynchronized after a bounce.

To do:

• Check synchronization of all video/audio created after 2020-05-26 13:00 (1pm) now that the hourly bounce feature is implemented.
• Check logs to ensure bounce is occurring only at the hour tops, and always at the hour tops. I am temporarily switching the bounce log message for this bounce to "info" for a little while. If it works, put the message back to "dbg" after this behavior is confirmed.

[tfabris]

I have confirmed that doing the top-of-the-hour bounce fixes the issue temporarily. The problem is, the issue recurs at random times, so, sometimes the problem recurs mere minutes after a bounce had fixed it.

If the Synology techs had experimentally implemented this same kind of bounce at a lower level in the code, then, the reasons I'm still seeing the issue recur, must be because of that randomness.

Note that the issue isn't the audio stream... it's the video stream that's the problem. In every example of the issue I've seen, it is clear that the audio stream continues uninterrupted, with no glitches in the sound. The video stream is always the part that glitches. After the video stream experiences a glitch, then the video stream is now running a few seconds behind the audio. Is it possible that the Synology techs had been investigating the wrong path, trying to find problems in the audio encoding, when the real problem is actually with the video? Or more specifically, that the Live Broadcast feature is unable to detect when the video glitches, and thus doesn't know that it needs to bounce the audio stream too?

Example:

Below is a specific set of examples of all those things from yesterday. In particular, this is an example of the desynchronization recurring mere minutes after it was fixed by a bounce.

Audio is synchronized:

• 2020-05-26 04:50:28 PM - Squirfuffle on the roof of the feeder, sound is correctly synchronized: https://youtu.be/qvvbtdBJXvg?t=14490

The moment where a video glitch causes it to become desynchronized:

• 2020-05-26 04:56:41 PM - Video freezes and glitches briefly. You can see it occur clearly, by watching the clock baked into the video stream. This is the moment where the sound becomes desynchronized: The audio continues smoothly without glitching, but after this video glitch, the video is a few seconds behind the audio: https://youtu.be/qvvbtdBJXvg?t=14864

Examples of desynchronization immediately after the glitch:

• Note: With the squirrels, you have to have some experience with what to look and listen for. You can hear the sounds of the squirrel reaching into the feeder for another kibble, versus the sound of the squirrel chewing on a kibble. In reality, the squirrel does not chew while it is reaching into the feeder, but the desynchronized sound and video make it seem like it is still chewing while the video shows it reaching in. Specific examples of it are at these points:
• 2020-05-26 04:56:58 PM - https://youtu.be/qvvbtdBJXvg?t=14882
• 2020-05-26 04:58:43 PM - https://youtu.be/qvvbtdBJXvg?t=14988
• 2020-05-26 05:04:34 PM - https://youtu.be/qvvbtdBJXvg?t=15335

Stream is bounced by the script:

• 2020-05-26 05:04:38 PM - Stream is briefly stopped and restarted with an API command, the restart is visible on the video stream as a skip in the clock baked into the video stream: https://youtu.be/qvvbtdBJXvg?t=15341

Stream is synchronized after the bounce:

• 2020-05-26 05:10:16 PM - Steller's Jay arrives, the sound is correctly synchronized: https://youtu.be/qvvbtdBJXvg?t=15677

The moment where a video glitch causes it to become desynchronized:

• 2020-05-26 05:14:38 PM - Video freezes and glitches briefly. This the moment where the sound becomes desynchronized: The audio continues smoothly without glitching, but after this video glitch, the video is a few seconds behind the audio: https://youtu.be/qvvbtdBJXvg?t=15937

Example of desynchronization after the glitch:

• 2020-05-26 05:14:43 PM - Crow lands on feeder, and the sounds of its claws hitting the feeder, and the sound of the platform banging as it flies away, occur a couple of seconds earlier than the corresponding video: https://youtu.be/qvvbtdBJXvg?t=15943

[tfabris]

2020-07-08: Synology support has still been working on this, and they are going to do another round of experiments on my NAS. Details in the private support ticket.

[tfabris]

Synology says they have applied a patch to my unit and that I should observe the stream results and look for desynchronization errors in the results.

2020-07-09 04:04 AM PDT - Synology tech login registered on unit, patch likely applied at that time. 2020-07-09 10:45 AM PDT - Deactivated the hourly stream bounce feature in the CrowCam code, so that if the issue recurs, it will be detectable.

To do:

• Observe the stream audio+video, look for synch problems, and report results back to Synology ticket.
• Check for possible desynch: Check archived video after post-processing by YouTube is complete: 2020-07-09 07:03:49 am. (DONE: After post-processing, synch was perfect.)

[tfabris]

Reported to Synology:

I kept an eye on the stream for several days, and I didn't see any recurrences of desynchronization until Sunday the 12th (yesterday). So there is a chance that the recurrence of the issue has significantly decreased, though it is not gone.

(Since I'm merely observing the critters on the live cam, and the critters do not always make noises that correspond with clear obvious movements, so it is possible that there were other recurrences which I did not notice.)

Links to examples of desynchronization that I noticed:

2020-07-12 11:05:23 am - https://bit.ly/38R3KwK - Squirrel jumps onto feeder from above the camera. Video is slightly desynchronized from audio, you can hear the squirrel's footfalls on the roof of the feeder very briefly before you see the squirrel.

2020-07-12 11:43:28 am - https://bit.ly/32d21k3 - Crow - Video is slightly desynchronized from audio, you can hear the crow's feet hit the platform very briefly before you see its feet hit.

2020-07-12 11:44:05 am - https://bit.ly/2WhncxJ - Juvenile atop feeder begs from parent - Video is very clearly desynchronized from the audio. The juvenile is the one with the pink mouth who flaps its wings and opens its mouth wide when it begs. The juvenile voice has more of a "honk" sound than the parent, it sounds a little bit like a kazoo. You can clearly hear the juvenile cawing about one second before the video shows it, and it repeats several times, so it's very clear.

(By the way, I have checked the version of the video which is archived on the Synology NAS itself, by opening up Surveillance Station app and viewing the "Timeline" of that last instance that I linked above. The Video on the NAS is fine, the audio+video are in perfect synchronization, so the issue is reproducing only in the Live Broadcast to YouTube.)

[tfabris]

2020-07-29:

• The tech's patch which was applied to my unit on 2020-07-09 seems (not certain) to have reduced the frequency of the issue. I have disabled the "bounce every hour" feature in my CrowCam Controller script, and still I rarely see the problem recur.
• The patch fixes a problem where, if there is not enough audio data coming into the stream, then it becomes desynchronized. The patch detects the issue and forces a resynchronization. The techs are insistent that it's an audio data stream problem, even though I have seen it occur when the video glitches.
• When the patch performs the fix, the benefit of the fix is seen on the next keyframe boundary. My unit is set to the fastest possible keyframe interval of "30" which is about 1 second.
• Still, after the patch, I see the issue recur, though less frequently now. Perhaps there are multiple root causes of the issue and only one of them is fixed by the patch? I have suggested to the Synology techs that maybe video dropouts, in addition to audio dropouts, could cause the desynchronization issue.

Examples of recurrence after the 2020-07-09 patch was applied:

2020-07-22 3:30:15 pm - Audio is ahead of video by a few seconds, sounds of the crow's footfalls are heard a few seconds earlier than the actions are seen. Continues for at least 2 minutes or more: https://bit.ly/2CSYoWu 2020-07-22 5:05:07 pm - Same stream, audio is back to normal synchronization: https://bit.ly/3jQvdE3

2020-07-12 11:44 am - Audio is ahead of video: https://bit.ly/2WhncxJ

In the meantime:

• There was a Surveillance Station version update to version 8.2.8-6334. I applied this version update on 2020-07-29. This has overwritten the patch, and it does not contain the patch fix. I have requested that the techs please re-apply the patch and let me know if it can be included in the next public version update.
• The techs re-applied the patch on 2020-07-31 at 3:50 AM PDT.

[tfabris]

To do:

• Keep an eye out if Surveillance Station gets an update higher than 8.2.8-6334, and if so, have Synology go in and re-apply the manual patch.

[tfabris]

Syno techs patched the system after the most recent "normal" update. The patch occurred 2020-07-31 at approximately 3:45-3:50 am PDT.

To do:

• Check video at 2020-08-02 4:12:50 pm PDT - Jay visits feeder, but sound is noticed to be desynchronized on the live feed, recheck this after postprocessing to see if it's still desynchronized after a few days of the video sitting on YouTube's servers. Update: Here is a link: https://youtu.be/umEGAHwcUcI?t=75 - Need to report this.

[tfabris]

2020-08-09 - Examined the video stream for the last several days since the techs applied their manual patch on 2020-07-31. The only timing glitch I've seen is the previously-mentioned one at 2020-08-02 4:12:50 pm PDT.

Reporting to Synology Support that the problem is solved for me EXCEPT that it needs to be put into a public release to be considered finished.

[tfabris]

To do:

• Keep an eye out if Surveillance Station gets an update higher than 8.2.8-6334, and if so, have Synology go in and re-apply the manual patch.
• How to find this information:
• Log into Synology Surveillance Station.
• Press the "System" icon.
• Look for the line "Surveillance Station Version".
• If this version is higher than 8.2.8-6334, look at its version history at https://www.synology.com/en-us/releaseNote/SurveillanceStation and see if the fix is mentioned there.

[tfabris]

The corresponding bug is expected to be fixed in the next full release of Surveillance Station. The upcoming 8.2.8 hotfix will not include the fix.

In the meantime, to keep my existing patch:

• Back up the existing file - just the single executable "ssrtmpclientd" (Surveillance Station RTMP Client Daemon) - at an SSH prompt. Back it up to your home directory ("~") and use the "-p" option to preserve its file properties and permissions. (This has been done.)

  sudo cp -p /var/packages/SurveillanceStation/target/sbin/ssrtmpclientd ~

• After the hot fix of 8.2.8 gets installed, restore the file from your home directory with these command at the SSH prompt:

sudo synoservicectl --status pkgctl-SurveillanceStation
sudo synoservicectl --stop pkgctl-SurveillanceStation
sudo synoservicectl --status pkgctl-SurveillanceStation

sudo cp -p ~/ssrtmpclientd /var/packages/SurveillanceStation/target/sbin/

sudo synoservicectl --start pkgctl-SurveillanceStation
sudo synoservicectl --status pkgctl-SurveillanceStation

(The above has been done - 2020-11-15 - Surveillance station version was 8.2.8-6335 at the time.)

[tfabris]

Audio desynchronization was noticed on 2020-08-16 - No changes or updates had been made to the SS software, and, I checked the logs and there were no detected problems with my home internet during this period of time.

3:21 pm - Crow: https://youtu.be/PuwxChpIXak?t=2368 4:14 pm - Crow, mouth open: https://youtu.be/PuwxChpIXak?t=5542 4:17 pm - Crow family visit, 3 crows. Very hot day, crows have their mouths open: https://youtu.be/PuwxChpIXak?t=5703 4:18:51 pm - Crows - Audio is correctly in synch: https://youtu.be/PuwxChpIXak?t=5802 4:19:18 pm - Video glitch, audio becomes desynchronized at the point of the video glitch: https://youtu.be/PuwxChpIXak?t=5826 4:19:36 pm - Crow departs, audio desynchronization can be noticed at the point of departure: https://youtu.be/PuwxChpIXak?t=5841 4:20:20 pm - Video glitch/jump (no one around to check audio timing): https://youtu.be/PuwxChpIXak?t=5886 4:44:26 pm - Squirrel, audio and video are back in sync: https://youtu.be/PuwxChpIXak?t=7334

[tfabris]

Audio desynchronization noticed: 2020-09-03 Thu - 11:15:48 am - Steller's Jay, sound is desynchronized: https://youtu.be/6yIcWebh0hY?t=964

I noticed this on 2020-09-04. Then I checked the version of the software and it was the 8.2.8 version which was expected to mess this up. Applying the manual patch now, on 2020-09-04 8:45 am.

[tfabris]

I'm noticing the audio desynchronization even without the RTMP daemon in the mix. Even when not broadcasting to YouTube, just viewing Surveillance Station directly through its web UI, the desynch is there, and that's with the latest version of the software.

[tfabris]

Tried a firmware upgrade of the Dahua camera. Its UI tool said "the firmware is up to date" which was a lie, there was a newer version downloadable from their web site, dated August 2022. Installed that firmware and it doesn't fix the problem, there's still a desynch between audio and video.