billw2 / pikrellcam

Raspberry Pi motion vector detection program with OSD web interface.
GNU General Public License v3.0
263 stars 70 forks source link

drift audio from video out of sync #47

Open AntonJouri opened 5 years ago

AntonJouri commented 5 years ago

When recording video with audio, the mic is ahead of the video, and it is getting worse every minute. It is known, that the fps of Raspi-Camera and USB-Microphones usually don’t match. So I wonder, how do you compensate the drift?

I tried to compensate by setting the mp4box-fps just by 1 higher than fps, e.g. fps=23, mp4box=24, what tuned out as video is ahead of the mic-audio. I tried comma-values in the config-file directly, like fps=23.4 (period) or 23,3 (comma), and mp4box=23,8 or 23.9, and many other pairs of values, but as long as only integer values are processed, I can not find a matching value to compensate the drift.

Or, is there any parameter to adjust for the process that converts h246 and audio to mp4 to minimize the drift?

I´m using Raspberry Pi3+ with BusterLite as basis for PiKrellCam. May be I should downgrade?

Right now I´m out of ideas. Any hint appreciated.

Greetings, Gunneg

billw2 commented 5 years ago

On Wed, 31 Jul 2019 14:17:54 -0700 AntonJouri notifications@github.com wrote:

When recording video with audio, the mic is ahead of the video, and it is getting worse every minute. It is known, that the fps of Raspi-Camera and USB-Microphones usually don’t match. So I wonder, how do you compensate the drift?

I tried to compensate by setting the mp4box-fps just by 1 higher than fps, e.g. fps=23, mp4box=24, what tuned out as video is ahead of the mic-audio. I tried comma-values in the config-file directly, like fps=23.4 (period) or 23,3 (comma), and mp4box=23,8 or 23.9, and many other pairs of values, but as long as only integer values are processed, I can not find a matching value to compensate the drift.

Or, is there any parameter to adjust for the process that converts h246 and audio to mp4 to minimize the drift?

I´m using Raspberry Pi3+ with BusterLite as basis for PiKrellCam. May be I should downgrade?

Right now I´m out of ideas. Any hint appreciated.

Greetings, Gunneg

The config takes only integer values so there is no way to do any compensation now. I could change the type to float so factional values could be entered, but your asking this has made me think that a better way would be for pikrellcam to calculate a fps needed to make video time match audio time and use that in the mp4box. I'm wondering about the magnitude of the drift you get, so could you look in the pikrellcam log and tell me some rate results you get for 2 or 3 typical video records? Sound like you have long videos, but for a short video I get this for a "Motion record stop" line in the log:

Motion record stop (size:5.0MB < /tmp:257.5MB) vid_time:10.04 vid_fps:24.00 audio_frames:481965 audio_rate: 48000

If I could see some log numbers like that for some of your videos it would help and in the meantime I'll think about making the changes to pikrellcam of fractional fps config and automatic calculating a mp4box fps.

Thanks, Bill

-- Bill Wilson billw@gkrellm.net

AntonJouri commented 5 years ago

Thank you very much for your quick response, Bill.

I made a clean install and 4 different recordings of 2 minutes each.

1) 24 fps and 0 mp4box_fps From the beginning, audio is a little faster than video, after 2 minutes audio is 2 seconds faster than the video.

2) 24 fps and 25 mp4box_fps For the first 30 seconds it seems to be in sync, but then audio is slower than video, after 2 minutes 2 seconds behind.

3) 23 fps and 24 mp4box_fps (almost the same result) For the first 30 seconds it seems to be in sync, but then audio is slower than video, after 2 minutes 2 seconds behind.

4) 23 fps and 22 mp4box_fps After 2 minutes video is about 4 seconds faster than audio.

======================================================== ======= PiKrellCam 4.3.0 started at 2019-08-02 17:18:47

17:18:47 : Failed to open motion regions file: /home/pi/.pikrellcam/motion-regions.conf 17:18:47 : execl[wait:1]: sudo chown pi.www-data /home/pi/pikrellcam/www command FIFO: /home/pi/pikrellcam/www/FIFO motion_detects FIFO : /home/pi/pikrellcam/www/motion_detects_FIFO audio FIFO : /home/pi/pikrellcam/www/audio_FIFO mjpeg stream: /run/pikrellcam/mjpeg.jpg 17:18:47 : execl[wait:1]: sudo mkdir -p /home/pi/pikrellcam/media 17:18:47 : execl[wait:1]: sudo chown pi.www-data /home/pi/pikrellcam/media 17:18:47 : execl[wait:1]: sudo chmod 775 /home/pi/pikrellcam/media 17:18:47 : execl[wait:1]: sudo mkdir -p /home/pi/pikrellcam/media/archive 17:18:47 : execl[wait:1]: sudo chown pi.www-data /home/pi/pikrellcam/media/archive 17:18:47 : execl[wait:1]: sudo chmod 775 /home/pi/pikrellcam/media/archive 17:18:47 : execl[wait:1]: sudo mkdir -p /home/pi/pikrellcam/media/loop 17:18:47 : execl[wait:1]: sudo chown pi.www-data /home/pi/pikrellcam/media/loop 17:18:47 : execl[wait:1]: sudo chmod 775 /home/pi/pikrellcam/media/loop 17:18:47 : execl[wait:1]: /home/pi/pikrellcam/scripts-dist/_init /home/pi/pikrellcam /home/pi/pikrellcam/media/archive /home/pi/pikrellcam/media /run/pikrellcam/mjpeg.jpg /home/pi/pikrellcam/www/FIFO /tmp/pikrellcam.log servos_off /home/pi/pikrellcam/media/loop making www/state link to /run/pikrellcam/state making www/mjpeg.jpg link to /run/pikrellcam/mjpeg.jpg www/media link is already set to /home/pi/pikrellcam/media www/archive link is already set to /home/pi/pikrellcam/media/archive making www/loop link to /home/pi/pikrellcam/media/loop /home/pi/pikrellcam/www/config.php: LOG_FILE not changed. /home/pi/pikrellcam/www/config.php: MJPEG_FILE not changed. /home/pi/pikrellcam/www/config.php: FIFO_FILE not changed. /home/pi/pikrellcam/www/config.php: PIKRELLCAM not changed. /home/pi/pikrellcam/www/config.php: SERVOS_ENABLE not changed. /home/pi/pikrellcam/www/config.php: VERSION updated to: 4.3.0 17:18:47 : execl[wait:1]: /home/pi/pikrellcam/scripts/startup /home/pi/pikrellcam /home/pi/pikrellcam/media /home/pi/pikrellcam/media/archive /tmp/pikrellcam.log 17:18:47 : execl[wait:1]: sudo chown pi.www-data /tmp/pikrellcam.log 17:18:47 : execl[wait:1]: sudo chmod 664 /tmp/pikrellcam.log 17:18:47 : execl[wait:1]: sudo mkdir -p /run/pikrellcam 17:18:47 : execl[wait:1]: sudo chown pi.www-data /run/pikrellcam 17:18:47 : execl[wait:1]: sudo chmod 775 /run/pikrellcam 17:18:47 : execl[wait:1]: sudo mkdir -p /home/pi/pikrellcam/media/videos 17:18:47 : execl[wait:1]: sudo chown pi.www-data /home/pi/pikrellcam/media/videos 17:18:47 : execl[wait:1]: sudo chmod 775 /home/pi/pikrellcam/media/videos 17:18:47 : execl[wait:1]: sudo mkdir -p /home/pi/pikrellcam/media/thumbs 17:18:47 : execl[wait:1]: sudo chown pi.www-data /home/pi/pikrellcam/media/thumbs 17:18:47 : execl[wait:1]: sudo chmod 775 /home/pi/pikrellcam/media/thumbs 17:18:47 : execl[wait:1]: sudo mkdir -p /home/pi/pikrellcam/media/stills 17:18:47 : execl[wait:1]: sudo chown pi.www-data /home/pi/pikrellcam/media/stills 17:18:47 : execl[wait:1]: sudo chmod 775 /home/pi/pikrellcam/media/stills 17:18:47 : execl[wait:1]: sudo mkdir -p /home/pi/pikrellcam/media/timelapse 17:18:47 : execl[wait:1]: sudo chown pi.www-data /home/pi/pikrellcam/media/timelapse 17:18:47 : execl[wait:1]: sudo chmod 775 /home/pi/pikrellcam/media/timelapse 17:18:47 : execl[wait:1]: sudo mkdir -p /home/pi/pikrellcam/media/stills/.thumbs 17:18:47 : execl[wait:1]: sudo chown pi.www-data /home/pi/pikrellcam/media/stills/.thumbs 17:18:47 : execl[wait:1]: sudo chmod 775 /home/pi/pikrellcam/media/stills/.thumbs 17:18:47 : execl[wait:1]: sudo mkdir -p /home/pi/pikrellcam/media/loop/videos 17:18:47 : execl[wait:1]: sudo chown pi.www-data /home/pi/pikrellcam/media/loop/videos 17:18:47 : execl[wait:1]: sudo chmod 775 /home/pi/pikrellcam/media/loop/videos 17:18:47 : execl[wait:1]: sudo mkdir -p /home/pi/pikrellcam/media/loop/thumbs 17:18:47 : execl[wait:1]: sudo chown pi.www-data /home/pi/pikrellcam/media/loop/thumbs 17:18:47 : execl[wait:1]: sudo chmod 775 /home/pi/pikrellcam/media/loop/thumbs 17:18:47 : execl[wait:1]: sudo chown pi.www-data /home/pi/pikrellcam/www/motion_detects_FIFO 17:18:47 : execl[wait:1]: sudo chmod 664 /home/pi/pikrellcam/www/motion_detects_FIFO 17:18:47 : execl[wait:1]: sudo chown pi.www-data /home/pi/pikrellcam/www/audio_FIFO 17:18:47 : execl[wait:1]: sudo chmod 664 /home/pi/pikrellcam/www/audio_FIFO 17:18:47 : video circular buffer - 26.25 MB (35 seconds, 6.0 Mbits/sec) 17:18:47 : Server running...waiting for connections. 17:18:47 : MJPEG server is listening on port '9999' sunrise/sunset times: valid dawn/dusk times: valid dawn: 13:25 sunrise: 13:51 sunset: 3:23 dusk: 3:49 17:18:52 : execl[wait:1]: /home/pi/pikrellcam/scripts-dist/_log-lines 500 /tmp/pikrellcam.log 17:20:26 : audio circular buffer - 1680.00 KB 840.00 KFrames (35 sec, 24000 samples/sec, 1 channels) 17:20:26 : mic opened: period_frames:600 period_buffer_size:1200 rate:24000 channels:1 MP3 quality:7 17:24:30 : command_process: record on 17:24:30 : Manual record start - /home/pi/pikrellcam/media/videos/manual_2019-08-02_17.24.30_0.h264 17:24:30 : preview save: copy /run/pikrellcam/mjpeg.jpg -> /run/pikrellcam/manual_2019-08-02_17.24.30_0.jpg 17:24:30 : execl[wait:0]: /home/pi/pikrellcam/scripts-dist/_thumb /run/pikrellcam/manual_2019-08-02_17.24.30_0.jpg /tmp/pikrellcam.log /home/pi/pikrellcam/media/thumbs/manual_2019-08-02_17.24.30_0.th.jpg 150x84 0 0 0 0; rm -f /run/pikrellcam/manual_2019-08-02_17.24.30_0.jpg convert /run/pikrellcam/manual_2019-08-02_17.24.30_0.jpg -resize 150x84! /home/pi/pikrellcam/media/thumbs/manual_2019-08-02_17.24.30_0.th.jpg 17:26:34 : command_process: record off 17:26:34 : execl[wait:0]: nice -n 5 MP4Box -quiet -tmp /tmp -fps 24.007 -add /home/pi/pikrellcam/media/videos/manual_2019-08-02_17.24.30_0.h264 -add /run/pikrellcam/manual_2019-08-02_17.24.30_0.mp3 /home/pi/pikrellcam/media/videos/manual_2019-08-02_17.24.30_0.mp4 2> /dev/null && rm -f /home/pi/pikrellcam/media/videos/manual_2019-08-02_17.24.30_0.h264 /run/pikrellcam/manual_2019-08-02_17.24.30_0.mp3; rm -f /run/pikrellcam/video_converting; 17:26:34 : Manual record stop (size:93.3MB < /tmp:9432.1MB) vid_time:124.55 vid_fps:24.01 audio_frames:2989094 audio_rate:24000 17:26:55 : command_process: record on 17:27:29 : command_process: record on 17:27:29 : Manual record start - /home/pi/pikrellcam/media/videos/manual_2019-08-02_17.27.27_1.h264 17:27:29 : preview save: copy /run/pikrellcam/mjpeg.jpg -> /run/pikrellcam/manual_2019-08-02_17.27.27_1.jpg 17:27:29 : execl[wait:0]: /home/pi/pikrellcam/scripts-dist/_thumb /run/pikrellcam/manual_2019-08-02_17.27.27_1.jpg /tmp/pikrellcam.log /home/pi/pikrellcam/media/thumbs/manual_2019-08-02_17.27.27_1.th.jpg 150x84 0 0 0 0; rm -f /run/pikrellcam/manual_2019-08-02_17.27.27_1.jpg convert /run/pikrellcam/manual_2019-08-02_17.27.27_1.jpg -resize 150x84! /home/pi/pikrellcam/media/thumbs/manual_2019-08-02_17.27.27_1.th.jpg 17:29:30 : command_process: record off 17:29:30 : execl[wait:0]: nice -n 5 MP4Box -quiet -tmp /tmp -fps 25.000 -add /home/pi/pikrellcam/media/videos/manual_2019-08-02_17.27.27_1.h264 -add /run/pikrellcam/manual_2019-08-02_17.27.27_1.mp3 /home/pi/pikrellcam/media/videos/manual_2019-08-02_17.27.27_1.mp4 2> /dev/null && rm -f /home/pi/pikrellcam/media/videos/manual_2019-08-02_17.27.27_1.h264 /run/pikrellcam/manual_2019-08-02_17.27.27_1.mp3; rm -f /run/pikrellcam/video_converting; 17:29:30 : Manual record stop (size:91.7MB < /tmp:9362.9MB) vid_time:122.21 vid_fps:24.01 audio_frames:2933111 audio_rate:24000 17:30:13 : command_process: record on 17:30:13 : Manual record start - /home/pi/pikrellcam/media/videos/manual_2019-08-02_17.30.12_2.h264 17:30:13 : preview save: copy /run/pikrellcam/mjpeg.jpg -> /run/pikrellcam/manual_2019-08-02_17.30.12_2.jpg 17:30:13 : execl[wait:0]: /home/pi/pikrellcam/scripts-dist/_thumb /run/pikrellcam/manual_2019-08-02_17.30.12_2.jpg /tmp/pikrellcam.log /home/pi/pikrellcam/media/thumbs/manual_2019-08-02_17.30.12_2.th.jpg 150x84 0 0 0 0; rm -f /run/pikrellcam/manual_2019-08-02_17.30.12_2.jpg convert /run/pikrellcam/manual_2019-08-02_17.30.12_2.jpg -resize 150x84! /home/pi/pikrellcam/media/thumbs/manual_2019-08-02_17.30.12_2.th.jpg 17:32:14 : command_process: record off 17:32:14 : execl[wait:0]: nice -n 5 MP4Box -quiet -tmp /tmp -fps 24.000 -add /home/pi/pikrellcam/media/videos/manual_2019-08-02_17.30.12_2.h264 -add /run/pikrellcam/manual_2019-08-02_17.30.12_2.mp3 /home/pi/pikrellcam/media/videos/manual_2019-08-02_17.30.12_2.mp4 2> /dev/null && rm -f /home/pi/pikrellcam/media/videos/manual_2019-08-02_17.30.12_2.h264 /run/pikrellcam/manual_2019-08-02_17.30.12_2.mp3; rm -f /run/pikrellcam/video_converting; 17:32:14 : Manual record stop (size:91.6MB < /tmp:9293.8MB) vid_time:122.21 vid_fps:23.01 audio_frames:2932937 audio_rate:24000 17:32:52 : command_process: record on 17:32:52 : Manual record start - /home/pi/pikrellcam/media/videos/manual_2019-08-02_17.32.51_3.h264 17:32:53 : preview save: copy /run/pikrellcam/mjpeg.jpg -> /run/pikrellcam/manual_2019-08-02_17.32.51_3.jpg 17:32:53 : execl[wait:0]: /home/pi/pikrellcam/scripts-dist/_thumb /run/pikrellcam/manual_2019-08-02_17.32.51_3.jpg /tmp/pikrellcam.log /home/pi/pikrellcam/media/thumbs/manual_2019-08-02_17.32.51_3.th.jpg 150x84 0 0 0 0; rm -f /run/pikrellcam/manual_2019-08-02_17.32.51_3.jpg convert /run/pikrellcam/manual_2019-08-02_17.32.51_3.jpg -resize 150x84! /home/pi/pikrellcam/media/thumbs/manual_2019-08-02_17.32.51_3.th.jpg 17:34:54 : command_process: record off 17:34:54 : execl[wait:0]: nice -n 5 MP4Box -quiet -tmp /tmp -fps 22.000 -add /home/pi/pikrellcam/media/videos/manual_2019-08-02_17.32.51_3.h264 -add /run/pikrellcam/manual_2019-08-02_17.32.51_3.mp3 /home/pi/pikrellcam/media/videos/manual_2019-08-02_17.32.51_3.mp4 2> /dev/null && rm -f /home/pi/pikrellcam/media/videos/manual_2019-08-02_17.32.51_3.h264 /run/pikrellcam/manual_2019-08-02_17.32.51_3.mp3; rm -f /run/pikrellcam/video_converting; 17:34:54 : Manual record stop (size:91.9MB < /tmp:9224.5MB) vid_time:122.64 vid_fps:23.01 audio_frames:2943367 audio_rate:24000

I also found the two comments on raspberry.org/forums/ http://raspberry.org/forums/ when I searched for the keyword „drift“.

by jbeale https://www.raspberrypi.org/forums/memberlist.php?mode=viewprofile&u=4008 Fri May 06, 2016 12:21 am Forum: Camera board https://www.raspberrypi.org/forums/viewforum.php?f=43 Topic: PiKrellCam: motion vector detect + OSD web interface https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=115583&hilit=drift Replies: 1441 Views: 301058 Re: PiKrellCam: motion vector detect + OSD web interface https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=115583&p=968635&hilit=drift#p968635 Do we know if NFS on the Pi has the performance required to handle live video recording? Has anyone shown that to work?

Regarding accurate timestamps, I have noticed that the "burned-in" date/time on the video occasionally has a hiccup when it doesn't update for a second or more. Also, on my cameras the nominally 24 fps is more accurately 23.04 fps. I measured this using an external strobe light and adjusting the rate for minimum drift. Jump to post https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=115583&p=968635&hilit=drift#p968635 by jbeale https://www.raspberrypi.org/forums/memberlist.php?mode=viewprofile&u=4008 Fri Nov 27, 2015 8:13 pm Forum: Camera board https://www.raspberrypi.org/forums/viewforum.php?f=43 Topic: PiKrellCam: motion vector detect + OSD web interface https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=115583&hilit=drift Replies: 1441 Views: 301058 "24 fps" isn't quite https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=115583&p=851023&hilit=drift#p851023 It probably doesn't matter to most people, but FWIW when PiKrellCam on my Pi is set to 24 fps, the actual recorded frame rate is approximately 23.05 fps. I measured this using an LED driven by a stable pulse generator and noticing the vertical drift rate of the horizontal image bands, due to interaction between the light and the rolling shutter start time on each scanline. I confirmed the pulse timing to within 0.1% with a scope.

I tried different cameras on the Raspberry PI 3+ but the drift is all the same.

This is the microphone from Amazon: USB Mikrofon, Gyvazla Omnidirektionaler Kondensator Lavalier Clip On Mic für Computer, Laptop, Podcast, Interviews, Netzwerksingen, Skype, MSN, Audio Video Recording [Plug & Play]

If you need as examples some videos with different settings that are out of sync, I can upload them on a server for you.

If it is a problem with the PI hardware and its internal clock going a little to slow or to fast, then may be your idea of a floating point ratio between fps and mp4box_fps might do the trick ;-)

Greetings, Gunneg

Am 01.08.2019 um 17:18 schrieb billw2 notifications@github.com:

On Wed, 31 Jul 2019 14:17:54 -0700 AntonJouri notifications@github.com wrote:

When recording video with audio, the mic is ahead of the video, and it is getting worse every minute. It is known, that the fps of Raspi-Camera and USB-Microphones usually don’t match. So I wonder, how do you compensate the drift?

I tried to compensate by setting the mp4box-fps just by 1 higher than fps, e.g. fps=23, mp4box=24, what tuned out as video is ahead of the mic-audio. I tried comma-values in the config-file directly, like fps=23.4 (period) or 23,3 (comma), and mp4box=23,8 or 23.9, and many other pairs of values, but as long as only integer values are processed, I can not find a matching value to compensate the drift.

Or, is there any parameter to adjust for the process that converts h246 and audio to mp4 to minimize the drift?

I´m using Raspberry Pi3+ with BusterLite as basis for PiKrellCam. May be I should downgrade?

Right now I´m out of ideas. Any hint appreciated.

Greetings, Gunneg

The config takes only integer values so there is no way to do any compensation now. I could change the type to float so factional values could be entered, but your asking this has made me think that a better way would be for pikrellcam to calculate a fps needed to make video time match audio time and use that in the mp4box. I'm wondering about the magnitude of the drift you get, so could you look in the pikrellcam log and tell me some rate results you get for 2 or 3 typical video records? Sound like you have long videos, but for a short video I get this for a "Motion record stop" line in the log:

Motion record stop (size:5.0MB < /tmp:257.5MB) vid_time:10.04 vid_fps:24.00 audio_frames:481965 audio_rate: 48000

If I could see some log numbers like that for some of your videos it would help and in the meantime I'll think about making the changes to pikrellcam of fractional fps config and automatic calculating a mp4box fps.

Thanks, Bill

-- Bill Wilson billw@gkrellm.net — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/billw2/pikrellcam/issues/47?email_source=notifications&email_token=AMYPTVYUCOWCWDL7JPE7DTLQCL5DRA5CNFSM4IIKUQ52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3K56LQ#issuecomment-517332782, or mute the thread https://github.com/notifications/unsubscribe-auth/AMYPTVY754T4SXSB7UE2KIDQCL5DRANCNFSM4IIKUQ5Q.

billw2 commented 5 years ago

Thanks for the data. I need to spend some time working out if I can get pikrellcam to auto detect and compensate for the audio drift. But in the meantime, I just added changes for setting fractional mp4box fps values and if you upgrade you can experiment with that.

Bill

On Fri, 02 Aug 2019 09:58:55 -0700 AntonJouri notifications@github.com wrote:

Thank you very much for your quick response, Bill.

I made a clean install and 4 different recordings of 2 minutes each.

1) 24 fps and 0 mp4box_fps From the beginning, audio is a little faster than video, after 2 minutes audio is 2 seconds faster than the video.

2) 24 fps and 25 mp4box_fps For the first 30 seconds it seems to be in sync, but then audio is slower than video, after 2 minutes 2 seconds behind.

3) 23 fps and 24 mp4box_fps (almost the same result) For the first 30 seconds it seems to be in sync, but then audio is slower than video, after 2 minutes 2 seconds behind.

4) 23 fps and 22 mp4box_fps After 2 minutes video is about 4 seconds faster than audio.

...

AntonJouri commented 5 years ago

Thanks, Bill. I will do, a.s.a.p. on monday morning :-) And I´m looking forward for the auto-detection-function. Thanks again for your help an your great PiKrellCam.

Guntram

Am 04.08.2019 um 17:42 schrieb billw2 notifications@github.com:

Thanks for the data. I need to spend some time working out if I can get pikrellcam to auto detect and compensate for the audio drift. But in the meantime, I just added changes for setting fractional mp4box fps values and if you upgrade you can experiment with that.

Bill

n1c085 commented 4 years ago

Hi, I'm experiencing the same problem with the audio. Is the auto-detection already implemented? Can you give me some advice on how to set the fps and mp4box values to compensate the audio shift?

Here's the log output for a video file: vid_time:46.02 vid_fps:24.01 audio_frames:1104599 audio_rate:24000

Thanks, Nick