Audio lag in participant recordings

[smeylan]

Describe the bug Downloaded participant recordings have an approximately 2 second lag between video events and audio (video precedes audio)

Which frame or frames does it affect? exp-lookit-composite-video-trial, exp-lookit-images-audio

Example https://lookit.mit.edu/exp/studies/50/

To Reproduce Steps to reproduce the behavior (edit the example below):

1. Go to production and log in as an experimenter.
2. Create a study including the following frame definition.

{
  "frames": {   
    "video-config": {
      "kind": "exp-video-config",
      "troubleshootingIntro": "This is a standard frame you probably want to put at the very start of your study. You can customize this bit of text; the rest is standard and maintained by Lookit."
    },
    "video-consent": {
      "kind": "exp-lookit-video-consent",
      "PIName": "Lookit Tutorial Participant",
      "datause": "We are interested in how your child uses statistical evidence to figure out the cause of an event. A research assistant will watch your video and mark down your child's answer to the question at the end of the story, and as well as other information such as interactions between you and your child during the story.",
      "payment": "After you finish the study, we will email you a $5 BabyStore gift card within approximately three days. To be eligible for the gift card your child must be in the age range for this study, you need to submit a valid consent statement, and we need to see that there is a child with you. But we will send a gift card even if you do not finish the whole study or we are not able to use your child's data! There are no other direct benefits to you or your child from participating, but we hope you will enjoy the experience.",
      "purpose": "This study is about how children use statistical information to adjust their beliefs about cause and effect.",
      "PIContact": "Jane Smith at (123) 456-7890",
      "procedures": "In this study you child will view a digital 'storybook' about Bunny, who sometimes gets a tummyache. Each day Bunny eats different foods and does different activities, and we hear whether she gets a tummyache. Sometimes, Bunny feels scared because of show-and-tell. We are interested in how the pattern of evidence influences your child's beliefs about what causes Bunny's tummyache. We will ask you (the parent) to avoid discussing why Bunny has a tummyache until the end of the study. There are no anticipated risks associated with participating.",
      "institution": "Science University"
    },
    "storybook-plearn": {
      "kind": "group",
      "frameList": [
        {
          "images": [
            {
              "id": "storybook_title",
              "src": "https://umzy19bu4.ddns.net/plearn/order1/img/0_title.png",
              "top": 0,
              "left": 0,
              "height": 100
            }
          ],
          "parentTextBlock": {
            "css": {
              "font-size": "1.5em"
            },
            "text": "Let's read this book! It's called \"What's Here? A book about different things in different places\"",
            "title": "Parent reads to child..."
          }
        },
        {
          "images": [
            {
              "id": "storybook_1_1",
              "src": "https://umzy19bu4.ddns.net/plearn/order1/img/1_book_0.png",
              "top": 0,
              "left": 0,
              "height": 100
            }
          ],
          "parentTextBlock": {
            "css": {
              "font-size": "1.5em"
            },
            "text": "Look! There's one book! Can you say \"book\"?",
            "title": "Parent reads to child..."
          }
        },
        {
          "images": [
            {
              "id": "storybook_2_1",
              "src": "https://umzy19bu4.ddns.net/plearn/order1/img/2_chair_0.png",
              "top": 0,
              "left": 0,
              "height": 100
            }
          ],
          "parentTextBlock": {
            "css": {
              "font-size": "1.5em"
            },
            "text": "Look! There's one chair! Can you say \"chair\"?",
            "title": "Parent reads to child..."
          }
        },
        {
          "images": [
            {
              "id": "storybook_5_2",
              "src": "https://umzy19bu4.ddns.net/plearn/order1/img/5_fork_1.png",
              "top": 0,
              "left": 0,
              "height": 100
            }
          ],
          "parentTextBlock": {
            "css": {
              "font-size": "1.5em"
            },
            "text": "What's here?",
            "title": "Parent reads to child..."
          }
        },
        {
          "images": [
            {
              "id": "storybook_6_2",
              "src": "https://umzy19bu4.ddns.net/plearn/order1/img/6_gop_1.png",
              "top": 0,
              "left": 0,
              "height": 100
            }
          ],
          "parentTextBlock": {
            "css": {
              "font-size": "1.5em"
            },
            "text": "What's here?",
            "title": "Parent reads to child..."
          }
        },
        {
          "images": [
            {
              "id": "storybook_end",
              "src": "https://umzy19bu4.ddns.net/plearn/order1/img/20_the_end.png",
              "top": 0,
              "left": 0,
              "height": 100
            }
          ],
          "parentTextBlock": {
            "css": {
              "font-size": "1.5em"
            },
            "text": "The end! Great job!",
            "title": "Parent reads to child..."
          }
        }
      ],
      "commonFrameProperties": {
        "kind": "exp-lookit-images-audio",
        "baseDir": "https://umzy19bu4.ddns.net/plearn/order1/",
        "audioTypes": [
          "mp3",
          "ogg"
        ],
        "autoProceed": false,
        "doRecording": true,
        "parentTextBlock": {
          "css": {
            "font-size": "1.5em"
          },
          "text": "FIXME",
          "title": "For parents"
        }
      }
    },
    "gazetracking-1": {
      "kind": "exp-lookit-composite-video-trial",
      "isLast": false,
      "baseDir": "https://umzy19bu4.ddns.net/plearn/order1/",
      "sources": "1",
      "testCount": 1,
      "altSources": "1",
      "audioTypes": [
        "ogg",
        "mp3"
      ],
      "pauseAudio": "pause",
      "videoTypes": [
        "webm",
        "mp4"
      ],
      "startRecordingAutomatically": true,
      "attnSources": "attentiongrabber",
      "musicSources": "1",
      "unpauseAudio": "return_after_pause",
      "announceLength": 0,
      "backgroundColor": "black",
      "calibrationLength": 0,
      "calibrationAudioSources": "chimes",
      "calibrationVideoSources": "attentiongrabber"
    },
    "gazetracking-2": {
      "kind": "exp-lookit-composite-video-trial",
      "isLast": false,
      "baseDir": "https://umzy19bu4.ddns.net/plearn/order1/",
      "sources": "2",
      "testCount": 1,
      "altSources": "2",
      "audioTypes": [
        "ogg",
        "mp3"
      ],
      "pauseAudio": "pause",
      "videoTypes": [
        "webm",
        "mp4"
      ],
      "startRecordingAutomatically": true,
      "attnSources": "attentiongrabber",
      "musicSources": "2",
      "unpauseAudio": "return_after_pause",
      "announceLength": 0,
      "backgroundColor": "black",
      "calibrationLength": 0,
      "calibrationAudioSources": "chimes",
      "calibrationVideoSources": "attentiongrabber"
    }
  },
  "sequence": [
    "video-config",
    "video-consent",    
    "gazetracking-1",
    "gazetracking-2",
    "storybook-plearn"
  ]
}

4. Click 'preview' from the study edit page.
5. Proceed through the study
6. Accept the consent for the preview
7. Go to "View Study Responses" / "Videos", download the test trial videos and see that they all have audio lag around 2 seconds

Expected behavior Audio and video should be synced in the participant recordings (ie no delay)

Videos Storybook example: https://duke.box.com/shared/static/fbvgs1xv2aab6fy4qiopfqgbeno6jlyq.mp4. I read the prompt in the book. Preferential looking example: https://duke.box.com/shared/static/g9kudzvwidfca59y2gkf3cv5dnmri1ej.mp4. I clap at the point of disambiguation and count thereafter

Environment (please complete):

• Lookit environment: production, previewing a study
• frameplayer version: 2aa08ee6132cd6351eed58abc2253368c14ad184
• OS X 10.14; internal webcam; AirPods as well as internal microphone + headphones
• Chrome 84

[smeylan]

In case it's useful here's the event log. It looks like there is a ~2s delay between videoStreamConnection and startRecording

2020-08-05T19:34:36.377Z exp-lookit-images-audio:setupVideoRecorder 2020-08-05T19:34:36.379Z exp-lookit-images-audio:recorderReady 2020-08-05T19:34:36.624Z exp-lookit-images-audio:recorder.hasCamAccess 2020-08-05T19:34:36.630Z exp-lookit-images-audio:videoStreamConnection 2020-08-05T19:34:38.769Z exp-lookit-images-audio:startRecording 2020-08-05T19:34:38.770Z exp-lookit-images-audio:finishAudio 2020-08-05T19:34:38.770Z exp-lookit-images-audio:displayImages 2020-08-05T19:34:49.376Z exp-lookit-images-audio:trialComplete 2020-08-05T19:34:49.377Z exp-lookit-images-audio:stoppingCapture

[kimberscott]

If this is reproducible we should definitely prioritize it! We haven't generally been seeing this though so I'd like to get a better handle on the circumstances where it happens.

Does it also affect the consent video you record?

Would you be able to share the preview of the study you used to demo and have some other researchers try it out to collect some video examples from a variety of setups? (Naively I'd guess you'd want to collect some basic information about upload speed.)

[smeylan]

It does not affect the consent video.

Happy to share the preview -- what do I need to do beyond include the JSON above?

I am asking a labmate if they can replicate the issue on their machine

[smeylan]

Failed to replicate on Firefox 79.0, Ubuntu 16.04, integrated webcam on thinkpad t430, 62 down / 11 up; 51 ms ping. Finding more testers now...

Issue on OS X confirmed with 400 down / 400 up, 10 ms ping; so it doesn't seem to be a network issue

[smeylan]

Federica Bulgarelli in Elika's lab reproduced the error in Chrome 84, on OS X 10.14.6, with the integrated webcam on a 2019 MacBook Pro.

Firefox 79 is unaffected -- no audio lag -- on my own machine (10.14.6), so it appears to be something chrome-specific

[kimberscott]

Happy to share the preview -- what do I need to do beyond include the JSON above?

Check the "share preview" box under Edit Study, save, & you should see a link like this that you can share so that anyone can preview this study, without having to add them specifically to your study or have them make a new study with this json. That way you'll get the video/data all in one place too.

It's encouraging that this doesn't affect consent, at least, in that that suggests a problem with ember-lookit-frameplayer code rather than e.g. Pipe!

[smeylan]

Preview link is https://lookit.mit.edu/exp/studies/28d2d07e-92d7-45cb-b6d3-60dbfc910e2e/preview-detail/

[smeylan]

Audio lag was an issue again for a pilot participant who used Chrome.

[kimberscott]

Just to confirm, are you still seeing this? I still haven't been able to reproduce this in Chrome.

Can you try previewing in Chrome in the first few trials of: https://lookit.mit.edu/exp/studies/499/ (using attempt at fix) https://lookit.mit.edu/exp/studies/500/ (same protocol w/ latest version of master branch, to confirm we can reproduce error)

and let me know whether you see the audio lag in each?