Black video in active speaker artifacts

[razorxan]

What happened and what did you expect to happen?

The app we developed is using CreateMediaCapturePipelineCommand to record the audio/video streams the meeting sessions.

the configurations of the pipeline are:

ChimeSdkMeetingConfiguration: {
    ArtifactsConfiguration: {
        Audio: {
            MuxType: 'AudioWithActiveSpeakerVideo'
        },
        Video: {
            State: 'Enabled',
            MuxType: 'VideoOnly'
        },
        Content: {
            State: 'Enabled',
            MuxType: 'ContentOnly'
        }
    }
}

The attendees in the meeting see and hear eachother, nevertheless the active speaker artifacts (in the audio path of the sink s3 bucket), are BLACK VIDEOS with audible audio.Seems like the active speaker is not detected, at least , his video stream is not attached to the active-speaker mp4 chunk.

Videos artifacts (video only) of the individual attendees, are correctly showing video content.

Have you reviewed our existing documentation?

• [X] GitHub issues
• [X] FAQs page
• [X] README page
• [X] Feature-specific guides

Reproduction steps

Once created a chime meeting with the @aws-sdk/client-chime, I just create a pipeline like so

await chime.send(new CreateMediaCapturePipelineCommand({
    SourceType: 'ChimeSdkMeeting',
    SinkType: "S3Bucket",
    SinkArn: `arn:aws:s3:::${bucket_name}/${code}/recordings/${+Date.now()}--${meeting.Meeting.MeetingId}`,
    SourceArn: `arn:aws:chime::XXXXXXXXXX:meeting:${meeting.Meeting.MeetingId}`,
    ChimeSdkMeetingConfiguration: {
        ArtifactsConfiguration: {
            Audio: {
                MuxType: 'AudioWithActiveSpeakerVideo'
            },
            Video: {
                State: 'Enabled',
                MuxType: 'VideoOnly'
            },
            Content: {
                State: 'Enabled',
                MuxType: 'ContentOnly'
            }
        }
    }
}));

Amazon Chime SDK for JavaScript version

amazon-chime-sdk-js version 3, @aws-sdk/client-chime version 3.36.1

What browsers are you seeing the problem on?

Chrome

Browser version

Version 108.0.5359.98 (Official Build) (x86_64)

Meeting and Attendee ID Information.

PipelineId: 1abe7502-02d2-4a5a-b02a-33da2af2e47d MeetingId: a80dcde3-d4dd-41e0-86b4-18b91b3e0706

Browser console logs

Not available

[razorxan]

In addition to the issue i found out that

if i remove the ChimeSdkMeetingConfiguration completely

the artifacts for the active speaker show the video stream as expected

[xuesichao]

Hi @razorxan , I tagged the corresponding service team for help.

I'm not that familiar with media capture pipeline. I tried to find some information from our documentation. From Working with media capture artifacts, it seems the different type of artifacts are captured separately. So I assume it's expected that the audio artifact has black/blank/no video stream attached.

You should be able to concatenate these media artifacts by following creating media concatenation pipelines. Could you please give it a try while waiting from the response from our service team?

[aws-rafams]

Hi @tenge-amzn, could you give some advice? TIA.

[tenge-amzn]

Will check. Maybe there's a bug on the config parsing. How about trying with this config:

ChimeSdkMeetingConfiguration: {
    ArtifactsConfiguration: {
        Audio: {
            MuxType: 'AudioWithActiveSpeakerVideo'
        },
        Video: {
            State: 'Disabled',
            MuxType: 'VideoOnly'
        },
        Content: {
            State: 'Disabled',
            MuxType: 'ContentOnly'
        }
    }
}

[tenge-amzn]

@PrijalBista We're investigating this issue. We haven't been able to reproduce so it might be related to video join timing. Meanwhile, are you able to try other layouts that's not bound to video timing? Like AudioWithCompositedVideo layout.

You can find examples here: https://aws.amazon.com/blogs/business-productivity/capture-amazon-chime-sdk-meetings-using-media-capture-pipelines/

[swapnil0504]

Any luck on this ? I'm also facing the same.