Log participant begins speaking, participant stops speaking

[JamesPHoughton]

As an experimenter, I want to log the times that participants begin and end speaking, so that I can display to participants a dial indicating how much of the time they have been speaking for so far, and also save outcome measurements like the number of turns taken, the length of each turn, how much each person spoke over the course of the conversation, how many people were participating, etc.

In our discussion with Daily tech reps, they indicate that the client-side API exposes events when a speaker starts and stops talking. We should be able to tap into this and append to an speaking time event log on the player object, so that after the fact we can reconstruct the whole history of when each player was talking by extracting events from all player objects.

We should also have a flag for whether a player is speaking currently, to help with the live meter. We'll probably need to compute the total conversation time when we get a player-stops-talking event, and save it to the player object, and then to display the meter to the participant, also will need to add the amount of time since they started talking.

This will happen in client/src/components/VideoCall.jsx.

This is going to require looking through the daily docs and finding out how to write a function similar to:

    callFrame.on("track-started", (event) => {
      // Why are these not triggering correctly???
      if (event.participant.local) {
        if (event.track.kind === "video") {
          player.set("videoEnabled", true);
          console.debug("player video started");
        } else if (event.track.kind === "audio") {
          player.set("audioEnabled", true);
          console.debug("player audio started");
        }
        console.debug("track-started", event);
      }
    });

except instead of "track-started" it fires on whatever callback is associated with the player speaking or stopping.

If it turns out we cant do this qwiht a daily callback we may neet to take volume off the microphone level.

[JamesPHoughton]

Meeting event called "active speaker change" and "active speaker mode change"? Track started also tracks audio - what does it do?

Can we get live volume levels from somewhere? What does active speaker change and callbacks do?

Millie has looked at some of this too. Michael and MIllie will do some partner programming.

What gets logged in the callback. empirica has "append"

player.append("key", value")

key should be something like "speakingEvents", event should be something like

{
type: "start" | "stop"
timestamp: Date.now()
cumulative: currentCumulative + elapsed in this speaking chunk
}

set player current speaking:

game.set("currentSpeaker", player.id)

Data collection

need to log the player speakingEvents in exportScienceData

    const playerData = {
      deliberationId: participantData.deliberationId,
      batchId,
      config: batch?.get("config"),
      timeArrived: player?.get("timeArrived"),
      consent: player?.get("consent"),
      introSequence: player?.get("introSequence"),
      gameId,
      treatment: player?.get("treatment"),
      position: player?.get("position"),
      recordingIds: player?.get("dailyIds"),
      recordingRoomName: game?.get("dailyRoomName"),
      surveys,
      prompts,
      qualtrics,
      QCSurvey: player?.get("QCSurvey"),
      exitStatus: player?.get("exitStatus"),
      exportErrors,
      // Todo: speakerEvents  <-- put it here
    };

[JamesPHoughton]

Hi @gumillie222,

The last notes I have on this issue are that you have it working using the speaker change callback - can you give me an update in this issue just so we can track status, and your progress on using the "volume" information instead of the callback?

[gumillie222]

ran into this issue

James said we might need to npm install a new library. Will try that in a new branch later.

[gumillie222]

Client page successfully loaded after I npm installed the daily-react library, but the following error is showing:

I've tried to debug it for a bit but couldn't figure it out. @JamesPHoughton do you have any idea why?

[JamesPHoughton]

hooks are tricky, there are some very particular rules about when you can call them. Here are the official docs on hooks, but it's a tricky subject, which means lots of people have good youtube videos explaining it.