Watts-Lab / deliberation-empirica

Empirica V2 framework
MIT License
6 stars 0 forks source link

Log participant begins speaking, participant stops speaking #99

Closed JamesPHoughton closed 1 year ago

JamesPHoughton commented 2 years ago

As an experimenter, I want to log the times that participants begin and end speaking, so that I can display to participants a dial indicating how much of the time they have been speaking for so far, and also save outcome measurements like the number of turns taken, the length of each turn, how much each person spoke over the course of the conversation, how many people were participating, etc.

In our discussion with Daily tech reps, they indicate that the client-side API exposes events when a speaker starts and stops talking. We should be able to tap into this and append to an speaking time event log on the player object, so that after the fact we can reconstruct the whole history of when each player was talking by extracting events from all player objects.

We should also have a flag for whether a player is speaking currently, to help with the live meter. We'll probably need to compute the total conversation time when we get a player-stops-talking event, and save it to the player object, and then to display the meter to the participant, also will need to add the amount of time since they started talking.

This will happen in client/src/components/VideoCall.jsx.

This is going to require looking through the daily docs and finding out how to write a function similar to:

    callFrame.on("track-started", (event) => {
      // Why are these not triggering correctly???
      if (event.participant.local) {
        if (event.track.kind === "video") {
          player.set("videoEnabled", true);
          console.debug("player video started");
        } else if (event.track.kind === "audio") {
          player.set("audioEnabled", true);
          console.debug("player audio started");
        }
        console.debug("track-started", event);
      }
    });

except instead of "track-started" it fires on whatever callback is associated with the player speaking or stopping.

If it turns out we cant do this qwiht a daily callback we may neet to take volume off the microphone level.

JamesPHoughton commented 1 year ago

Meeting event called "active speaker change" and "active speaker mode change"? Track started also tracks audio - what does it do?

Can we get live volume levels from somewhere? What does active speaker change and callbacks do?

Millie has looked at some of this too. Michael and MIllie will do some partner programming.

What gets logged in the callback. empirica has "append"

player.append("key", value")

key should be something like "speakingEvents", event should be something like

{
type: "start" | "stop"
timestamp: Date.now()
cumulative: currentCumulative + elapsed in this speaking chunk
}

set player current speaking:

game.set("currentSpeaker", player.id)

Data collection

need to log the player speakingEvents in exportScienceData

    const playerData = {
      deliberationId: participantData.deliberationId,
      batchId,
      config: batch?.get("config"),
      timeArrived: player?.get("timeArrived"),
      consent: player?.get("consent"),
      introSequence: player?.get("introSequence"),
      gameId,
      treatment: player?.get("treatment"),
      position: player?.get("position"),
      recordingIds: player?.get("dailyIds"),
      recordingRoomName: game?.get("dailyRoomName"),
      surveys,
      prompts,
      qualtrics,
      QCSurvey: player?.get("QCSurvey"),
      exitStatus: player?.get("exitStatus"),
      exportErrors,
      // Todo: speakerEvents  <-- put it here
    };
JamesPHoughton commented 1 year ago

Hi @gumillie222,

The last notes I have on this issue are that you have it working using the speaker change callback - can you give me an update in this issue just so we can track status, and your progress on using the "volume" information instead of the callback?

gumillie222 commented 1 year ago

ran into this issue screenshot_2023-06-26_at_4 56 07_pm_720

James said we might need to npm install a new library. Will try that in a new branch later.

gumillie222 commented 1 year ago

Client page successfully loaded after I npm installed the daily-react library, but the following error is showing:

Screenshot 2023-06-30 at 2 37 46 AM

I've tried to debug it for a bit but couldn't figure it out. @JamesPHoughton do you have any idea why?

JamesPHoughton commented 1 year ago

hooks are tricky, there are some very particular rules about when you can call them. Here are the official docs on hooks, but it's a tricky subject, which means lots of people have good youtube videos explaining it.