figure out how to save daily video recordings to separate locations

[JamesPHoughton]

Right now, every consersation has its own folder in our S3 bucket, but we can't share a particular collaborator's videos with them in an easy way. Would like to be able to specify a parameter in the batch config (videoStorageLocation?) that then allows the videos to be stored in unique places.

my guess is that we could put them all in the existing S3, but under different folders, but also want to explore the route of creating separate S3 buckets for each collaborator.

[gumillie222]

Here's a document of how to create folders in buckets manually. Still looking into how to do so with code.

[JamesPHoughton]

maybe for now, we create the S3 buckets manually, and then just set the videos to stream to them in the config? That would be ok for a while, probably.

[gumillie222]

Ran into an issue: right now I'm trying to access batch configuration through useGlobal(), which is a hook and is showing error since meetingRoom.js(the file that specifies s3 buckets configuration) is in the backend. I'm wondering if there is another way to access batch config? Or specifying the storageLocation parameter using some other ways?

[JamesPHoughton]

You can get the globals in a callback from the context object (called ctx). It's generally the first argument in the function you pass to Empirica.on, like this:

Empirica.on("batch", async (ctx, { batch }) => {
    ...
    ctx.globals.get(...);

What we should probably do is get the variable from the context object and then pass that down into the function from meetingRoom.js that needs it. Something kinda like:

    const videoSaveBucket = ctx.globals.get(...)  // etc.
    const room = await CreateRoom(game.id, videoSaveBucket); 

[JamesPHoughton]

As you're working through this, can you work out how to get the full bucket and path to the video for each participant, and then save that to the player object and export it in the science data export? We can meet and talk about how that could work

[JamesPHoughton]

I'm leaning more towards using separate S3 buckets that we set up manually (for now). One advantage of this is that we can locate buckets in Europe when we need to in order to meet GDPR requirements.

[gumillie222]

As you're working through this, can you work out how to get the full bucket and path to the video for each participant, and then save that to the player object and export it in the science data export? We can meet and talk about how that could work

The default path is {domain_name}/{room_name}/{epoch_time}, but depending on if/how we set up different folders or not, it might change. For saving and exporting, it looks like in meetingRoom.js, we can player.set("pathToVideo", {path}, and in exportScienceData, add this into the playerData.

[gumillie222]

Under what condition will videos be saved into S3 buckets? For example, when I'm running some experiments locally, would the videos be saved?

[JamesPHoughton]

At the moment, yes. I was thinking we could have a bucket where we direct "testing" videos. We could also have a switch that tells daily not to record, I think.

[gumillie222]

I took a crack at sorting things into folders. recordings_template seemed not to work as intended. Adding "/" into name of the file was also causing problems. What we could do though, is to have a folder that contains all the files of recordings with the same videoStorageLocation config. The drawback is that there will be a lot of files in the same folder, so it might be hard to tell apart which files belong to the same game. But it's not impossible, as the file names should contain information about batch, player, room, etc.

[gumillie222]

If we take on the buckets path, do we know how many / what different buckets we need to set up manually? I'm assuming for putting files into different buckets we also get it from the parameter from batch config?

[gumillie222]

have a default bucket for all video files if not providing the correct videoStorageConfig ideally grab the list of authorized buckets from aws

James: would also be a good idea to figure out how to set the bucket region in the config, so that we can keep data in europe for their privacy regulations and also figure out how to turn off recording altogether, maybe by passing in “None” to the videoStorageLocation

[JamesPHoughton]

The main tasks in this issue are completed in https://github.com/Watts-Lab/deliberation-empirica/pull/546.

Testing that a valid bucket has been provided has been moved to https://github.com/Watts-Lab/deliberation-empirica/issues/557.

Excellent work on this one, @gumillie222