Set up S3 Bucket for streaming deliberation videos

[JamesPHoughton]

We will need an S3 bucket that we can put recordings into. We won't be accessing the videos very frequently, at least at first - just to check them for QC. Eventually we'll want to run automated analysis against them.

I don't know much about how this works, so I have a few questions:

1. Are there different storage tiers with different costs, eg. long duration, high availability, etc.? Which do we want?
2. How do I access the bucket? Is this through the AWS console, logging in as my user?
3. How do we set who within our lab can access the bucket?
4. How do we give Daily permission to write to the bucket? How does Daily know where to write?
5. How do we give other researchers access to the bucket?
6. How do we give our analysis programs access to the bucket?
7. How much does it actually cost to store videos?

@rivera-lanasm

[rivera-lanasm]

Are there different storage tiers with different costs, eg. long duration, high availability, etc.? Which do we want?

• yes, AWS S3 has a few different tiers that differ in cost for data storage and put/get requests
• you can see a breakdown of the pricing model here
• The default tier is S3 Standard, which is meant for frequently accessed data. The storage price is $0.023 per GB (for the first 50 TB)
• S3 Intelligent Tiering is a relatively new offering that automatically moves your data between tiers (from higher to lower availability) as time goes on, automating cost savings. You can read more about that here, also includes good details about S3 tiers, and some detailed S3 cost breakdown examples for hypothetical data workloads.

How do I access the bucket? Is this through the AWS console, logging in as my user?

• you should have access to the AWS console in the browser via aws.cloud.upenn.edu/, and should be able to sign on with your PennKey
• there is also the AWS CLI and Python API
• I've been developing and using a package bundling frequently used methods based on the Python API for the lab
• I can create a bucket in the Deliberation AWS account to start testing, is there a name in mind? The naming convention is "wattslab-deliberation-..."

How do we set who within our lab can access the bucket?

• AWS permissions are administered by Penn though "users" linked to our PennKeys
• Access is administered based on who (what AWS Users) in the lab has access to the Deliberation AWS Account. By default, anyone who has access to the Deliberation account should also have access to the bucket, unless we further restrict the bucket access

How do we give Daily permission to write to the bucket? How does Daily know where to write?

• I've set up something similar before, giving an external party partial access to an AWS bucket (read/write permission for example).
• We can create a temporary IAM User that just has the permissions we specify. Passing the credentials to the data vendor, they can use this in their development process, and we can shut the user down at any time.

How do we give other researchers access to the bucket? How do we give our analysis programs access to the bucket?

• as long as they have access to the deliberation AWS account they should be fine

How much does it actually cost to store videos?

• If we can work with some sample video files and have an idea of the amount of videos / month expected to be written to the bucket, we can start estimating costs going forward.

Other thoughts In my experience, it's very important to map out how you expect the data to enter S3 and be utilized by users as soon as possible. The main things to keep in mind are:

1. Writing data to S3 in such a way it is easy to search for by downstream processes. This means using S3 partitions well.
   • partitions
2. Avoid large number of small files where possible. API calls are charged per object, regardless of its size. Uploading 1-byte costs the same as uploading 1GB.
3. S3 is a key-value store, not a file system
4. Think early on about how to archive data you don't plan to use for the foreseeable future, but would still like to have. Collecting files with zip for example.

[Alan-Qiao]

This is documentation from daily on recording to custom s3 bucket. https://docs.daily.co/guides/products/live-streaming-recording/storing-recordings-in-a-custom-s3-bucket

[JamesPHoughton]

Thanks @rivera-lanasm, this is a really helpful breakdown. Can you help me set up the following?

S3 bucket

• named "wattslab-deliberation-videos"
• using Intelligent Tiering, everything down to the Archive Instant access tier, but not the asynchronous access levels.
• daily needs the bucket to enable "bucket versioning" (https://docs.daily.co/guides/products/live-streaming-recording/storing-recordings-in-a-custom-s3-bucket#s3-bucket-configuration-requirements)
• should be in a US location, beyond that whatever is likely to be cheapest

IAM policy

Daily has some requirements for access permissions, using this policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:ListBucketMultipartUploads",
        "s3:AbortMultipartUpload",
        "s3:ListBucketVersions",
        "s3:ListBucket",
        "s3:GetObjectVersion",
        "s3:ListMultipartUploadParts"
      ],
      "Resource": [
        "arn:aws:s3:::your-bucket-name",
        "arn:aws:s3:::your-bucket-name/*"
      ]
    }
  ]
}

IAM role

Daily docs Trusted Entity Type: AWS Account Trusted AWS account ID: 291871421005 Required external ID: deliberation Maximum session duration: 12 hours

We'll need the Amazon Resource Name (ARN) for the role, and the bucket region to include in the daily API calls.

[JamesPHoughton]

From @rivera-lanasm:

• Defining iam policies with jsons is generally best I think
• For region we use us east 1 across the board I think
• bucket versioning is more a feature for us to use rather than Daily (so they don’t have to resend us data if we accidentally delete for example) so we should consider how to use that in the process (https://docs.aws.amazon.com/AmazonS3/latest/userguide/versioning-workflows.html)

Setting up the bucket

• name - "wattslab-deliberation-videos"
• region - "us-east-1"
• object ownership - ACLs disabled
• public access - Block all public access
• Bucket Versioning - enabled
• Tags - none
• Default Encryption - Disable
• Object Lock - Disable

Creating IAM policy

• name - daily_video_upload
• tags - {"project":"deliberation", "service":"daily.co"}
• permissions:

  {
  "Version": "2012-10-17",
  "Statement": [
  {
    "Sid": "VisualEditor0",
    "Effect": "Allow",
    "Action": [
      "s3:PutObject",
      "s3:GetObject",
      "s3:ListBucketMultipartUploads",
      "s3:AbortMultipartUpload",
      "s3:ListBucketVersions",
      "s3:ListBucket",
      "s3:GetObjectVersion",
      "s3:ListMultipartUploadParts"
    ],
    "Resource": [
      "arn:aws:s3:::wattslab-deliberation-videos",
      "arn:aws:s3:::wattslab-deliberation-videos/*"
    ]
  }
  ]
  }

Setting up the role:

• trusted entity type - AWS account
• which AWS account - another AWS account: 291871421005
• require external id - "deliberation"
• permission policy - "daily_video_upload"
• role name - "dailyco_video_upload"
• tags - {"project":"deliberation", "service":"daily.co"}

  
  {
  "Version": "2012-10-17",
  "Statement": [
      {
          "Effect": "Allow",
          "Action": "sts:AssumeRole",
          "Principal": {
              "AWS": "291871421005"
          },
          "Condition": {
              "StringEquals": {
                  "sts:ExternalId": "deliberation"
              }
          }
      }
  ]
  }

## Questions for @rivera-lanasm 

- [ ] Should we use [default encryption of videos](https://docs.aws.amazon.com/AmazonS3/latest/userguide/default-bucket-encryption.html)? What friction does it introduce, and what additional security does it buy beyond the existing permission set? Does it cost money?
- [ ] Is it a good idea to use "object lock" for our videos?
- [ ] How do we set up the intelligent tiering? Or is that enabled by default? There is a setting for an archive configuration, but that seems to be for the asynchronous access options.
- [ ] Do these ARN's need to be kept secret, or can they be committed to a public repository

## Todo:

- [ ] Delete old bucket `wattslab-deliberation-eyeson` that we are not using any more
- [ ] Check in with daily about why they suggest versioning, and what we expect it to cost.

[rivera-lanasm]

• ARN's themselves aren't sensitive information, they aren't credentials. https://devops.stackexchange.com/questions/11101/should-aws-arn-values-be-treated-as-secrets

• I've only used default encryption, I've never read anything encouraging something other than that

• Object lock is a setting you can enable when creating a bucket, and gives option to "lock" objects so they can't be deleted or overwritten. This seems like a similar service as bucket/object versioning. So maybe makes sense to go with one or the other https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html

• I haven't set up s3 intelligent tiering before, but my understanding is that you can transfer from the default tier to intelligent tiering. I'll have to read more into this

[JamesPHoughton]

After reaching out to daily to see why it wasn't working, they replied with:

On a previous case where we've seen the same error occur, this has been solved by increasing the maximum session duration to 12 hours

I've updated the session length to 12 hours.