This issue is to track and scope the remaining work around replacing Pipe with RecordRTC and AWS S3 in the lookit-api. Please edit/add/comment with anything I got wrong or missed!
These are things we need to do on the lookit API side to get RecordRTC/S3 working:
[x] 1. Update studies/experiment_builder.py with the new S3 environment variables needed for the docker container that builds the ember app. Done on staging/develophere.
[x] 2. Handle permissions for managing access to S3 videos. #1237
[x] 3. Pipe webhook: At the moment, we're using Pipe's webhook that fires upon uploading to S3. The handler that renames the video file is in /exp/views/video.py. We may need to trigger this in another way (since I believe it is dependent on Pipe?) and change the way it is authenticated (since it currently uses a Pipe webhook key). (No need to trigger the same view that's used for Pipe - instead we can just add to the existing API)
[x] 4. Add API endpoint that takes new video data POSTed from S3 and creates new video object in database. This will use token authentication rather than user credentials. #1237
[x] 5. Modify the views that provide video recordings so that the recordings are fetched from the correct bucket, depending on whether the study used pipe or recordrtc. These views include the participant study history page, researcher consent rulings, and researcher response video download (any other views?). #1245 (this also required adding recording_method to data sent from EFP, added here: https://github.com/lookit/ember-lookit-frameplayer/commit/8285603260e649afd22579a44a5539412582672f)
[x] 6. Write new video API tests in api/tests/. #1237
[x] 7. Write the AWS Lambda function that will be used for (a) saving a video object to the database after it has been saved to S3 and (b) logging the S3-related activity (using AWS CloudWatch).
Some of this work has generated new issues:
We also need a way of replicating this system (video upload to S3 -> Lambda -> lookit-api) in our local development environment, and deploying local changes to the cloud Lambda function. #1290
Ideally we would also provide the videos in mp4 format for researchers. #1255
These are things we'll need to do once Pipe is actually deprecated:
Possible database migration and related code query changes if we change column names such as pipe_name, pipe_numeric_id.
Update contributing.md and other docs related to setting up local dev environment. (Added to #1290)
@mekline update on this issue: of the list of 7 items above, 5 are either completed or will be completed very soon (when we merge #1237). These are the two remaining items that I'm working on right now:
Modify the views that provide video recordings so that the recordings are fetched from the correct bucket, depending on whether the study used pipe or recordrtc. --> #1245
Write the AWS Lambda function that will be used for (a) saving a video object to the database after it has been saved to S3 and (b) logging the S3-related activity. This function is partially written - I need to make it work for the different environments (local dev, staging, production) and it still needs to be reviewed.
This issue is to track and scope the remaining work around replacing Pipe with RecordRTC and AWS S3 in the lookit-api. Please edit/add/comment with anything I got wrong or missed!
These are things we need to do on the lookit API side to get RecordRTC/S3 working:
studies/experiment_builder.py
with the new S3 environment variables needed for the docker container that builds the ember app. Done on staging/develop
here.Pipe webhook: At the moment, we're using Pipe's webhook that fires upon uploading to S3. The handler that renames the video file is in(No need to trigger the same view that's used for Pipe - instead we can just add to the existing API)/exp/views/video.py
. We may need to trigger this in another way (since I believe it is dependent on Pipe?) and change the way it is authenticated (since it currently uses a Pipe webhook key).recording_method
to data sent from EFP, added here: https://github.com/lookit/ember-lookit-frameplayer/commit/8285603260e649afd22579a44a5539412582672f)api/tests/
. #1237Some of this work has generated new issues:
These are things we'll need to do once Pipe is actually deprecated:
pipe_name
,pipe_numeric_id
.contributing.md
and other docs related to setting up local dev environment. (Added to #1290)