The AWS S3 multi-part upload system (used in EFP and jsPsych) can result in incomplete video uploads (e.g. if the complete request fails or the participant leaves the study early - see https://github.com/lookit/ember-lookit-frameplayer/issues/395), and researchers are requesting access to those partial recordings. We need an automated task for completing partial/incomplete video uploads.
Description
We are trying to prevent video recording uploads from remaining in an incomplete state (see https://github.com/lookit/ember-lookit-frameplayer/issues/395), but that is not always possible, so we still need a way of handling the video uploads that remain incomplete in S3. We discussed simply deleting the incomplete uploads after some duration, but we have been receiving feedback from researchers that videos from incomplete sessions are still helpful (for verifying that a child participated for compensation, and sometimes obtaining useable data, e.g. when the video recording is mostly complete). Because of the frequency of requests we're getting for partial/incomplete video recordings, it makes sense to create a Celery job to automatically try to complete any incomplete video uploads so that they become available to the researcher.
It is not always possible to complete an incomplete upload. Sometimes the upload was created but is empty, corrupted, or missing parts. In these cases, we should probably just delete the partial upload in order to clear our S3 storage space.
Acceptance criteria
[x] Decide on a minimum age for the upload creation before attempting to complete it, to ensure that we're not automatically completing the uploads from in-progress sessions. One day (24 hrs) is probably sufficient.
[ ] Communicate to researchers that they may need to wait a certain amount of time (e.g. 1 day) to see incomplete videos appear in participants' responses.
[x] Decide what to do with incomplete uploads that cannot be completed via the Celery task. Delete immediately, or allow the AWS rule to delete it after some period (30 days)? Log their presence/deletion somewhere?Decided to leave them and allow the AWS lifecycle rule to delete after 30 days. Deleting via the lookit-api seems unnecessarily risking, and leaving them for 30 days also serves as a temporary "log" - we can check the in-progress MPU list if a researcher wants to know why an incomplete video hasn't shown up after ~1 day.
Summary
The AWS S3 multi-part upload system (used in EFP and jsPsych) can result in incomplete video uploads (e.g. if the complete request fails or the participant leaves the study early - see https://github.com/lookit/ember-lookit-frameplayer/issues/395), and researchers are requesting access to those partial recordings. We need an automated task for completing partial/incomplete video uploads.
Description
We are trying to prevent video recording uploads from remaining in an incomplete state (see https://github.com/lookit/ember-lookit-frameplayer/issues/395), but that is not always possible, so we still need a way of handling the video uploads that remain incomplete in S3. We discussed simply deleting the incomplete uploads after some duration, but we have been receiving feedback from researchers that videos from incomplete sessions are still helpful (for verifying that a child participated for compensation, and sometimes obtaining useable data, e.g. when the video recording is mostly complete). Because of the frequency of requests we're getting for partial/incomplete video recordings, it makes sense to create a Celery job to automatically try to complete any incomplete video uploads so that they become available to the researcher.
It is not always possible to complete an incomplete upload. Sometimes the upload was created but is empty, corrupted, or missing parts. In these cases, we should probably just delete the partial upload in order to clear our S3 storage space.
Acceptance criteria
Delete immediately, or allow the AWS rule to delete it after some period (30 days)?Log their presence/deletion somewhere?Decided to leave them and allow the AWS lifecycle rule to delete after 30 days. Deleting via the lookit-api seems unnecessarily risking, and leaving them for 30 days also serves as a temporary "log" - we can check the in-progress MPU list if a researcher wants to know why an incomplete video hasn't shown up after ~1 day.