Streamlining Data Archival Requests

[balajialg]

Summary

We currently get around 5-10 requests every month from students to retrieve their previous semester's worth of data. Some of the common challenges encountered while servicing such requests are,

• Students request their previous semester's or year's worth of data. Currently, We don't have a clear policy around how many semesters' worth of data can be archived and shared. As a result, we store all the previous semester's data without deleting it regularly. Storage keeps growing continuously costing us money in the longer run.
• Students are not aware of our existing archival policies. Without clarity, they request data from courses they took multiple years ago.
• Entire responsibility to execute the end-to-end process for retrieving the archival data resides at @felder's end which creates an additional bottleneck from a bandwidth standpoint.

User Stories

• As a current or former datahub user, I'd like to retrieve a copy of my archived files so that I can use them as a reference for my personal/professional goals.
• As an infrastructure admin, I want to control the growth of the hub disk space in order to reduce the money spent.

Important information

Some of the proposed solutions from the sprint planning meeting:

Tasks to be done:

• [ ] Define a policy around how many months/years' worth of data can be archived and retrieved from Datahub.
• [ ] Communicate (preferably by email) with students during a specific timeline (probably at the end of the semester) to download their semester's worth of data.
• [ ] For course-specific hubs, Get permission from instructors to wipe the disk at the end of the semester.
• [ ] Develop a UI-based solution that allows students to retrieve their data without any manual intervention!

[felder]

I'd say the user story here should be modified.

As a current or former datahub user, I'd like to retrieve a copy of my archived files.

I believe accounts are only archived after 12 months of inactivity. So it's more than just the previous semester. In fact it's also possible requests could come in from people who have graduated!

[balajialg]

@felder Thanks for the correction, Updated!

[yuvipanda]

In the long run, it might be nice to just have a URL in the note we leave the user that they can use to fetch the files themselves, without any involvement from us whatsoever.

[balajialg]

@yuvipanda Exactly, that's the ideal endpoint for this user story. We should not be involved in user's requests for their data!

[yuvipanda]

Related to https://github.com/berkeley-dsep-infra/datahub/issues/2866