Provide backup feature for projects

[carlybatist]

Context

From @carlybatist and Ant/Marconi -

We should have a way to provide partners with a backup of all their data from a project (all raw recordings, results of jobs they've run, all metadata, all Insights maps/figures).

This feature will fulfill compliance with EU law (GDPR) - all users should (read: need to) have a way to download their data.

Expected results

• [x] Project settings page is changed to 1 column layout as per design
• [x] Project settings contains "Project backup" section
• [x] "Request backup" is disabled if you are not a project owner (same as delete)
• [x] "Request backup" is disabled if there was a backup requested in the last 7 days
• [x] When the user presses "Request backup", a new row shows up on the right with status "Requested"
• [x] The row changes from "Requested" to "In progress" when the job starts
• [x] Within 24 hours, the job is completed and shows "Success"
• [x] When the job is completed, an email is sent to the user containing the download link.
• [x] The link to the project backup expires after 7 days.
• [x] Only show the last 3 requests. Disable the expire link.
• [x] Any tables with a uri/url/file associated with them will have an S3 signed url generated (to enable programatic downloading) that will expire after 7 days
• [x] Project owner and Admin can make backup request

Additional notes

• The user cannot cancel or delete a backup

Design

https://www.figma.com/file/GjR3UAHkQyGvx1iZzdZdDw/%F0%9F%92%BB-Arbimon-Platform?type=design&node-id=9914-6375&mode=design&t=ZfYWta8cBYYoo7ts-4

Copy:

You can request a comprehensive backup—including all raw recordings, job results, metadata, and Insights visuals—every 7 days. Once requested, your download link will be ready within 24 hours. Remember, download links expire after 7 days, so be sure to save your backup promptly.

Proposed technical implementation

1. When the project admin/owner requests a backup, it adds a row to a new "backup" table containing project id, requested by (user), requested at (timestamp), expires at (timestamp), status (requested, processing, available), url, size (number of mbs).
2. After the user requests a backup, they will not be able to request another backup for 30 days. (to limit overuse of this feature)
3. Background job runs every night (middle of the night US time) that will pick up any requested backups and start processing them one at a time.
4. Backup process:
   • change status to processing
   • export sites.csv, species.csv
   • export recordings.csv (generating a signed url for each uri)
   • export playlists.csv, playlist_recordings.csv
   • export templates.csv (generating a signed url for each uri)
   • export recording_validations.csv
   • export pattern_matchings.csv, pattern_matching_rois.csv (generating a signed url for each uri), (maybe pattern_matching_validations.csv?)
   • export soundscapes.csv (generating a signed url for each uri) _- export classifications.csv, model_training.csv (maybe trainingset.csv)? - export AED.csv, clustering.csv
   • zip all csv files, upload to S3 export folder
   • change status to available
   • send email to requesting user that the backup is available with the link in the email

Related APIs

• [x] Set up new database table to store backup data (#1903)
• [x] Backup - GET backup history list (by project id) (#1903)
• [x] Backup - POST backup to create a new backup request (#1903)
• [x] #1904

Other related tasks

• [ ] Publish python script to download all files from backup csv

[tiffanylee0125]

Prototype

[tiffanylee0125]

Empty state

Display past requests

@antonyharfield is this what you have in mind?

[tiffanylee0125]

Should people have the ability to cancel a backup and get another backup opportunity in the same 30 days as when the original backup was requested?

[carlybatist]

The easiest answer would be no?

[antonyharfield]

The part that's been started is the table export for sites and recordings, and adding signed urls for recordings. It is here: apps/cli/src/export/project-csv/index.ts.

[naluinui]

Dev updates

We have finished the user facing part, and now ready to test on https://dev.arbimon.org/

Project settings page (change design as per Figma)

https://github.com/rfcx/arbimon/assets/9149523/7b248f93-7f66-405f-a0e2-b38f697c7046

• change to 1 column as per design
• tweak some styling (input box border color, button size)
• move up save button to the top of the list
• warn users if there is some unsaved changes before they leave the page

Project backup

https://github.com/rfcx/arbimon/assets/9149523/6d58e578-40c1-49bc-bef9-637925f07237

• Only "owner" can see the Project backup section
• User can request backup (every 15 mins in dev, so it's easier to test)
• User can see the list of requested backup & its status

What's left

We're now working the cronjob part to actually do the export job, send email, and update status back to the UI.

FYI @koonchaya you can go ahead and get familiar with the changes on dev :)

[koonchaya]

• [x] Save button is on top of the page (in the design is in the bottom)

• [x] Reposition date picker delete button

• [x] Test project section is missing the warning message

[naluinui]

Save button is on top of the page (in the design is in the bottom)

Which Figma file do you compare with Noon? Please see our discussion here. For the full perspective behind moving the button to the top.

[naluinui]

Test project section is missing the warning message

This will only show up when you have published insight page. (Same as existing flow)

Version	WITH published insight	WITHOUT published insight
Current
New design

[naluinui]

Reposition date picker delete button

This is due to current library we used for date picker. It'll be fixed as part of date picker task https://github.com/rfcx/arbimon/issues/1573 next sprint

[koonchaya]

Requested popup:

• [x] header text (Backup requested/Request project backup) font isn't the same as design
• [ ] background color of the popups are different than the design (This current color is being used on the popup of the website)
• [ ] there is no "X" button on popup in the design

Design	Current

Design	Current

[naluinui]

Thanks @koonchaya, I just fixed the font size and icon padding as part of this task.

I'm not sure about the background color & close icon as those still valid in other places. I think we might want to keep them consistent throughout the platform and change them altogether. (there seems to be 8 places using the current style).

Please create another improvement task for this and we can tackle all for the 8 popups we have at once (we may need to do the same in arbimon-legacy too)

[koonchaya]

Do we allow only project owner to request backup?

[koonchaya]

@naluinui Can you adjust the warning box like design?	Design	Current

[koonchaya]

How do I check if the url in export files are signed url?

[antonyharfield]

How do I check if the url in export files are signed url?

If you open a new private/incognito browser window (so that you are not logged in) and paste in the url then it should download a file.

[naluinui]

Do we allow only project owner to request backup?

Yes, only project owner can see this UI

[koonchaya]

This is the error when I tried to open Url from

1. recordings.csv

2. templates.csv

3. pattern_matching_rois.csv

4. soundscapes.csv? - I don't have soundscape job in my project to check.

Example files

recordings.csv

templates.csv

pattern_matching_rois.csv

Additional

• My pattern_matching_validations.csv is empty. I don't know what would be in the validations export. (My project has 1 species validated)

pattern_matching_validations.csv

PM job that has the validation https://staging.arbimon.org/project/create-18-apr-2024/analysis/patternmatching/611

[naluinui]

💡Ideas from Sprint review call

• [ ] add brief information about each csv files
• [ ] https://github.com/rfcx/arbimon/issues/1943

[koonchaya]

Add more export files

• export classifications.csv, model_training.csv (maybe training_set.csv?)
• export AED.csv, clustering.csv

[antonyharfield]

Solution to permission problem

https://stackoverflow.com/questions/72552236/generate-s3-presigned-url-in-cross-account-bucket

[koonchaya]

I tested the Admin backup and waited overnight. The request is still in progress. Can you check why it is taking a long time? This is my test project https://staging.arbimon.org/p/the-rooftop/settings

[koonchaya]

@naluinui @antonyharfield The url in

• pattern_matching_rois.csv isn't working

• recordings.csv isn't working

FYI, I just downloaded this file today (8 May)

[koonchaya]

@antonyharfield @LucyDimitrova @naluinui I checked url from recordings.csv which uploaded in 2024 and file timestamp in 2022, I still got the error. The file works ok in the project

My test project: https://staging.arbimon.org/p/create-18-apr-2024/settings Recording id: 7827271 Site id: 11350

[LucyDimitrova]

@koonchaya, the latest changes (adding classification results to the backup and splitting bigger files into smaller ones, with up to 200k rows) are now in staging.

[koonchaya]

Url from recordings and templates are working but url from pattern_matching_rois doesn't work.

[koonchaya]

Export files

pattern_matching_rois

pattern_matchings

playlist_recordings_001

playlists

recording_validations

recordings

rfm_classifications_001

rfm_models

sites

soundscapes

species

templates

[koonchaya]

There is no data in some of the export files. I am not sure if that's just from my project or something isn't working. Project: BCI-Panama_2018

• url inside the soundscape.csv.
• some of the source_project in templates.csv don't show data
• score and validated are not showing in pattern_matching_rois.csv
• datetime_utc in recordings.csv is mostly empty (not sure what is suppose to be in this column, timestamps?)

Suggestions Export data

• I find it difficult for me to go back to the project in Arbimon by seeing the site id from my export data. There is nowhere I can identify the site id with the site name. It would be good to have site id and site name for the reference. For example, I can find the playlist name with the playlist id from playlist export.
• I am not sure of how to find the original recordings from the recording id. If there is the filename and timestamps come along with the recordings would be helpful to go back to the original recordings in Arbimon.
• What are the songtype id numbers stand for?

[LucyDimitrova]

I've checked the missing data in the export and found the following:

• soundscapes.csv - the uri was missing from the sql query that gets the data from the database, so it wasn't populated in the csv file as well. I've pushed a quick fix for that.
• the other three files (templates.csv, pattern_matching_rois.csv and recordings.csv) - the data in the files seems to match what we have in the database for this project. Most of the fields are indeed NULL in the database, so they're empty in the csv files as well.

[koonchaya]

@LucyDimitrova Can you add another column to add songtype name in "rfm_classifications_001"? I am going to double check another round of the data in all export files. Summary Backup export: I double check export files in another project. These are things I think we need to do

• rfm_classification.csv needs to add songtype name in another column ✅
• playlists.csv doesn't have metadata
• recordings.csv datetime_utc is showing date but still no timestamps anywhere in the sheet

[naluinui]

FYI team, this feature is now available on production (for internal users who logged in with RFCx email) in the recent release on https://arbimon.org/.

Hope this help us testing it with real data too, before we release to the rest of the users.

[carlybatist]

FYI team, this feature is now available on production (for internal users who logged in with RFCx email) in the recent release on https://arbimon.org/.

Hope this help we testing it with real data too, before we release to the rest of the users.

@koonchaya @naluinui I've requested backup of an Arbimon workshop project on production to test how it goes and see the files, once it's done I'll add the files here for everyone to see and suggest any changes to columns/etc.

[naluinui]

Great! Thanks @carlybatist

[LucyDimitrova]

• rfm_classification.csv needs to add songtype name in another column ✅ - fixed on dev/staging
• playlists.csv doesn't have metadata - no metadata in database as well
• recordings.csv datetime_utc is showing date but still no timestamps anywhere in the sheet ✅ - fixed on dev/staging

[koonchaya]

Remark the downloaded file types from url in the following sheet

• recordings.csv - .flac
• soundscapes.csv - .png
• templates.csv - .png

[koonchaya]

Feedback from @carlybatist https://rfcx.slack.com/archives/C06PD7U65NF/p1716213253364409?thread_ts=1715951951.248849&cid=C06PD7U65NF

here’s the project backup request for that project some suggested changes (mostly changing internal IDs to what they actually show/mean in arbimon):

• pattern_matching_rois.csv - this should be like the regular PM exports, so there should be columns for the PM job name, species name, songtype name, file that the roi is from
• pattern_matchings.csv - add columns for playlist name and template name
• playlist_recordings.csv - add columns for playlist name and recording/file name
• playlists.csv - what do the the ‘playlist_type_id’ and ‘metadata’ columns mean?
• recording_validations.csv - add columns for recording name, species name, songtype name
  • also, what is the different between ‘present’ and ‘present_review’ columns?
• soundscapes.csv - add column for playlist name
• templates.csv - add columns for species name, songtype name, recording/file name
  • also, change column ‘name’ to ‘template_name’

[naluinui]

Related task: Publish python script to download all files from backup csv

[LucyDimitrova]

The strikethrough items are done ✅:

• pattern_matching_rois.csv - this should be like the regular PM exports, so there should be columns for the PM job name, species name, songtype name, file that the roi is from
• pattern_matchings.csv - add columns for playlist name and template name
• playlist_recordings.csv - add columns for playlist name and recording/file name
• playlists.csv - what do the the ‘playlist_type_id’ and ‘metadata’ columns mean?
• recording_validations.csv - add columns for recording name, species name, songtype name
  • also, what is the different between ‘present’ and ‘present_review’ columns?
• soundscapes.csv - add column for playlist name
• templates.csv - add columns for species name, songtype name, recording/file name
  • also, change column ‘name’ to ‘template_name’

@antonyharfield, I'm not sure how to proceed with/answer these:

• pattern_matching_rois.csv - how to get the file that the roi is from?
• playlist_recordings.csv - recording/file name - I can't find a column in the db that stores this. Maybe meta? Or should this be derived from the uri column?
• playlists.csv - what do the the ‘playlist_type_id’ and ‘metadata’ columns mean?
• recording_validations.csv - what is the difference between ‘present’ and ‘present_review’ columns?
• templates.csv - how do we get recording/file name?

[koonchaya]

@LucyDimitrova @carlybatist I think this might help answering the questions

pattern_matching_rois.csv - how to get the file that the roi is from?

I think user can go to the recordings by using recording id and get the url for the recording from recordings.csv. I think adding the filename in recordings.csv can be easy and useful for user to find the original file.

playlist_recordings.csv - recording/file name

Should we include filename in recordings.csv?

recording_validations.csv - what is the difference between ‘present’ and ‘present_review’ columns?

I believe this is the validation as present where you validate from PM results and visualizer. I am not sure which one is from which page.

templates.csv - how do we get recording/file name?

User can check the original recordings in recordings.csv with recording id.

[carlybatist]

Ok yeah if it's a hassle to include the recording/file name, we can just stick to the recording ID in the templates, playlist, etc. files. But for the recording_validations.csv, we should make those two column headers more clear to identify which each represent

[koonchaya]

@LucyDimitrova The export is missing pattern_matchings.csv

[LucyDimitrova]

@koonchaya Fixed. Thanks for catching that!

[grindarius]

Released in v1.2.2 🚀

[carlybatist]

Shouldn't we keep it open until we finalize the the python script and documentation? @koonchaya

[koonchaya]

For the project backup, all requirements in this ticket are done. We still have the ticket for python scripts and documentation open. https://github.com/rfcx/arbimon/issues/1943

[naluinui]

Please note we still need to turn the flag to enable this to the external users. (currently only available to RFCx users)

https://github.com/rfcx/arbimon/blob/854e840c1e63efb94961ce229cadb5564626a192/apps/website/.env.production#L17-L18