search

MIT-LCP / physionet-build

The new PhysioNet platform.
https://physionet.org/
BSD 3-Clause "New" or "Revised" License
55 stars 19 forks source link

Allow published_project.gcp.sent_files to be toggled (True/False) via the admin console #2225

Open tompollard opened 2 months ago

tompollard commented 2 months ago

For large files, we often have to manually transfer the data to GCP using the approach outlined below. Either we:

  1. improve the process for transferring files to the cloud, so that it doesn't fail for large datasets or
  2. add a form field to the published-projects page in the console that allows the published_project.gcp.sent_files flag to be toggled.

Add data to Google Cloud bucket

First go to the project management page (e.g. https://physionet.org/console/published-projects//kinecal/1.0.3/) and click “Create bucket” to create a bucket on Google Cloud.

Now the bucket has been created, we can upload the files directly from the PhysioNet server using gsutil:

gsutil -m cp -r ./* gs://[kinecal-1.0.3.physionet.org](http://kinecal-1.0.3.physionet.org/)

The files are displayed if {% project.gcp and project.gcp.sent_files %} == True, so now we need to:

# Set the sent files flag to True
p = PublishedProject.objects.get(slug="kinecal", version="1.0.3")
p.gcp.sent_files = True
p.gcp.save()
tompollard commented 2 months ago

https://github.com/MIT-LCP/physionet-build/issues/1903 is related.