Support for S3/Ceph direct uploads.

BOINC / boinc

Open-source software for volunteer computing and grid computing.

https://boinc.berkeley.edu

GNU Lesser General Public License v3.0

1.96k stars 439 forks source link

Support for S3/Ceph direct uploads. #2765

Open lfield opened 5 years ago

lfield commented 5 years ago

For some projects data management could be improved by uploading files directly into an S3/Ceph storage system close to the volunteer. This can be done with a presigned URL after authentication. The data server protocol could be updated to send back to the client a direct URL where to put the file. More details can be found in the following paper.

Contact point: @AenBleidd

adamradocz commented 5 years ago

The paper's link is not correct.

lfield commented 5 years ago

Should work now

TheAspens commented 5 years ago

Oh - I like this presigned URL mechanism - I like this idea since I think that shifting to using object storage will be an important thing in the future. In fact it would be great if everything from create_work, downloads, uploads, validators and assimilators could use the object storage. This would greatly simplify project storage.