yadudoc / cloud_kotta

Enabling Secure and Scalable Data Analytics in the Cloud
https://turingcompute.net
Apache License 2.0
16 stars 10 forks source link

Add Multipart Upload / Globus #23

Open yadudoc opened 7 years ago

yadudoc commented 7 years ago

The upload system currently does not support multipart upload. This limits the upload size to under 5Gb.

Regardless of globus support, it makes sense to have multipart upload support for following reasons :

  1. Upload from browser is simpler
  2. Does not require installing and configuring a system unfamiliar to users.
  3. Source system might only have http/https port open.

Adding this feature would also require a check on filesize, since multipart only works for files with sizes > 5MB and < 5TB. Beyond 20Gb, it makes sense to shift to a reliable data transfer service such as globus.