ShawHahnLab / umbra

Python package and executable for Linux for managing Illumina sequencing runs
GNU Affero General Public License v3.0
3 stars 0 forks source link

Add chunked approach to box uploader #69

Closed ressy closed 4 years ago

ressy commented 4 years ago

Add an alternate upload approach for large files, modeled after boxsdk's Folder.get_chunked_uploader. Fixes #68. After updating my local install of the Python boxsdk package I had to make a few adjustments to match the newest version (folder IDs stored as strings instead of ints, handling another exception type in the case of auth failure, and casting folder listing output as list objects) but the basic method is to toggle between two upload interfaces depending on file size. The new test is fairly intensive so it only runs when supplied with real Box credentials, but running it with my own test folder and uploading a 100 MB dummy file worked.

This should hopefully alleviate our occasional timeouts when uploading large (multiple GB) bundles of data, but we'll just have to watch and see after updating the installed version.