search

berman-lab / ymap

YMAP - Yeast Mapping Analysis Pipeline : An online pipeline for the analysis of yeast genomic datasets.
MIT License
6 stars 6 forks source link

Implement resumable uploads via HTML5 file API #14

Open vladimirg opened 9 years ago

vladimirg commented 9 years ago

A nice-looking framework to use: http://www.resumablejs.com

darrenabbey commented 9 years ago

An alternate approach that came up in conversation with Chad and Judy when she was in town is to integrate Ymap with cloud storage solutions like DropBox/OneDrive/GoogleDrive/Box/etc. The user would provide a link to the file on one of those systems and then the server would negotiate the transfer in the background.

I can easily see how to set this up for transferring large files to the Ymap drop box, but I haven't looked at calling a system script after transfer completion as the HTML-upload user interface element does.

darrenabbey commented 9 years ago

This method would reduce the delays experienced by the user during project setup, but would require more back-end work to integrate with the different services.

For DropBox (https://www.dropbox.com/help/167) a user can copy a url to the shared file, but the shared url may refer to a simplified DropBox interface rather than the raw file location.

vladimirg commented 9 years ago

I considered this, but I don't know how many people use those services to store their sequences. If they don't already use such services, setting one up can be a hassle, especially due to memory limitations.

darrenabbey commented 9 years ago

The lab directors (names withheld) thought that many/most researchers would already be using some such services. Further discussion on your end may definitely be warranted.

vladimirg commented 9 years ago

We could make a scientific survey, have the results plotted by country, compute the probability of a researcher from a given country using Ymap... or we could just implement resumable uploads :) I imagine this to be a drop-in replacement for the current upload scheme, though that's just a preliminary opinion and remains to be investigated.

vladimirg commented 9 years ago

@darrenabbey, what upload framework(s?) are in use in Ymap currently? I've seen jQuery uploader and HTML5Uploader, they seem to be mutually exclusive. But jQuery, at least, claims to support resumable uploads.

UPDATE: I see now that HTML5Uploader is using jQuery. I found its updated code is here: https://github.com/filad/html5Uploader . Any reason to use that in particular?

darrenabbey commented 9 years ago

The main reason it is in use is that it was the first system I was able to get functional. This isn't a reason to keep using it, however, if a better/simpler/faster alternative is available.

Once concern I have is one aspect of the current server behavior. When the upload completes (or cuts off), it fires off a conclusion script which initiates the server processing of the pipeline. Any resumable upload framework would have to be able to recognize the difference between a correct and an errant conclusion of the file upload.

vladimirg commented 9 years ago

Fixed in afb947b.

vladimirg commented 7 years ago

Reopening because the current resumable upload component has all sorts of weird bugs. Changing it to something standard (most recent JQuery or the one suggested in the first comment of this issue) should fix most if not all of them, and be compatible across all browsers.