search

FineUploader / fine-uploader

Multiple file upload plugin with image previews, drag and drop, progress bars. S3 and Azure support, image scaling, form support, chunking, resume, pause, and tons of other features.
https://fineuploader.com
MIT License
8.18k stars 1.87k forks source link

tus support #1620

Open clickbait opened 8 years ago

clickbait commented 8 years ago

Type of issue

feature request

Uploader type

traditional

Feature request details

Detailed description of the feature

I'm considering using this library for a project of my own, but one of the things I need is support for the tus protocol.

I feel this feature would be beneficial to more than just myself.

rnicholus commented 8 years ago

Why is this protocol needed? The resume feature in Fine Uploader has been in place and working well for years.

If such a change were to be made, it would have to be non-breaking so exiting users would not have to change any server code. And this would only apply to traditional end points since Azure and S3 have their own protocols.

clickbait commented 8 years ago

I wouldn't say it's needed. It's just useful in that it's a protocol specifically made for the purpose of resumable uploads and I feel it is more efficient because of that.

If such a change were to be made, it would have to be non-breaking so exiting users would not have to change any server code.

I think if it were an option to use the tus protocol (default being false), it'd be non-breaking.

rnicholus commented 8 years ago

I'd be interested to hear how it is more efficient than the present method in place. Either way, this sort of change would be very low priority at the moment. I'm currently focusing most of my free time developing react-fine-uploader and making any needed related changes to fine-uploader. But I'm always willing to give feedback and direction for anyone willing to follow through with a serious pull request for a reasonable feature.

openscript commented 8 years ago

It's a pity that FineUploader defined it's own specific protocol for chunking and resuming, instead of using TUS. For TUS many end-points are already implemented and ready to go, whereas for the library-specific FineUploader protocol it's not.

rnicholus commented 8 years ago

It's a pity that FineUploader defined it's own specific protocol for chunking and resuming, instead of using TUS.

There was no such protocol when I implemented chunking in Fine Uploader. Fine Uploader was first in this regard.

For TUS many end-points are already implemented and ready to go, whereas for the library-specific FineUploader protocol it's not.

That's not true. There is library code in PHP, Java, Node.js, etc for handling Fine Uploader chunked upload requests. It's maintained in this very GitHub organization.

openscript commented 8 years ago

Okay, I understand, if there was nothing like that back then. I'm sorry.

I didn't find any libraries. All I found is some examples in here: https://github.com/FineUploader/server-examples

Where can I find the libraries?

openscript commented 8 years ago

..but back to the topic. Where would you start implementing TUS in FineUploader? It would be great to have some bullet points to start with. :-)

rnicholus commented 8 years ago

If you're using PHP, I've published the PHP traditional server to Composer. There is also a Golang library that supports chunking in this GitHub org at https://github.com/FineUploader/fineuploader-go-server, contributed and maintained by @orlandovald. It's available via go get. Then there are Java and Node.js implementations that support chunking in the server-examples repo as you have indicated. It would be great to get those pushed up to Maven Central and npm, respectively, but some refactoring would be needed to make them more configurable.

Where would you start implementing TUS in FineUploader

TUS support would have to be opt-in, since the current proprietary protocol has been in place since late 2012 and I want to avoid breaking all current projects out there that depend on Fine Uploader chunking. So, the following high-level changes/additions would be required, as I see it:

All in all, it seems like a big job.

openscript commented 8 years ago

Thank you for the detailed answer.

The project I need that for is running on Rails.

I've been scrolling through some code of FineUploader and also the TUS JS client implementation. I'm still estimating, which solution provides the desired functionality (chunking, resuming and client side image resize) and is realisable within an acceptable time, but it's hard to get an overview over FineUploader in that short time.

Right now it feels like taking the TUS JS client and a library to manipulate images (maybe cropper, but this adds jQuery as a dependency, which I don't like that much), is the best option. So I don't have one big thing, more a two or three quite tight libraries. It feels like this is easier to manage and handle.

rnicholus commented 8 years ago

Implementing TUS would likely be a good amount of work, and I'm pretty busy ATM supporting Fine Uploader, updating the several-years-old build process (#1569) and developing a React wrapper for the library.

Right now it feels like taking the TUS JS client and a library to manipulate images (maybe cropper, but this adds jQuery as a dependency, which I don't like that much), is the best option.

This is certainly an option, but this means you'll have to implement image scaling, file (and possibly folder) dropping, <input type="file"> support + styling, and any other feature Fine Uploader provides, yourself. But I'm not sure of your specific requirements.

I commented in the Ruby library you are referring to. It's quite simple to handle Fine Uploader's chunking protocol. It may be easiest to just write a plug-in for that library to support Fine Uploader chunking. Additional benefit could come out of that effort, such as a generic Ruby backend example for FU.

openscript commented 8 years ago

I've started working on the integration of TUS into FineUploader. I've decided for the following approach:

  1. Getting FineUpload build and tests up and running
  2. Hacking something quick and dirty together, which works against the official TUS server implementation. I replaced stuff in the traditional handler.
  3. Starting over from the traditional handler with a documented, tested and sophisticated implementation of TUS into FineUploader.
  4. Add a new module for TUS and alter the MakeFile. That's where I'm now.

The next step is going to be to use the code from the quick and dirty hack and more from the TUS JS client to make it working in the clean implementation.

I've started a document about the integration in here. My fork of the clean implementation can be found here.

Discoveries so far are:

I'll working again on the integration this Friday.

rnicholus commented 8 years ago

The support for old browser (especially the ones without the File API), blow up the FineUploader. The first implementation is going to be without taking care of support for old browsers.

I'm not sure what you mean here. Fine Uploader support for older browsers works as expected, and I'm not aware of any issues. What issues are you seeing?

There are no integration tests.

I suppose it depends how you define integration tests. In one sense, there are a lot of integration tests, but there are no webdriver tests, that is true. I never got around to writing these, and that would take an immense amount of time to even test the most trivial features.

Concurrent chunk uploading is not going to be supported by the TUS implementation

That's unfortunate. Concurrent chunking was a highly requests Fine Uploader feature, and it has the potential to speed up single large file uploads significantly. I wonder if the TUS spec can be updated to account for this.

One other thing - in your fork, it looks like you've reformatted almost all of the files in the repository. I'd ask that you not do this and focus specifically on the feature at hand. I suspect this will already be a very time consuming feature to write, test, and document, and I don't want the scope to get out of control.

rnicholus commented 8 years ago

As an aside, I'd love to get proper selenium/WD tests started, but that's for another case. In the meantime, I've attempted to make the existing "unit" tests as comprehensive as possible over the years. Either way, manual testing will always be required, as the number of workflows supported by Fine Uploader is substantial, and I simply don't yet have tests that cover all of those in particular.

openscript commented 8 years ago

I'm not sure what you mean here. Fine Uploader support for older browsers works as expected, and I'm not aware of any issues. What issues are you seeing?

This is not an issue! I just don't guarantee that it's going to work for all browser from the beginning.

I suppose it depends how you define integration tests. In one sense, there are a lot of integration tests, but there are no webdriver tests, that is true. I never got around to writing these, and that would take an immense amount of time to even test the most trivial features.

Yes, 'click'-tests consuming a lot of time.

That's unfortunate. Concurrent chunking was a highly requests Fine Uploader feature, and it has the potential to speed up single large file uploads significantly. I wonder if the TUS spec can be updated to account for this.

I absolutely support this, but one step after another.

One other thing - in your fork, it looks like you've reformatted almost all of the files in the repository. I'd ask that you not do this and focus specifically on the feature at hand. I suspect this will already be a very time consuming feature to write, test, and document, and I don't want the scope to get out of control.

This is already undone. I didn't want to do that. This was part of my quick and dirty implementation.

rnicholus commented 8 years ago

I just don't guarantee that it's going to work for all browser from the beginning.

Yes, chunking is simply not possible w/out the File API. I assume the feature you create will be tied to some new option or options. By default, any option that elects to use TUS support should be ignored if the browser doesn't support chunking, and Fine Uploader should fall back to the non-File API code that uploads files using form submits inside of hidden iframes.

The features.js module can be used to determine the capabilities of the current browser.

All in all, sounds great. Feel free to loop me in, ask questions, or ask for code reviews and help at any time.

rnicholus commented 8 years ago

BTW, it's probably good that you didn't get too far updating the build-related code to handle TUS support, as I literally ripped out all of the old build-related code and replaced it with a single Makefile a couple days ago.

janko commented 8 years ago

Concurrent chunk uploading is not going to be supported by the TUS implementation, because as far as I see the TUS protocol has no concurrent support.

After some digging, I found out about the Concatenation extension to the TUS protocol, which enables parallel chunk uploads!

Rubytus unfortunately doesn't support it yet, but hopefully it shouldn't be too difficult to add it. In any case it would be great if the FineUploader implementation can somehow autodetect whether the server supports Concatenation (since it's an optional extension), and it if it does use parallel chunk uploads, otherwise use sequential chunk uploads.