search

shrinerb / shrine

File Attachment toolkit for Ruby applications
https://shrinerb.com
MIT License
3.18k stars 275 forks source link

Status of chunking and resuming uploads #72

Closed openscript closed 8 years ago

openscript commented 8 years ago

I've been searching through the documentation, but I didn't find anything about chunking and resuming of uploads. I'm asking, because I plan to use shrine with the fineuploader. There is a pretty nice documentation, what the backend should be capable of in here: http://docs.fineuploader.com/endpoint_handlers/traditional.html

So my question is, what should I do with UUID from the fineuploader? Does Shrine support chunking? How about resuming of uploads?

Thank you in advance for your answer :-)

janko commented 8 years ago

Thank you for asking this, now I will not close this issue like Paperclip did, until we reach a satisfying result πŸ˜ƒ

You're right, Shrine at the moment doesn't support chunked or resumable uploads. This is something that I thought about in the past, however, my conclusion was it makes much more sense to direct upload to an external service which does support these things.

Amazon S3 supports multipart uploads, and with that resumable uploads as well. From what I can see, FineUploader has support for uploading files directly to S3 and, unlike jQuery-File-Upload, it can do so in chunks by utilizing S3's multipart API. You can see Direct Uploads to S3 on how to setup direct S3 uploads with Shrine, along with some client-side tips. Would that be feasible for you?

In general I'm not closed to the idea of implementing chunked/resumable uploads in Shrine, but in that case instead of implementing a library-specific protocol like FineUploader imposes, it would be much better to implement TUS. Fortunately, there is already rubytus gem which gives you an app that implements the TUS protocol, storing files on the filesystem, and the author says that his company has been using it in production since 2013 (https://github.com/tus/tus.io/issues/28#issuecomment-149789948).

Let me know if any of these options would work for you.

openscript commented 8 years ago

Wow! Such an immediate response. Thank you :-)

Using S3 or Azure is not an option for this project, because of the data protection law, the project needs to be hosted in a specific country.

As far as I understand chunked uploads are specified in HTTP 1.1, whereas resumable uploads are not specified. Unfortunately FineUploader seems to use a library-specific protocol for chunked and resumable uploads.

Probably I need chunking, because of very big files. Then resuming is also super useful. I don't know yet, what I going to do.

janko commented 8 years ago

I completely understand if you cannot use S3 or Azure; the goal of Shrine is to work for everyone, so we'll figure something out πŸ˜ƒ

There is something called "chunked transfer encoding" which is part of HTTP 1.1, however, while it does allow sending data in chunks, I'm almost 100% sure that it doesn't let you specify the order. This means that you cannot upload chunks in parallel, or retry a chunk.

So, to give you more insight about RubyTus, it's a Rack application that accepts file uploads with the TUS protocol, and the end result is a file on the filesystem. That means that you can define Shrine's :cache directory to be the one where RubyTus saves the files, and then on client-side you can just add the location into Shrine's JSON representation of an attachment, the same way you do with direct S3 uploads. So if we can upload files to RubyTus, it should be good, right?

I think FineUploader wouldn't work in that case, as its protocol is pretty much sealed. Maybe there are apps which implement uploads with this protocol, just like RubyTus does for TUS. But instead of FineUploader you could just use a JS file upload library which supports TUS, probably best to go with the official tus-js-client.

Do you think that could work for you?

openscript commented 8 years ago

Sorry for not responding lately. The weather was great and my motorbike smiled at me :-)

I really like the idea to use the well specified TUS protocol and I would love to go that way, if it doesn't consume too much time. I've been doing some research and I didn't find a popular uploading framework, which supports TUS, even though many of them have requests for TUS in their issue trackers (Plupload, FineUploader)

So I see two possibilities, which could work:

  1. We could work together on that. We implement the support for TUS in Shrine and we implement in one of the JS uploading frameworks support for TUS. I think this would be a benefit for Shrine and makes it more popular, but I think it's more time consuming than the other possibility below. As I see we live in the same timezone and I've some time for that on the upcoming weekends. It would be fun for me doing something like that. The TUS protocol is already implemented in JavaScript and it doesn't have dependencies. I think it would be quite easy to implement it into FineUploader.
  2. I implement the FineUploader protocol as a plugin in Shrine. I think it's not impossible and I estimate 20 hours of time for doing that. I'm so into FineUploader, because it's already in use in another project.

I've implemented the uploading today in my project with Shrine and FineUploader, but of course without chunking and resuming.

What do you think of all that?

rnicholus commented 8 years ago

Hey there, Fine Uploader developer here. The case where TUS support has been discussed in Fine Uploader can be found at FineUploader/fine-uploader#1620. In there, you'll also see historical information that explains why FU did not use TUS (TL;DR - chunking in FU came before TUS).

While it would be great to add optional TUS support to FU, I think the easier option, in the context of this case, is to add support for Fine Uploader chunked requests to shrine. Then again, I'm not familiar with shrine, so I could be way off-base with that assertion. Either way, I'm available to provide support and advice on the Fine Uploader side of things.

janko commented 8 years ago

@openscript If we decide to go for the 1st option, there is actually nothing that needs to be done on the Ruby side, because as I mentioned there is already a production-ready implementation of TUS, Rubytus. I think there would we very little benefit in writing a new TUS implementation as a Shrine plugin; you could just do direct uploads like to any other service. IMO it's much better to use an existing well-tested solution, and it can easily be hooked up with Shrine.

If we go with the 2nd option (@rnicholus thank you very much for your input), I think it would be also great to implement a generic server (like Rubytus), something like "fineupload-ruby-server". It would just be a Rack application (I personally prefer Roda, but Rubytus chose Goliath), which could then be either mounted in your router or run separately. I think it would cool to have that, as FineUploader looks like a really advanced JS file upload library.

Let me know if you would like to take a stab at it. Otherwise I could try taking a look into it the next week. Or we could work on it together; it's been so long since I last pair programmed πŸ˜ƒ

Since this is technically not related to Shrine, I will close this issue. But feel free to continue the discussion here, or we can also switch to email.

rnicholus commented 8 years ago

My Ruby skills are pretty non-existent rusty. But if any FU-related questions come up, please do let me know. If this is a generic ruby server, perhaps this can be developed as a repo in the Fine Uploader org.

janko commented 8 years ago

@rnicholus Thanks, if we decide to build a ruby server for FineUploader, I also think that the "FineUploader" organization would be a great place for it.

I think in the long run FineUploader would greatly benefit from having support for TUS, because it's a standalone and fully specified protocol, and really many people collaborated to bring it to 1.0, with contributions from companies like Vimeo. I think it's great to agree on a generic protocol and make servers for it, and then have different JS libraries which support that same protocol.

@openscript However, FineUploader's protocol is also nicely specified, and many server implementations exist as well. And since FineUploader seems to have significant advantages over libraries like tus-js-client, it might be worthwhile to implement a Ruby server for it. Even though that also sounds like a very decent amount of work, for me personally it would be easier, because my Ruby skills are much hotter than my JavaScript skills.

But, there is actually a 3rd option that comes to mind. What if we created a Rack middleware that sits in front of Rubytus, and translates the FineUploader protocol into TUS? That way we get to leverage Rubytus, as I think it already solves a lot of problems which are protocol-agnostic, without having to change anything from FineUploader's side.

@openscript From these three options I'm now most inclined to start with the 3rd, as that seems like it would be the least amount of work, and we get to reuse existing generic components which is awesome. This way our "fineuploader server" would automatically benefit from any improvements that land on Rubytus. What do you think?

janko commented 8 years ago

Actually, I'm not so sure that the 3rd option would require less work than others, because it requires understanding FineUploader protocol, TUS protocol, and Rubytus itself. But I think it would be the more useful than implementing a standalone FineUploader server.

It's also worth noting that Rubytus doesn't seem to fully support TUS 1.0 yet (https://github.com/picocandy/rubytus/issues/2), so it might not automatically work with tus-js-client at the moment.

There is also another option, I think, and that is to use an existing FineUploader server written in another language. fineuploader-go-server looks really good and would probably be the best bet, because from what I understood you don't need to have Go installed on the server to run a Go program. It might sound strange to use an app written in another language, but from the client side you don't actually care in what language an app is written in, you just care that it accepts requests and returns responses. And you shouldn't need to modify anything, just run it on some port like any other app.

To summarize, I think we have five options here:

  1. Instead of FineUploader use tus-js-client with Rubytus
  2. Add TUS support to FineUploader and use Rubytus (probably most useful in the long run)
  3. Implement a standalone FineUploader-compliant Ruby server
  4. Implement a Rack middleware for Rubytus which translates FineUploader requests into TUS and TUS responses into FineUploader (although I don't know if this is even possible)
  5. Use an existing FineUploader-compliant server like fineuploader-go-server

@openscript Since I probably won't have time to help with 3rd or 4th option, it's up to you to choose whatever you think it's best for you.

openscript commented 8 years ago

I read through the whole issue again and tried to come to a conclusion, where to go from here. I've been working on several Rails projects since 2011 and handling uploaded files was always a pain. In my opinion there was never a rock solid and flexible solution for doing that. I've tried many combinations of Paperclip, Carrierwave, PluploadJS, jQuery File Upload or FineUploader. There was always something I wasn't happy about or something which needed some wizardry and tinkering. It would be great to have a strong, flexible, foolproof solution for uploading files to Ruby apps.

During today I made for all options a little diagram, so it's easier to get an overview. I wrote down the pros and cons as well:

Option 1 – FineUploader Ruby server

Option1

Pros:

Cons:

Option 2 – Adding TUS to FineUploader

Option2

Pros:

Cons:

Option 3 – FineUploader-to-TUS middleware

Option3

Pros:

Cons:

Option 4 – TUS JS library

Option4

Pros:

Cons:

Option 5 – FineUploader Go server

Option5

Pros:

Cons:

openscript commented 8 years ago

I want to go for option 2. I've already forked FineUploader and started integrating TUS. I've set up and got the TUS server written in go runningm so Ihave something to run FineUploader with TUS against. I think with some support from @rnicholus this is getting on track soon!

It would be great to have a sample project with Shrine and RubyTUS. I haven't been digging to deep into RubyTUS, but I've seen too, that the last commit is a year ago and TUS 1.0 is not complete. Probably we need to get RubyTUS to TUS 1.0, but I think that's also not so hard. @janko-m do you think you have the time to set that example project up? :-)

What do you think of all that? Do you think it's possible to get the option 2 implemented until the 21. August?

rnicholus commented 8 years ago

@openscript You've spent quite a bit of time outlining all of the options, very impressive. Since the change here will be to Fine Uploader (option 2), can you outline your proposed changes to Fine Uploader in FineUploader/fine-uploader#1620? Perhaps I can offer some advice or locate a potential issue before you get too deep into implementing your solution.

The chunking logic in Fine Uploader is quite complex, mostly due to the concurrent chunking feature I added a couple years ago. This feature allows bandwidth to be maximized when uploading a single file. Essentially, it breaks the file up into multiple chunks and sends as many of the chunks as possible (while respecting the maxConnections option) at once. This was quite difficult to implement, and I realized why other libraries punted on this feature in the past, but the speed gains are notable.

janko commented 8 years ago

@openscript Wow, so impressed with these diagrams and the pros & cons sections. I edited your comment to just note what each option relates to (I hope you don't mind). It's great that you're going for adding TUS support to FineUploader. @rnicholus It's awesome that you've implemented concurrent chunking feature into FineUploader, hopefully that will work the same way with TUS.

I will gladly set up an example project that integrates Shrine and Rubytus. I will first use tus-js-client and check whether Rubytus works with it, and try to make necessary PRs if it doesn't. And later once you manage to get TUS into FineUploader, we'll switch tus-js-client with FineUploader.

I would also like to get an opinion from @MarkMurphy, as I was reading his discussion around resumable uploads on Paperclip, and he seemed very knowledgeable about the topic. Mark, if you're not too busy, how did you end up implementing resumable uploads? Did you manage to integrate Rubytus with your Ruby application? If yes, what JS library did you use with it?

MarkMurphy commented 8 years ago

@janko-m Thanks, and yes I did help write some of the TUS protocol. I built a custom solution for a Ruby on Rails api. Some of it was inspired by code from Rubytus. I set it up so that it would easily couple with Paperclip. The api is currently only consumed by an iOS application so I can't speak for any client side libraries.

I don't mind sharing some of my api code.

Here's the gist:

https://gist.github.com/MarkMurphy/76dae9307cb67d56951e13a63df99b19

janko commented 8 years ago

@MarkMurphy Thank you very much for sharing this!

openscript commented 8 years ago

I've added a comment about the progress with the TUS implementation in FineUploader: https://github.com/FineUploader/fine-uploader/issues/1620#issuecomment-238994943

janko commented 8 years ago

@openscript Just to report the status of the Shrine TUS demo.

So, Rubytus at the moment doesn't work with tus-js-client, I'm trying to fix it now. It is failing on the CORS preflight request, and I noticed that Rubytus was using old headers, so I tried copying over CORS headers from tusd, but for some reason it's still failing, I need to figure out why.

In the meanwhile I tried using tusd (a Go server), since it's the official one, and it's mature and up-to-date. And I managed to get things working. To really test resumability I tried using S3 backend for tusd (so that it's really uploading remotely), and it worked as well. However, the resumability didn't work, because tus-js-client only seems the remember the file once it's fully uploaded (it makes a request to tusd with the wrong ID when trying to resume). That's probably just a bug in tus-js-client, but that is irrelevant if we're going to use FineUploader.

So, once the TUS server finishes uploading, it returns the URL to the file, and I use shrine-url to store that URL as a cached file, which can then be promoted to any Shrine storage. So, the only part that's not working properly is uploading itself, which is the part that's unrelated to Shrine.

janko commented 8 years ago

@openscript So, I managed to get Rubytus working with tus-js-client, and now it works locally both creating and resuming (although there might be an edge case that I've missed). I made the following changes:

**Rubytus changes** ``` diff commit c8c83736994124db3c337e22b4195591c15de64a Author: Janko Marohnić Date: Sat Aug 13 02:38:06 2016 +0800 Get things working diff --git a/.ruby-version b/.ruby-version deleted file mode 100644 index 68b3a4c..0000000 --- a/.ruby-version +++ /dev/null @@ -1 +0,0 @@ -1.9.3-p551 diff --git a/lib/rubytus/constants.rb b/lib/rubytus/constants.rb index d93af20..49f63cd 100644 --- a/lib/rubytus/constants.rb +++ b/lib/rubytus/constants.rb @@ -16,7 +16,7 @@ module Rubytus DEFAULT_MAX_SIZE = 1073741824 SUPPORTED_VERSIONS = ['1.0.0'] - SUPPORTED_EXTENSIONS = ['file-creation'] + SUPPORTED_EXTENSIONS = ['creation'] STATUS_OK = 200 STATUS_CREATED = 201 diff --git a/lib/rubytus/helpers.rb b/lib/rubytus/helpers.rb index 21dc232..9b33ffa 100644 --- a/lib/rubytus/helpers.rb +++ b/lib/rubytus/helpers.rb @@ -1,6 +1,7 @@ require 'rubytus/constants' require 'rubytus/common' require 'rubytus/error' +require 'base64' module Rubytus module Helpers @@ -12,7 +13,7 @@ module Rubytus env['api.headers'].merge!(handle_cors(request, headers)) - if request.collection? || request.resource? + if (request.collection? || request.resource?) && !(request.options? || request.get?) validates_supported_version(headers["Tus-Resumable"]) end @@ -27,7 +28,7 @@ module Rubytus def validates_offset(req_offset, info_offset) if req_offset > info_offset - error!(STATUS_FORBIDDEN, "Offset: #{req_offset} exceeds current offset: #{info_offset}") + error!(STATUS_FORBIDDEN, "Upload-Offset: #{req_offset} exceeds current offset: #{info_offset}") end end @@ -71,18 +72,26 @@ module Rubytus def parse_metadata(metadata) return if metadata.nil? - arr = metadata.split(' ') + pairs = metadata.split(",").map { |string| string.split(" ") } - if (arr.length % 2 == 1) + if pairs.any? { |pair| pair.size != 2 } error!(STATUS_BAD_REQUEST, "Metadata must be a key-value pair") end - Hash[*arr].inject({}) do |h, (k, v)| + Hash[pairs].inject({}) do |h, (k, v)| h[k] = Base64.decode64(v) h end end + def serialize_metadata(metadata) + encoded_list = metadata.map do |key, value| + "#{key} #{Base64.encode64(value)}" + end + + encoded_list.join(",") + end + def handle_cors(request, headers) origin = headers['Origin'] @@ -92,11 +101,11 @@ module Rubytus cors_headers['Access-Control-Allow-Origin'] = origin if request.options? - cors_headers['Access-Control-Allow-Methods'] = "POST, HEAD, PATCH, OPTIONS" - cors_headers['Access-Control-Allow-Headers'] = "Origin, X-Requested-With, Content-Type, Entity-Length, Offset, TUS-Resumable" + cors_headers['Access-Control-Allow-Methods'] = "POST, GET, HEAD, PATCH, DELETE, OPTIONS" + cors_headers['Access-Control-Allow-Headers'] = "Origin, X-Requested-With, Content-Type, Upload-Length, Upload-Offset, Tus-Resumable, Upload-Metadata" cors_headers['Access-Control-Max-Age'] = "86400" else - cors_headers['Access-Control-Expose-Headers'] = "Offset, Location, Entity-Length, TUS-Version, TUS-Resumable, TUS-Max-Size, TUS-Extension" + cors_headers['Access-Control-Expose-Headers'] = "Upload-Offset, Location, Upload-Length, Tus-Version, Tus-Resumable, Tus-Max-Size, Tus-Extension, Upload-Metadata" end cors_headers @@ -117,7 +126,7 @@ module Rubytus env['api.uid'] = uid env['api.entity_length'] = request.entity_length env['api.resource_url'] = request.resource_url(uid) - env['api.metadata'] = parse_metadata(headers['Metadata']) + env['api.metadata'] = parse_metadata(headers['Upload-Metadata']) end end @@ -134,6 +143,10 @@ module Rubytus env['api.action'] = :patch env['api.buffers'] = '' env['api.offset'] = request.offset + + env['api.headers'].merge!( + 'Upload-Offset' => (request.offset + request.content_length).to_s + ) end if request.get? diff --git a/lib/rubytus/info.rb b/lib/rubytus/info.rb index 7e41e0d..ad3cbd7 100644 --- a/lib/rubytus/info.rb +++ b/lib/rubytus/info.rb @@ -3,25 +3,33 @@ require 'json' module Rubytus class Info < Hash def initialize(args = {}) - self['Offset'] = args[:offset] || 0 - self['EntityLength'] = args[:entity_length] || 0 - self['Meta'] = args[:meta] || nil + self['Upload-Offset'] = args[:offset] || 0 + self['Upload-Length'] = args[:entity_length] || 0 + self['Upload-Metadata'] = args[:meta] || nil end def offset=(value) - self['Offset'] = value.to_i + self['Upload-Offset'] = value.to_i end def offset - self['Offset'] + self['Upload-Offset'] end def entity_length=(value) - self['EntityLength'] = value.to_i + self['Upload-Length'] = value.to_i end def entity_length - self['EntityLength'] + self['Upload-Length'] + end + + def metadata=(value) + self['Upload-Metadata'] = value + end + + def metadata + self['Upload-Metadata'] end def remaining_length diff --git a/lib/rubytus/middlewares/storage_barrier.rb b/lib/rubytus/middlewares/storage_barrier.rb index 786d3c6..7c36537 100644 --- a/lib/rubytus/middlewares/storage_barrier.rb +++ b/lib/rubytus/middlewares/storage_barrier.rb @@ -1,9 +1,12 @@ +require 'rubytus/constants' +require 'rubytus/helpers' require 'rubytus/error' module Rubytus module Middlewares class StorageBarrier include Rubytus::Constants + include Rubytus::Helpers include Goliath::Rack::AsyncMiddleware def post_process(env, status, headers, body) @@ -16,7 +19,7 @@ module Rubytus when :create status = STATUS_CREATED headers['Location'] = env['api.resource_url'] - storage.create_file(env['api.uid'], env['api.entity_length']) + storage.create_file(env['api.uid'], env['api.entity_length'], env['api.metadata']) when :head info = storage.read_info(env['api.uid']) @@ -24,11 +27,28 @@ module Rubytus if info.nil? status = STATUS_NOT_FOUND else - headers['Offset'] = info.offset.to_s + headers['Upload-Offset'] = info.offset.to_s + headers['Upload-Length'] = info.entity_length.to_s + headers['Upload-Metadata'] = serialize_metadata(info.metadata) if info.metadata end when :get - body = storage.read_file(env['api.uid']) + path = storage.file_path(env['api.uid']) + info = storage.read_info(env['api.uid']) + + if metadata = info.metadata + headers['Content-Disposition'] = "attachment; filename=\"#{metadata['filename']}\"" if metadata['filename'] + headers['Content-Type'] = metadata['content_type'] if metadata['content_type'] + end + + file = ::Rack::File.new(File.dirname(path)) + file.path = path + + result = file.serving(env) + + status = result[0] + headers = result[1].merge(headers) + body = result[2] end rescue PermissionError => e raise Goliath::Validation::Error.new(500, e.message) diff --git a/lib/rubytus/request.rb b/lib/rubytus/request.rb index 0b69abe..c8d9f1f 100644 --- a/lib/rubytus/request.rb +++ b/lib/rubytus/request.rb @@ -43,11 +43,11 @@ module Rubytus end def entity_length - fetch_positive_header('HTTP_ENTITY_LENGTH') + fetch_positive_header('HTTP_UPLOAD_LENGTH') end def offset - fetch_positive_header('HTTP_OFFSET') + fetch_positive_header('HTTP_UPLOAD_OFFSET') end def base_path @@ -88,8 +88,8 @@ module Rubytus error!(STATUS_BAD_REQUEST, "#{header_orig} header must not be empty") end - if (header_name == 'HTTP_ENTITY_LENGTH') && (value < 0) - error!(STATUS_BAD_REQUEST, "Invalid Entity-Length: #{value}. It should non-negative integer or string 'streaming'") + if (header_name == 'HTTP_UPLOAD_LENGTH') && (value < 0) + error!(STATUS_BAD_REQUEST, "Invalid Upload-Length: #{value}. It should non-negative integer or string 'streaming'") end value diff --git a/lib/rubytus/storage.rb b/lib/rubytus/storage.rb index c00a3c1..a0fe2f5 100644 --- a/lib/rubytus/storage.rb +++ b/lib/rubytus/storage.rb @@ -48,8 +48,9 @@ module Rubytus def create_file(uid, entity_length, metadata = {}) fpath = file_path(uid) ipath = info_path(uid) - info = Rubytus::Info.new(metadata) + info = Rubytus::Info.new info.entity_length = entity_length + info.metadata = metadata if metadata begin File.open(fpath, 'w') {} @@ -94,7 +95,7 @@ module Rubytus begin data = File.open(ipath, 'r') { |f| f.read } - JSON.parse(data, :object_class => Rubytus::Info) + parse_info(data) rescue SystemCallError => e raise(PermissionError, e.message) if e.class.name.start_with?('Errno::') end @@ -113,5 +114,13 @@ module Rubytus raise(PermissionError, e.message) if e.class.name.start_with?('Errno::') end end + + private + + def parse_info(data) + info = Rubytus::Info.new + info.replace(JSON.parse(data)) + info + end end end diff --git a/test/rubytus/test_command.rb b/test/rubytus/test_command.rb index 368da77..48f668a 100644 --- a/test/rubytus/test_command.rb +++ b/test/rubytus/test_command.rb @@ -204,7 +204,7 @@ class TestRubytusCommand < MiniTest::Test :path => "/uploads/#{uid}", :body => 'abc', :head => protocol_header.merge({ - 'Offset' => '0', + 'Upload-Offset' => '0', 'Entity-Length' => '3', 'Content-Type' => 'plain/text' }) @@ -230,7 +230,7 @@ class TestRubytusCommand < MiniTest::Test :path => "/uploads/#{ruid}", :body => 'abc', :head => protocol_header.merge({ - 'Offset' => '0', + 'Upload-Offset' => '0', 'Entity-Length' => '3', 'Content-Type' => 'application/offset+octet-stream' }) @@ -256,7 +256,7 @@ class TestRubytusCommand < MiniTest::Test :path => "/uploads/#{ruid}", :body => 'abc', :head => protocol_header.merge({ - 'Offset' => '3', + 'Upload-Offset' => '3', 'Entity-Length' => '3', 'Content-Type' => 'application/offset+octet-stream' }) @@ -281,8 +281,8 @@ class TestRubytusCommand < MiniTest::Test :path => "/uploads/#{ruid}", :body => 'abcdef', :head => protocol_header.merge({ - 'Offset' => '0', - 'Entity-Length' => '6', + 'Upload-Offset' => '0', + 'Upload-Length' => '6', 'Content-Type' => 'application/offset+octet-stream' }) } @@ -300,8 +300,8 @@ class TestRubytusCommand < MiniTest::Test :path => "/uploads/#{uid}", :body => 'abc', :head => protocol_header.merge({ - 'Offset' => '0', - 'Entity-Length' => '3', + 'Upload-Offset' => '0', + 'Upload-Length' => '3', 'Content-Type' => 'application/offset+octet-stream' }) } diff --git a/test/rubytus/test_storage.rb b/test/rubytus/test_storage.rb index b36fbee..7fe0584 100644 --- a/test/rubytus/test_storage.rb +++ b/test/rubytus/test_storage.rb @@ -48,12 +48,12 @@ class TestStorage < MiniTest::Test def test_read_info File.open(@storage.info_path(@uid), 'w') do |f| - f.write('{"Offset":100,"EntityLength":500,"Meta":null}') + f.write('{"Upload-Offset":100,"Upload-Length":500,"Meta":null}') end info = @storage.read_info(@uid) assert_kind_of Hash, info - assert_equal 100, info['Offset'] + assert_equal 100, info['Upload-Offset'] end def test_read_info_failure @@ -63,19 +63,19 @@ class TestStorage < MiniTest::Test def test_update_info File.open(@storage.info_path(@uid), 'w') do |f| - f.write('{"Offset":100,"EntityLength":500,"Meta":null}') + f.write('{"Upload-Offset":100,"Upload-Length":500,"Meta":null}') end @storage.update_info(@uid, 250) info = @storage.read_info(@uid) assert_kind_of Hash, info - assert_equal 250, info['Offset'] + assert_equal 250, info['Upload-Offset'] end def test_update_info_failure storage = Rubytus::Storage.new(read_only_options) - stub(storage).read_info(@uid) { Rubytus::Info.new('Offset' => 100) } + stub(storage).read_info(@uid) { Rubytus::Info.new('Upload-Offset' => 100) } assert_raises(Rubytus::PermissionError) { storage.update_info(@uid, 250) } end ```

However, when attempting to make a pull request with organized commits, I realized that there are still some things in the specification that are not part of Rubytus, but I had troubles implementing it because Goliath is a complex web framework πŸ˜ƒ

In Goliath you can return early by raising a validation error, but there currently isn't a way to set headers when raising an error. I made a PR to Goliath, which should get merged soon, but I'm not sure when a new version of Goliath will be released (the last version was released in June 2014), and we would need a version to put it in rubytus.gemspec.

However, I feel like using Goliath is really an overkill here, the asynchronicity makes it much more difficult to work with than e.g. Roda. And I feel that EventMachine only shines when you do slow IO, and here we're just writing to filesystem. Even after that, Rubytus is still missing some TUS extensions (notably concatenation, which I find really important because it enables parallel chunked uploads).

So, now that read the whole TUS 1.0 thoroughly, I decided that it makes much more sense to make my own implementation in Roda. That would make it much easier to maintain, since not many people are familiar with Goliath. And also current Rubytus code is pretty difficult to follow, since it's scattered across middlewares and subclasses. I feel like it would it would take similar amount of time to make Rubytus work fully by the specification than it would take to make a Roda-based TUS 1.0 implementation. And it would bring me a lot of joy to do this, because I absolutely love Roda. πŸ˜ƒ

janko commented 8 years ago

It is done: https://github.com/janko-m/tus-ruby-server

I also created a demo app which integrates it with Shrine: https://github.com/janko-m/shrine-tus-demo

erikdahlstrand commented 8 years ago

That is truly awesome @janko-m!