how to use with ActionController::Live

[johrstrom]

I know the README says

RailsStreaming will not use ActionController::Live
and the ZIP output will run in the same thread as your main request.

But I'm wondering if there's a way to use this library with ActionController::Live. I guess I'm looking for a way to use ZipStreamer#write_deflated_file but on a the Rails response stream object instead of a ZipStreamer object.

I understand if you close this as won't do/can't do. I guess I'm just wondering if there's any way to use ActionController::Live with this library.

[julik]

That's a very valid question, no worries. This is going to be a long comment so strap in :-)

The reason why I decided to remove ActionController::Live from zip_tricks (and not use it in zip_kit) is because the Live responses spin up a separate thread from which your response body gets served. I have tried to figure out why exactly this happens - asked @tenderlove about this - but could not find any pointers. The issue is that this separate thread breaks tests and a few more things that rely on running inside the Rails controller context. For example, the database connection can get switched out for another inside of your streaming thread. Apartment (https://github.com/influitive/apartment) would switch to a different database inside this response writing thread, etc. And it's not really clear why one would want to do this - if you are going to be using a database connection anyway, you will still be using one - just in a different thread.

And in tests ActionController::Live just produces empty responses - at least with rspec-rails - which probably doesn't support ActionController::Live at all. This meant that it was hard to test zip_tricks-generated responses.

Then I started looking and I found that the reason why RailsStreaming was not working was not because I was not using Live, it was because we were not setting headers to disengage the Rack::ETag middleware 🤡 which actually was a Rack-originated issue (see https://github.com/rack/rack/issues/1619) Once that got sorted streaming actually started working without ActionController::Live, and there is no longer that issue with your response body running in a separate thread and all.

That said, you totally can use zip_kit with ActionController::Live. What you would need to do is write your own method which does roughly what the method in zip_kit_stream does, but use the stream object as the output. Roughly like so:

def download_with_live
  headers = ZipKit::OutputEnumerator.new.streaming_http_headers
  response.headers.merge!(headers)

  send_stream(filename: "subscribers.zip") do |stream|
    ZipTricks::Streamer.open(stream) do |zip|
      zip.write_file("some.txt") {|io| io << "Ohai there!" }
    end
  end
end

But did you try using zip_kit_stream without Live first?

[johrstrom]

Thank you so much for the reply!

That said, you totally can use zip_kit with ActionController::Live. What you would need to do is write your own method which does roughly what the method in zip_kit_stream does, but use the stream object as the output. Roughly like so:

Thank you so much for this. I'll close this now as your response is surely enough to get me going.

But did you try using zip_kit_stream without Live first?

Yes this works fine, but we need Live in the API to stream larger files to reduce memory consumption (it's a file CRUD api).

Thanks again for the gem and the response!

[julik]

Yes this works fine, but we need Live in the API to stream larger files to reduce memory consumption (it's a file CRUD api).

Well, hope this works out then. In theory using ZipKit::RailsStreaming should not consume any more memory than using Live (less, actually, because you won't need an extra pthread with a stack and all), but hope this helps. Good luck using zip_kit 🎩

[julik]

Actually come to think of it - RailsStreaming might work out of the box for you also with Live. I will play around - it should not be a big deal if the helper gets included in a controller which also uses Live on other actions

[johrstrom]

Thanks for the additional info. To close the loop on this - in case someone else sees this in the future - the snippet of code you've provided works like a charm.

[julik]

I've made some tweaks in RailsStreaming so now you can just use that with Live 🥳 thank you for bringing this to my attention