Streaming HTTP responses

[SquidDev]

The HTTP API currently buffers responses on the Java side, only firing a http_success/http_failure message once the response has been closed. It would be neat if there was an option to open HTTP handles in "streaming" mode, where the http_success event is queued immediately after headers are received. The various read methods would block until the HTTP body is received:

local response = http.get { url = "https://example.com", streaming = true }
while true do
  -- Read some data, blocking until available.
  local body = response:read(8192)
  if not body then return end

  -- Do something with this chunk of data
end

The main motivation for this is better support for downloading large (or possibly infinitely long) files without having to load the whole thing into memory. More complex use-cases are still better served by websockets.

Some concerns/comments:

• I think the correct API here is that .read() is what's responsible for reading from the socket, yielding if there's insufficient data. This is different to the rest of CC's API; rednet and websockets passively send events - all .receive() is doing is listening to them.

  This is pretty similar to what CCTweaks's tcp support did, and I wasn't wild about it then. Also haven't looked into how this would be implemented on top of Netty or, indeed, if it's possible!

• There's a temptation here to make the Java API only provide streaming sockets, and then pre-process the events in Lua, capturing HTTP handles before they're passed to user code.

  It'd definitely simplify implementation on the Java side, but not 100% sure. It's definitely a bit of a departure from CraftOS's current hands-off design :).

[Commandcracker]

I'm looking forward to HTTP Streaming! 😄

The main motivation for this is better support for downloading large (or possibly infinitely long) files without having to load the whole thing into memory. More complex use-cases are still better served by websockets.

The problem with websockets is that their max message size is 128 * 1024 = 131072 while HTTP's max download size is 16 * 1024 * 1024 = 16777216. This means it would take me around 128 web-socket messages to download the same amount of data. This would take more time, and it is not acceptable for my use case of video streaming. It works for small screens but not if I want to stream to a max sized monitor, so I'm basically forced to use multiple HTTP requests if I want to do that. HTTP Streaming would be perfect for my use case as it removes the need of multiple requests (improving the speed).

In addition, HTTP Streaming would make it possible to have Server-Sent Events (SSE) which I'm a fan of :) Here is a nice write-up that I found about HTTP Streaming, but it is probably more useful for using it regarding the headers.

I might try to implement it my self, but my Java is a bit rusty. (skill issues :D)