Chunked Encoding (Streaming) Example

johnjelinek commented 9 years ago

Does cljs-ajax support comet-style http streaming via chunked-encoding?

JulianBirch commented 9 years ago

Not as yet, although it would be possible. I'm curious as to why you'd want to do this rather than use a websocket, though.

On Friday, April 17, 2015, John Jelinek IV notifications@github.com wrote:

Does cljs-ajax support comet-style http streaming via chunked-encoding?

— Reply to this email directly or view it on GitHub https://github.com/JulianBirch/cljs-ajax/issues/76.

Sent from an iPhone, please excuse brevity and typos.

johnjelinek commented 9 years ago

I'm depending on a streaming webapi and don't want to build an intermediary server to convert the stream into a websocket. Also, I don't need full-duplex communication.

— Sent from Mailbox

On Thu, Apr 16, 2015 at 11:07 PM, Julian Birch notifications@github.com wrote:

Not as yet, although it would be possible. I'm curious as to why you'd want to do this rather than use a websocket, though. On Friday, April 17, 2015, John Jelinek IV notifications@github.com wrote:

Does cljs-ajax support comet-style http streaming via chunked-encoding?

— Reply to this email directly or view it on GitHub https://github.com/JulianBirch/cljs-ajax/issues/76.

Sent from an iPhone, please excuse brevity and typos.

Reply to this email directly or view it on GitHub: https://github.com/JulianBirch/cljs-ajax/issues/76#issuecomment-93889087

JulianBirch commented 9 years ago

Makes sense. What sort of API would you expect in these circumstance (aka lets design it and see if it'll work :))

On Friday, April 17, 2015, John Jelinek IV notifications@github.com wrote:

I'm depending on a streaming webapi and don't want to build an intermediary server to convert the stream into a websocket. Also, I don't need full-duplex communication.

— Sent from Mailbox

On Thu, Apr 16, 2015 at 11:07 PM, Julian Birch <notifications@github.com javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

Not as yet, although it would be possible. I'm curious as to why you'd want to do this rather than use a websocket, though. On Friday, April 17, 2015, John Jelinek IV <notifications@github.com javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

Does cljs-ajax support comet-style http streaming via chunked-encoding?

— Reply to this email directly or view it on GitHub https://github.com/JulianBirch/cljs-ajax/issues/76.

Sent from an iPhone, please excuse brevity and typos.

Reply to this email directly or view it on GitHub: https://github.com/JulianBirch/cljs-ajax/issues/76#issuecomment-93889087

— Reply to this email directly or view it on GitHub https://github.com/JulianBirch/cljs-ajax/issues/76#issuecomment-93891225 .

Sent from an iPhone, please excuse brevity and typos.

johnjelinek commented 9 years ago

(streaming-ajax url (fn [chunk] (println (parse-json-to-map chunk))) "GET")

Maybe something like this, but you'd need to see how that plays well with core.async.

— Sent from Mailbox

On Thu, Apr 16, 2015 at 11:33 PM, Julian Birch notifications@github.com wrote:

Makes sense. What sort of API would you expect in these circumstance (aka lets design it and see if it'll work :)) On Friday, April 17, 2015, John Jelinek IV notifications@github.com wrote:

I'm depending on a streaming webapi and don't want to build an intermediary server to convert the stream into a websocket. Also, I don't need full-duplex communication.

— Sent from Mailbox

On Thu, Apr 16, 2015 at 11:07 PM, Julian Birch <notifications@github.com javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

Not as yet, although it would be possible. I'm curious as to why you'd want to do this rather than use a websocket, though. On Friday, April 17, 2015, John Jelinek IV <notifications@github.com javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

Does cljs-ajax support comet-style http streaming via chunked-encoding?

— Reply to this email directly or view it on GitHub https://github.com/JulianBirch/cljs-ajax/issues/76.

Sent from an iPhone, please excuse brevity and typos.

Reply to this email directly or view it on GitHub: https://github.com/JulianBirch/cljs-ajax/issues/76#issuecomment-93889087

— Reply to this email directly or view it on GitHub https://github.com/JulianBirch/cljs-ajax/issues/76#issuecomment-93891225 .

Sent from an iPhone, please excuse brevity and typos.

Reply to this email directly or view it on GitHub: https://github.com/JulianBirch/cljs-ajax/issues/76#issuecomment-93891309

JulianBirch commented 9 years ago

The thing that's holding me up on this is that I don't have any clojure code to use to test this. I don't suppose you have an example that would work?

cvillecsteele commented 8 years ago

+1111 on this... I need long poll also...

JulianBirch commented 8 years ago

It remains a good idea, and definitely within the remit of the project. I still need someone to come up with some httpkit code I can test against, however. :)

Just something that goes Hello CHUNK Polling CHUNK World will do the trick.

On Tuesday, 26 January 2016, Colin Steele notifications@github.com wrote:

+1111 on this... I need long poll also...

— Reply to this email directly or view it on GitHub https://github.com/JulianBirch/cljs-ajax/issues/76#issuecomment-175071362 .

Sent from an iPhone, please excuse brevity and typos.

RadicalZephyr commented 7 years ago

Hi @JulianBirch, there's a pretty good example of chunked encoding in the httpkit docs here: http://www.http-kit.org/server.html#async

Here's the same example without the implicit use clauses implied by the docs page:

(ns chunked.test
  (:require
    [org.httpkit.server :as s]
    [org.httpkit.timer :as t]))

(defn handler [request]
  (s/with-channel request channel
    (s/on-close channel (fn [status] (println "channel closed, " status)))
    (s/send! channel
             {:status 200
              :headers {"Content-Type" "text/html"}}
             false)
    (loop [id 0]
      (when (< id 10)
        (t/schedule-task (* id 200) ;; send a message every 200ms
                         (s/send! channel (str "message from server #" id) false))
        (recur (inc id))))
    (t/schedule-task 10000 (s/close channel))))

(s/run-server handler {:port 9090})

Folcon commented 6 years ago

@JulianBirch Not sure if you're still interesting in capturing incremental downloads from a streaming http response? I've gotten a version working with goog.net.XhrIo or is that becoming deprecated entirely? Not certain how good the process is, but it seems to work. I'm happy to try and work out some kind of minimal changeset if you're interested?

It's primarily based around hooking into here:

(when (fn? progress-handler)
      (doto this
        (.setProgressEventsEnabled true)
        (events/listen goog.net.EventType.PROGRESS progress-handler)))

It might be worthwhile having different ones for EventType.PROGRESS EventType.DOWNLOAD_PROGRESS EventType.UPLOAD_PROGRESS.

:progress-handler (fn [e] (let [resp-text (.getResponseText (.-currentTarget e))]
                                resp-text))

You can then do something with the partial result, in my case I did some chucked streaming of edn. Bit rough and ready, but it works =)...

JulianBirch commented 6 years ago

Hi, I'm super interested in getting this formalized. There's some obstacles, though, so I'll talk you through where I'm up to in terms of thinking about the problem:

From a different issue, I wrote:

OK, I'm pushing sorting this out properly until 0.8. The existing code will go into 0.7, but it's very XhrIo specific and plain won't work with a Java implementation. I think we're going to need two handlers: upload-progress and download-progress. The callback methods should probably take two parameters: byte counts current and total. total will be null if we can't work it out on download.

Since writing that, I've realised that download-progress also needs a third parameter: the decoded content. (Also, if you have a download progress handler your "final" handler shouldn't have content.)

However, the biggest obstacle is getting some decent integration tests running. I really need to fire up a server and run tests against it (I've got this for Java but not JavaScript). @RadicalZephyr posted some code to this earlier up.

So, I'll happily take any improvements you've got, but to get it as a supported feature I really need the working integration tests.

Folcon commented 6 years ago

@JulianBirch That's great, I've only got it working in ClojureScript land =)...

What kind of tests are you looking for? I've been treating my side pretty shabbily as the system I've built is pretty resilient already to being given bad data, so if it doesn't recognise something it discards it, and it's ok getting repeated entries, so I just dump it in and it gets handled by my downstream code.

I'll be tuning this up later, but it's on the order of months from now...

In terms of decoded content, that get's output from here:

:progress-handler (fn [e] (let [resp-text (.getResponseText (.-currentTarget e))]
                                resp-text))

From my perspective, just being able to hook in here with either goog.net.EventType: PROGRESS, UPLOAD_PROGRESS, or DOWNLOAD_PROGRESS and a function to apply against it solves a host of issues. Where if any of those event functions are defined, trigger (.setProgressEventsEnabled true).

I admit however that I haven't got as clear a view as you do about the overall architecture of the library, so you may have a more elegant approach than "did the user define one of": :progress-handler, :upload-progress-handler, or :download-progress-handler and is the handler a function?

So what do you need from me? I'm happy to chime back with what I can help with.

JulianBirch commented 6 years ago

Okay, the really basic thing that would help me a lot is an addition to the travis CI build that a) starts up http-kit and creates an endpoint called "/chunked". There's an existing http-kit bit in the tests, and it would be nice to only have the one, but I'll happily take it either way.

The end point does the following:

Sends {:a 1}
Sends {:b 2}
Sends {:c 3}
Closes

Then on the other side, a piece of code that connects up and verifies it receives that data in the progress handler.

I'm thinking :download-progress-handler should be a function that takes three parameters:

count of bytes so far
total count of bytes (if Content-Length was set, null if not)
the translated content that was just received

I'm not sure we want :progress-handler on its own: it strikes me as just complicating the API for the sake of a minority of users's convenience.

:upload-progress-handler should probably just be a function taking two parameters

count of bytes so far
total count of bytes (always known this time)

Is this any help?

Folcon commented 6 years ago

So it is and it isn't :)...

That sounds great, however I'm not certain that the download-progress-handler would work as described? At least if it's based on goog.net.EventType.DOWNLOAD_PROGRESS, it might be possible, but so far I've not had much success getting the content just received, it's why I ended up using goog.net.EventType.PROGRESS. This might have been done for performance reasons, or equally I might have missed something. I can definitely give it a shot to confirm though.

Regarding the test case, you want something similar to this I assume? If so I'm happy to put a first pass together that if you're ok to review, I can adjust :)...

JulianBirch commented 6 years ago

Not sure. I think it’s possible with XmlHttpRequest but I could be wrong. (I am planning to make that the default one of these days.)

Yes, that’s the kind of thing. It’s harder to get working when not everything is in Java, sadly.

Looking forward to it

On Fri, 2 Nov 2018 at 19:46, Folcon notifications@github.com wrote:

So it is and it isn't :)...

That sounds great, however I'm not certain that the download-progress-handler would work as described? At least if it's based on goog.net.EventType.DOWNLOAD_PROGRESS, it might be possible, but so far I've not had much success getting the content just received, it's why I ended up using goog.net.EventType.PROGRESS. This might have been done for performance reasons, or equally I might have missed something. I can definitely give it a shot to confirm though.

Regarding the test case, you want something similar to this I assume https://github.com/JulianBirch/cljs-ajax/blob/master/test/ajax/test/server_interaction.clj#L32? If so I'm happy to put a first pass together that if you're ok to review, I can adjust :)...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/JulianBirch/cljs-ajax/issues/76#issuecomment-435473987, or mute the thread https://github.com/notifications/unsubscribe-auth/AAFIkwkb3eaPrBem5PT7phbKPb5ky-zrks5urKEIgaJpZM4ECUqu .

-- Sent from an iPhone, please excuse brevity and typos.

JulianBirch / cljs-ajax