search

jarohen / chord

A library designed to bridge the gap between the triad of CLJ/CLJS, web-sockets and core.async.
439 stars 40 forks source link

Automatic formatting/parsing of messages on the channels #4

Closed jarohen closed 10 years ago

jarohen commented 10 years ago

As per Henrik Eneroth's comment on Clojure group https://groups.google.com/d/msg/clojure/rnDBK-tcnOQ/I2itOgc33CUJ

jarohen commented 10 years ago

cc @eneroth

eneroth commented 10 years ago

Hey @james-henderson,

Having thought about this a bit, I'm not sure there's any inherent benefit to just wrap strings in (read-string) or so, just to sort of emulate edn support. The cool thing would be if we could leverage the fact that edn is a very efficient format when symbols can be interned and so on. Maybe even Fressian could be an alternative for encoding the information over the channel?

jarohen commented 10 years ago

Hi @eneroth - thanks for getting back!

I'm not sure I really understand your idea about 'just wrapping strings in (read-string)' - what did you mean by this?

I've never used Fressian itself but I have used nippy which seems to have a similar purpose. It's definitely a good idea to offer EDN compression over the channel but I'm not sure I'd include it by default - it's far too useful to debug exactly what has been sent over the wire using browser websocket debuggers. Maybe you could specify a :format when creating the websocket, taking one of #{:edn :edn-compressed :string} etc?

James

eneroth commented 10 years ago

You're right of course, the less complected this is, the better. Could I simply specify whether I'm sending a string or sending byte code, and leave it at that? Then I could use fressian encoding/decoding on both ends while specifying that the content is byte code, for example.

What I meant is that sending edn is of course already possible. I can just stringify it and then use (read-string) on that string on the other end, knowing that edn has been sent.

jarohen commented 10 years ago

What I meant is that sending edn is of course already possible. I can just stringify it and then use (read-string) on that string on the other end, knowing that edn has been sent.

Ah, get it, yes! But having used Chord in a couple of projects now, it's pretty tiresome to remember to do this every time!

Could I simply specify whether I'm sending a string or sending byte code, and leave it at that?

Well, you can provide :read-ch and :write-ch as options, which leaves you free to intercept the websocket with whatever data structure makes sense for you (although I'm not sure how this would react to binary at the moment - worth a look though!)

eneroth commented 10 years ago

True. Actually, transmitting edn would make for a more general case than transmitting strings, so it's probably a better default after all. There'd be a slight overhead for strings, i.e.: "\"string\"" instead of "string", but what the hell. Being able to rely on it being edn every time is probably a more sensible standard.

Yeah, not all websocket clients handle binary very well, but it is possible to detect. So I'd be able to determine when it is possible to use Fressian (or similar) to cut down on the amount of data transmitted, and use stringified edn for the rest. For a large number of clients, Fressian can actually save quite a lot of bandwidth and therefore money if you're hosting on for example Amazon.

Oh, I quite enjoy Chord by the way! Representing websockets as channels make a lot of sense.

jarohen commented 10 years ago

True. Actually, transmitting edn would make for a more general case than transmitting strings, so it's probably a better default after all. There'd be a slight overhead for strings, i.e.: "\"string\"" instead of "string", but what the hell. Being able to rely on it being edn every time is probably a more sensible standard.

Agreed - this one shouldn't be too difficult so I may be able to get this done over the next few days or so

Yeah, not all websocket clients handle binary very well, but it is possible to detect. So I'd be able to determine when it is possible to use Fressian (or similar) to cut down on the amount of data transmitted, and use stringified edn for the rest. For a large number of clients, Fressian can actually save quite a lot of bandwidth and therefore money if you're hosting on for example Amazon.

This one will be more difficult - I'm not too familiar with how well http-kit/js would support byte-arrays so will have to take more of a look into it. I do think it'd be worth doing though - as you say, sending EDN in a binary format can save a lot of bandwidth.

Oh, I quite enjoy Chord by the way! Representing websockets as channels make a lot of sense.

That's good to hear, thanks!

jarohen commented 10 years ago

Hi Henrik - apologies for taking a long time to get this implemented!

I've just pushed a 0.3.0 version with EDN parsing by default; essentially there's a new :format option which takes either :edn or :str (defaulting to EDN) - would be interested to get your thoughts!

It hasn't got support for Fressian as yet but should be reasonably easy to implement over the top (another multimethod implementation)

James

eneroth commented 10 years ago

Cool! Thanks! :-)