search

clj-commons / aleph

Asynchronous streaming communication for Clojure - web server, web client, and raw TCP/UDP
http://aleph.io
MIT License
2.54k stars 241 forks source link

Cannot update netty pipeline with outbound channel handlers #543

Open stijnopheide opened 4 years ago

stijnopheide commented 4 years ago

Currently transforming the netty pipeline with outbound channel handlers on a TCP server does not work. Aleph expects that you only put onto the stream something that can be converted to a ByteBuf by aleph.netty/to-byte-buf.

For incoming channel handlers you can change this behavior by setting :raw-stream? true, however that does not change the behavior on outbound channel handlers. See https://github.com/ztellman/aleph/blob/master/src/aleph/tcp.clj#L47.

Probably this same problem exists for TCP clients.

The use case is adding frame splitters/creators for Protobuf messages (https://netty.io/4.1/api/io/netty/handler/codec/protobuf/ProtobufDecoder.html) which is implemented as netty encoders/decoders. In the TCP handler you would receive a decoded Protobuf message and you have to return a Protobuf message, not a coercable-to-ByteBuf.

KingMob commented 2 years ago

Yeah, by and large, Aleph uses Netty, but doesn't support you using Netty very well. There may have been good reasons for this (like the ref-counting management required when using your own channel handlers), but I'm not privy to Zach's mind.

The general expectation is the main handler you supply takes care of everything. It would be nice to better support channel handlers, though, so if you want to make a PR for UDP and TCP that adds a :coerce-fn key to the options maps, I'd support that.

arnaudgeiser commented 1 year ago

Starting to work on this. We worked a lot with GRPC (and thus Protobuf) during last year, I consider this feature request valuable.

ohbadiah commented 11 months ago

Any update on this, out of curiosity?

arnaudgeiser commented 11 months ago

It's a small change but I had a hard time writing integration tests around GRPC. Not much progress since I decided to give it a shot one year ago.

ohbadiah commented 11 months ago

@arnaudgeiser is there a branch I could take a look at?

ohbadiah commented 11 months ago

I mentioned this issue in Clojurians slack here.

arnaudgeiser commented 11 months ago

I don't have much but the following branch : https://github.com/clj-commons/aleph/compare/protobuf?expand=1