search

stratosphere / stratosphere

Stratosphere is now Apache Flink.
https://github.com/apache/incubator-flink
Apache License 2.0
197 stars 84 forks source link

Offer buffer-oriented API for I/O and replace custom Java NIO TCP/IP code with Netty 4 library #892

Closed uce closed 10 years ago

uce commented 10 years ago

This PR includes an old PR (#511), which has not been merged and replaces the custom Java NIO TCP/IP code with the Netty 4 library on top of it. See #511 and #25 for some notes on the buffer-oriented API.

Netty replaces our custom NIO code, which was used to setup the TCP connections and queue network envelopes. The main Netty-related stuff is located in the runtime.io.network.netty package: NettyConnectionManager and OutboundConnectionQueue are the entry points for connection setup and envelope queueing. OutboundEnvelopeEncoder and InboundEnvelopeDecoder take care of serialization and deserialization. All main components have respective test cases. I also went over the buffer-orientation changes but did not add any further tests since the original PR yet.

I will now run the test job on a cluster and get back to confirm that this also fixes #652.

StephanEwen commented 10 years ago

Nice one!

Merging this after merging the "de-cloudify" code will be fun :D

StephanEwen commented 10 years ago

Merged in b0f8ba06953f978516044db220a89cacbcbb2d5c

Nice work :-)