Repair/Replace/Remove Netty

[kdubb]

During #349 I investigate the performance difference between the mainline and this driver. There is a real difference (to say the least) when testing on localhost. It is reduced significantly when testing across a link (even a very fast low latency link). It's worth investigating other options.

One of the key requirements is that it remain asynchronous, as we move forward we will need that functionality.

Options I know of:

• Homegrown Thread Per Connection (NIO or Blocking)
  • Spawn a thread for each connection to handle message delivery while writing directly to the socket. This is a fairly easy to implement solution when using blocking sockets.
  • According to my testing the real win here is to use blocking sockets but I could have done something drastically wrong while using NIO.
• Netty
  • Accept the penalty and work to make it as low as possible
• Netty OIO
  • This is "thread per connection" built on blocking sockets. Unfortunately it currently doesn't work and they've deprecated it. To get it to work for just a test you need to monkey around with it quite a bit. We could take it maintain it for our own use; advantage being that Netty NIO is interchangeable.
• Homegrown NIO
  • Build something catered to our needs but that uses NIO and works similar to Netty. There are obvious speed improvements to be made (e.g. writing to the socket from the connection thread) but I'm not 100% sure why Netty made those choices so this may be a fools errand.

[kdubb]

@normanmaurer I know you're a busy man but was wondering if you might weigh in here. I can elaborate on the performance issues we see (raised in #349) if need be.

[jesperpedersen]

There is http://xnio.jboss.org/ too

[kdubb]

@jesperpedersen I hacked together a simple Socket based implementation that uses a reader thread per connection and checked it into a branch here 60778131164a877de148af0059f0620f6da4b5ec. This hack alone gets makes ng only around 15% slower than og when using localhost and about 7% when going across a link.

Of particular interest. I tried a "slick" implementation first. I used a SocketChannel in blocking mode (aka blocking NIO) because Netty's ByteBuf has awesome features for scattering reads and gathering writes. I used them to buffer writes until flush and pull in separate buffers combining them in to a contiguous buffer without copying.... that was nearly as slow as Netty itself. It appears that using NIO is the issue; or I'm doing something very wrong.

This still uses Netty ByteBuf; if we ditch it that can be removed and all the reference counting that comes with it.

[kdubb]

@jesperpedersen @davecramer After the lengthy investigation & discussion in #349... wondering if you guys have seen this https://openjdk.java.net/jeps/353?

If java.net.Socket is soon to be built on NIO then any performance difference between the two must be vanishing.. and that might not be immediately be in good direction because for all I've seen and read NIO is slower.

[kdubb]

FYI JEP353 is implemented in JDK13

[vlsi]

java.net.Socket + ssl buffers data, and it seems there's no sensible way to check if the connection holds some data or not (e.g. server notifications). That is really painful for copy / replication processing over SSL.

[kdubb]

Sounds like that might be a +1 for Netty.

It's SSL uses Java's SSL handler (or a native OpenSSL/BoringSSL if available & chosen) but is layered in the Netty message stack (you just plug it in for SSL, leave it out for plain). All the normal message notifications are fired when data is received; you can even hook into notifications "below" SSL in the stack (while the packet is still encrypted).

I never did any performance comparison with SSL enabled might be an interesting study though.

[vlsi]

pgjdbc has recently switched to "ssl by default" for security reasons, and the default buffering in SSL implementation in Java did bite us a couple of times :-/

[kdubb]

I've opened issue in netty/netty#9627 to explore any solutions/help the Netty team might be able to offer.

[crinklywrappr]

What is the status of this issue? It looks like a potential solution was presented in the netty issue but nobody followed up.

[kdubb]

@crinklywrappr I've updated the Netty issue, there is no answer as to why blocking sockets are inherently slower than non-blocking Netty.

[davecramer]

@kdubb are they really? I thought that was only the case over very low latency connections. Once their was significant latency the differences become moot ?

[kdubb]

@davecramer You are correct. My testing showed that this issue nearly disappears once operating over any remote link (i.e. not a localhost connection).

My hypothesis is that this is a thread context switching issue, which seems to explain the data I've seen.

[kdubb]

FYI, this issue is still open mostly because I want to explore removing Netty once loom is ready.

[davecramer]

haha, that might be a while :)

[reneleonhardt]

Now that Virtual Threads are being used everywhere, can this "slow Netty" issue picked up again?