Incompatible event loop type with native transport enabled on linux

[anuranjit]

We are seeing high CPU Usage in vertx with eventloop jstack stuck on "at sun.nio.ch.EPollArrayWrapper.epollWait". So we were trying to use native transport on linux using "setPreferNativeTransport()" but we are getting following error.

       java.lang.IllegalStateException: incompatible event loop type: 
          io.netty.channel.epoll.EpollEventLoop
    at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:469)
    at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:80)
    at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:74)
    at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:331)
    at io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:163)
    at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:145)
    at com.github.mauricio.async.db.mysql.codec.MySQLConnectionHandler.connect(MySQLConnectionHandler.scala:87)
    at com.github.mauricio.async.db.mysql.MySQLConnection.connect(MySQLConnection.scala:84)
    at io.vertx.ext.asyncsql.impl.pool.AsyncConnectionPool.createConnection(AsyncConnectionPool.java:63)
    at io.vertx.ext.asyncsql.impl.pool.AsyncConnectionPool.createOrWaitForAvailableConnection(AsyncConnectionPool.java:84)
    at io.vertx.ext.asyncsql.impl.pool.AsyncConnectionPool.take(AsyncConnectionPool.java:93)
    at io.vertx.ext.asyncsql.impl.BaseSQLClient.getConnection(BaseSQLClient.java:68)
    at io.vertx.ext.asyncsql.impl.AsyncSQLClientImpl.getConnection(AsyncSQLClientImpl.java:54)
    at io.vertx.ext.asyncsql.impl.ClientWrapper.getConnection(ClientWrapper.java:52)
    at io.vertx.ext.sql.SQLClient.queryWithParams(SQLClient.java:109)
    at io.vertx.reactivex.ext.sql.SQLClient.queryWithParams(SQLClient.java:218)
    at io.vertx.reactivex.ext.sql.SQLClient.lambda$rxQueryWithParams$6(SQLClient.java:231)
    at io.vertx.reactivex.core.impl.AsyncResultSingle.subscribeActual(AsyncResultSingle.java:42)

We are using vertx-mysql-postgresql-client and it seems to not support native transport. Has anyone seen this problem ?

We are using Vertx: 3.5.3 vertx-mysql-postgresql-client: 3.5.3

[vietj]

Hi

this CPU issue sometimes happens with Netty, you might want to look at your kernel version and in the existing (closed) Netty issues https://github.com/netty/netty/issues?q=is%3Aissue+epollwait+is%3Aclosed https://github.com/netty/netty/issues?q=is:issue+epollwait+is:closed

concerning the mysql driver, I think it is because the underlying wrapped driver is not able to use a native transport.

if you are using Postgresql you could try to use the reactive-pg-client that works with native transport.

Julien

On 3 Jan 2019, at 15:47, Anuranjit notifications@github.com wrote:

We are seeing high CPU Usage in vertx with eventloop jstack stuck on "at sun.nio.ch.EPollArrayWrapper.epollWait". So we were trying to use native transport on linux using "setPreferNativeTransport()" but we are getting following error.

   java.lang.IllegalStateException: incompatible event loop type: 
      io.netty.channel.epoll.EpollEventLoop

at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:469) at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:80) at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:74) at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:331) at io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:163) at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:145) at com.github.mauricio.async.db.mysql.codec.MySQLConnectionHandler.connect(MySQLConnectionHandler.scala:87) at com.github.mauricio.async.db.mysql.MySQLConnection.connect(MySQLConnection.scala:84) at io.vertx.ext.asyncsql.impl.pool.AsyncConnectionPool.createConnection(AsyncConnectionPool.java:63) at io.vertx.ext.asyncsql.impl.pool.AsyncConnectionPool.createOrWaitForAvailableConnection(AsyncConnectionPool.java:84) at io.vertx.ext.asyncsql.impl.pool.AsyncConnectionPool.take(AsyncConnectionPool.java:93) at io.vertx.ext.asyncsql.impl.BaseSQLClient.getConnection(BaseSQLClient.java:68) at io.vertx.ext.asyncsql.impl.AsyncSQLClientImpl.getConnection(AsyncSQLClientImpl.java:54) at io.vertx.ext.asyncsql.impl.ClientWrapper.getConnection(ClientWrapper.java:52) at io.vertx.ext.sql.SQLClient.queryWithParams(SQLClient.java:109) at io.vertx.reactivex.ext.sql.SQLClient.queryWithParams(SQLClient.java:218) at io.vertx.reactivex.ext.sql.SQLClient.lambda$rxQueryWithParams$6(SQLClient.java:231) at io.vertx.reactivex.core.impl.AsyncResultSingle.subscribeActual(AsyncResultSingle.java:42)

We are using vertx-mysql-postgresql-client and it seems to not support native transport. Has anyone seen this problem ?

We are using Vertx: 3.5.3 vertx-mysql-postgresql-client: 3.5.3

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/vert-x3/vertx-mysql-postgresql-client/issues/135, or mute the thread https://github.com/notifications/unsubscribe-auth/AANxiowjZyDsIpG0NxZyqurTG6wZhNjxks5u_hf8gaJpZM4ZoNXK.

[oshai]

Native transport is not supported in the driver at the moment. However, from a first look lt looks pretty straightforward to add that: https://netty.io/wiki/native-transports.html. If you want I can do that.

[vietj]

it means you need to modify and recompile the scala driver I think @oshai or can it be done in the vertx mysql-pg client instead ?

[oshai]

I mean only in the new flavour of jasync.

[vietj]

@oshai ok sounds good too

[oshai]

I created version 0.8.60 of jasync with native epoll support

[andy-yx-chen]

0.8.60 does not pass the test because of the failure of the following stack

[main] INFO 🐳 [postgres:9.6.8] - Container postgres:9.6.8 started
Creating configuration for localhost:32823
failed to create connection
java.lang.IllegalStateException: incompatible event loop type: io.netty.channel.nio.NioEventLoop
        at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:469)
        at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:80)
        at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:74)
        at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86)
        at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:333)
        at io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:163)
        at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:145)
        at com.github.jasync.sql.db.postgresql.codec.PostgreSQLConnectionHandler.connect(PostgreSQLConnectionHandler.kt:94)
        at com.github.jasync.sql.db.postgresql.PostgreSQLConnection.connect(PostgreSQLConnection.kt:108)
        at io.vertx.ext.asyncsql.impl.pool.AsyncConnectionPool.createAndConnect(AsyncConnectionPool.java:98)
        at io.vertx.ext.asyncsql.impl.pool.AsyncConnectionPool.createConnection(AsyncConnectionPool.java:72)
        at io.vertx.ext.asyncsql.impl.pool.AsyncConnectionPool.createOrWaitForAvailableConnection(AsyncConnectionPool.java:128)

[andy-yx-chen]

I know what is the issue, will try to see if we are able to integrate, ideally, vertx and jasync should use the same eventloop group

[vietj]

@andy-yx-chen the same channel implementation too

[andy-yx-chen]

@vietj , yes, you are right, I will try to make the changes in jasync-sql and update it back

[andy-yx-chen]

@oshai I created a pr in jasync to infer SocketChannel type from the EventLoopGroup, which allows the user to decide which event loop group to use instead of forcing users to create eventloop groups from NettyUtils https://github.com/jasync-sql/jasync-sql/pull/69

[oshai]

@andy-yx-chen I released version 0.8.61 with this fix.

[oshai]

Update: I created version 0.8.62 with the native dependencies as optional, so I recommend using that one.

[andy-yx-chen]

@oshai thanks, this is good and bad, good thing, more flexible, bad thing, it may hide the issue in jasync-sql/jasync-sql#69, I am now integrating the change, will add test cases for native support as well

[oshai]

That's how I saw it is recommended, and used also in vertex and other libs. I don't think it will hide the issue in 69, and there is still a way to configure it.

[andy-yx-chen]

@vietj @oshai stay up again, but it's here now, https://github.com/vert-x3/vertx-mysql-postgresql-client/pull/137

[vietj]

great, I've added a comment for testing native transport in Travis.

[andy-yx-chen]

@vietj , updated, good suggestion, thanks!

[andy-yx-chen]

@codepitbull call for review, I think this is important to the project, epoll and kqueue can help to deal with the select issue when there are too many sockets

[codepitbull]

Looks perfect.

[andy-yx-chen]

I think we can close this now