SSH session crash for unprivileged user

[bmah888]

During the login of an unprivileged user, the enos backend can sometimes generate an exception related to permissions. This doesn't happen all of the time. Back-end stack trace:

163434 [pool-2-thread-5] WARN net.es.enos.kernel.users.Users - authUser entry
163435 [pool-2-thread-5] WARN net.es.enos.kernel.users.Users - looking for key for bmah2
163450 [pool-2-thread-5] WARN net.es.enos.kernel.users.Users - bmah2 has entered correct password
163451 [pool-2-thread-5] INFO org.apache.sshd.server.session.ServerSession - Session bmah2@/0:0:0:0:0:0:0:1:53058 authenticated
163457 [pool-2-thread-7] WARN org.apache.sshd.server.session.ServerSession - Exception caught
java.lang.SecurityException: Illegal Thread access from pool-2-thread-7 onto ENOS Shell User= bmah2
    at net.es.enos.kernel.security.KernelSecurityManager.checkAccess(KernelSecurityManager.java:75)
    at java.lang.Thread.checkAccess(Thread.java:1418)
    at java.lang.Thread.setPriority(Thread.java:1122)
    at java.lang.Thread.init(Thread.java:413)
    at java.lang.Thread.init(Thread.java:349)
    at java.lang.Thread.<init>(Thread.java:595)
    at net.es.enos.sshd.SshdShell.start(SshdShell.java:78)
    at org.apache.sshd.server.channel.ChannelSession.handleShell(ChannelSession.java:433)
    at org.apache.sshd.server.channel.ChannelSession.handleRequest(ChannelSession.java:284)
    at org.apache.sshd.server.channel.ChannelSession.handleRequest(ChannelSession.java:238)
    at org.apache.sshd.common.session.AbstractSession.channelRequest(AbstractSession.java:1083)
    at org.apache.sshd.server.session.ServerSession.running(ServerSession.java:240)
    at org.apache.sshd.server.session.ServerSession.handleMessage(ServerSession.java:216)
    at org.apache.sshd.common.session.AbstractSession.decode(AbstractSession.java:587)
    at org.apache.sshd.common.session.AbstractSession.messageReceived(AbstractSession.java:253)
    at org.apache.sshd.common.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:54)
    at org.apache.sshd.common.io.nio2.Nio2Session$2.completed(Nio2Session.java:231)
    at org.apache.sshd.common.io.nio2.Nio2Session$2.completed(Nio2Session.java:217)
    at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
    at sun.nio.ch.Invoker.invokeDirect(Invoker.java:145)
    at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)
    at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:275)
    at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:296)
    at java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:407)
    at org.apache.sshd.common.io.nio2.Nio2Session.startReading(Nio2Session.java:217)
    at org.apache.sshd.common.io.nio2.Nio2Session$2.completed(Nio2Session.java:232)
    at org.apache.sshd.common.io.nio2.Nio2Session$2.completed(Nio2Session.java:217)
    at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
    at sun.nio.ch.Invoker$2.run(Invoker.java:206)
    at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
163459 [pool-2-thread-7] INFO org.apache.sshd.server.session.ServerSession - Session bmah2@/0:0:0:0:0:0:0:1:53058 closed

[lomaxfrog]

I have not seen this problem for a while. Since many things have changed in the security, this bug may have gone away by now. Bruce, can you confirmed that you have not experienced it as well in the past few weeks ?

[lomaxfrog]

I have seen this error again this week and I found the root cause:

Mina's SSHD / NIO use a ThreadPoolExecutor (java.util.concurrent.TheadPoolExecutor) to handle the asynchronous I/O of the SSH session (i.e. In/Out/Err streams). The implementation of the thread pool dynamically creates and deletes threads in the pool, starting with an empty pool. This operation is performance by the thread that is actually performing the read or write on either of those streams. Those are the Shell threads, executing the user commands and programs and run with the privilege mode of the user. The security manager enforces that threads created by an enos thread keep the same user and privilege level as the creator, so an unprivileged thread cannot create a privilege one.

As a consequence, the first user to log in will set the user/privilege of the threads in the pool. If the pool needs more threads in future, in case of a spike in concurrent users loggued in. If the user is unprivileged, the shell will not be able to set user/privilege of another user.

[lomaxfrog]

Fix consisted of implementing a custom IoServiceFactory that creates the ThreadPool of worker threads with a fixed number (only core threads) and pre start them at boot time. This limits the number of users that can be log in at the same time. A more dynamic ThreadPool would be better but that would be an optimization.

The number of worker threads is a configuration in GlobalConfiguration.

This issue is fixed/closed