search

fabric8io / fabric8

fabric8 is an open source microservices platform based on Docker, Kubernetes and Jenkins
http://fabric8.io/
1.76k stars 504 forks source link

Improve logging when connecting to ports that are closed during container-create-ssh 1.x #1036

Open rawlingsj opened 10 years ago

rawlingsj commented 10 years ago

Currently it is not easy to see from the logs when issues occur as a result of closed ports when a remote container is created.

I wonder if we could build in some pre checks to ensure required ports like ZK, JMX & RMI, MavenProxy and Karaf ports are all open.

Below is an example of the current log if a firewall is preventing connectivity on a port when starting the remote container after it has been created using container-create-ssh. It can make it hard investigating issues when creating a fabric in a controlled environment.

2014-04-01 15:03:36,292 | INFO  | FelixStartLevel  | KarArtifactInstaller             | eployer.kar.KarArtifactInstaller   73 | 28 - org.apache.karaf.deployer.kar - 2.3.0.redhat-610376 | Karaf arc
hives will be extracted to /opt/rh/containers/esf02/fabric8-karaf-1.0.0.redhat-376/system
2014-04-01 15:03:36,307 | INFO  | FelixStartLevel  | KarArtifactInstaller             | eployer.kar.KarArtifactInstaller   74 | 28 - org.apache.karaf.deployer.kar - 2.3.0.redhat-610376 | Timestamp
s for Karaf archives will be extracted to /opt/rh/containers/esf02/fabric8-karaf-1.0.0.redhat-376/system/.timestamps
2014-04-01 15:03:40,800 | INFO  | pool-7-thread-1  | Nio2Session                      | .sshd.common.io.nio2.Nio2Session  132 | 35 - org.apache.sshd.core - 0.9.0 | Exception handler threw exceptio
n, closing the session
java.lang.IllegalStateException: No session available
    at org.apache.sshd.common.AbstractSessionIoHandler.exceptionCaught(AbstractSessionIoHandler.java:49)[35:org.apache.sshd.core:0.9.0]
    at org.apache.sshd.common.io.nio2.Nio2Session.exceptionCaught(Nio2Session.java:130)[35:org.apache.sshd.core:0.9.0]
    at org.apache.sshd.common.io.nio2.Nio2Session.access$100(Nio2Session.java:48)[35:org.apache.sshd.core:0.9.0]
    at org.apache.sshd.common.io.nio2.Nio2Session$1.failed(Nio2Session.java:150)[35:org.apache.sshd.core:0.9.0]
    at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:128)[:1.7.0_51]
    at sun.nio.ch.Invoker.invokeDirect(Invoker.java:145)[:1.7.0_51]
    at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implWrite(UnixAsynchronousSocketChannelImpl.java:736)[:1.7.0_51]
    at sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:381)[:1.7.0_51]
    at sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:398)[:1.7.0_51]
    at java.nio.channels.AsynchronousSocketChannel.write(AsynchronousSocketChannel.java:560)[:1.7.0_51]
    at org.apache.sshd.common.io.nio2.Nio2Session.startWriting(Nio2Session.java:143)[35:org.apache.sshd.core:0.9.0]
    at org.apache.sshd.common.io.nio2.Nio2Session.write(Nio2Session.java:119)[35:org.apache.sshd.core:0.9.0]
    at org.apache.sshd.common.session.AbstractSession.writePacket(AbstractSession.java:372)[35:org.apache.sshd.core:0.9.0]
    at org.apache.sshd.server.session.ServerSession.writePacket(ServerSession.java:129)[35:org.apache.sshd.core:0.9.0]
    at org.apache.sshd.common.session.AbstractSession.sendKexInit(AbstractSession.java:711)[35:org.apache.sshd.core:0.9.0]
    at org.apache.sshd.server.session.ServerSession.sendKexInit(ServerSession.java:352)[35:org.apache.sshd.core:0.9.0]
    at org.apache.sshd.server.session.ServerSession.<init>(ServerSession.java:91)[35:org.apache.sshd.core:0.9.0]
    at org.apache.sshd.server.session.SessionFactory.doCreateSession(SessionFactory.java:43)[35:org.apache.sshd.core:0.9.0]
    at org.apache.sshd.common.session.AbstractSessionFactory.createSession(AbstractSessionFactory.java:38)[35:org.apache.sshd.core:0.9.0]
    at org.apache.sshd.common.AbstractSessionIoHandler.sessionCreated(AbstractSessionIoHandler.java:36)[35:org.apache.sshd.core:0.9.0]
    at org.apache.sshd.common.io.nio2.Nio2Acceptor$AcceptCompletionHandler.completed(Nio2Acceptor.java:137)[35:org.apache.sshd.core:0.9.0]
    at org.apache.sshd.common.io.nio2.Nio2Acceptor$AcceptCompletionHandler.completed(Nio2Acceptor.java:118)[35:org.apache.sshd.core:0.9.0]
    at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)[:1.7.0_51]
    at sun.nio.ch.Invoker$2.run(Invoker.java:206)[:1.7.0_51]
    at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)[:1.7.0_51]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_51]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_51]
    at java.lang.Thread.run(Thread.java:744)[:1.7.0_51]
Caused by: java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcherImpl.write0(Native Method)[:1.7.0_51]
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)[:1.7.0_51]
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)[:1.7.0_51]
    at sun.nio.ch.IOUtil.write(IOUtil.java:65)[:1.7.0_51]
    at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implWrite(UnixAsynchronousSocketChannelImpl.java:691)[:1.7.0_51]
    ... 21 more
rawlingsj commented 10 years ago

This was in fact caused because remote containers couldnt SSH back to root fabric server. Added keys for bi-directional SSH fixed the issue.

davsclaus commented 9 years ago

Wonder if the Fuse team should add something about this to a KB or FAQ etc for Fuse 6.2

ffang commented 9 years ago

@rawlingsj Would you please elaborate how you add keys for bi-directional SSH, I run into similar issue but I can't get it work. Thanks.