neo4j.het.io database access not available

[jdauvill]

Hi, Unfortunately I can't access Hetionet at https://neo4j.het.io/browser/ See screen-shots below:

[dhimmel]

Thanks for the report.

Paging @dongbohu. We'll look into this ASAP, collecting debugging info to see why this has happened. And then restart the server. Should be back up by Tues Sep 14 sometime.

[dongbohu]

The neo4j web server seems working okay now. @dhimmel, did you restart the docker process?

[dhimmel]

No I didn't do anything. I can confirm that as of ~8 hours ago the problem existed, but now does not.

Any chance we can get some debugging info from the logs?

On Tue, Sep 24, 2019, 8:21 AM dongbohu notifications@github.com wrote:

The neo4j web server seems working okay now. @dhimmel https://github.com/dhimmel, did you restart the docker process?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hetio/hetionet/issues/25?email_source=notifications&email_token=AAIQ4B4GPLXFIVTRWRMTAZ3QLIA3ZA5CNFSM4IZJHUQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7OEAXQ#issuecomment-534528094, or mute the thread https://github.com/notifications/unsubscribe-auth/AAIQ4B4JPKRD6PUZKAL3F4DQLIA3ZANCNFSM4IZJHUQQ .

[dongbohu]

@dhimmel: I am in San Diego for DjangoCon now. If you can log into the AWS EC2 instance, feel free to check docker logs. If you can't, I will do it after I am back on Thursday.

[dongbohu]

This seems to be the error when the server was not working on 09/23:

``` 2019-09-23 15:27:43.402+0000 WARN /browser/vendor-37c4c192b97a512ed880.js org.eclipse.jetty.util.SharedBlockingCallback$BlockerTimeoutException java.io.IOException: org.eclipse.jetty.util.SharedBlockingCallback$BlockerTimeoutException at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:213) at org.eclipse.jetty.server.HttpOutput.sendContent(HttpOutput.java:585) at org.eclipse.jetty.servlet.DefaultServlet.sendData(DefaultServlet.java:960) at org.eclipse.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:534) at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) at org.neo4j.server.web.StaticContentFilter.doFilter(StaticContentFilter.java:52) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.neo4j.server.rest.dbms.AuthorizationDisabledFilter.doFilter(AuthorizationDisabledFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.neo4j.server.rest.web.CollectUserAgentFilter.doFilter(CollectUserAgentFilter.java:69) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:499) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:748) Caused by: org.eclipse.jetty.util.SharedBlockingCallback$BlockerTimeoutException at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:195) ... 33 more 2019-09-23 15:27:43.402+0000 WARN /browser/vendor-37c4c192b97a512ed880.js org.eclipse.jetty.util.SharedBlockingCallback$BlockerTimeoutException java.io.IOException: org.eclipse.jetty.util.SharedBlockingCallback$BlockerTimeoutException at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:213) at org.eclipse.jetty.server.HttpOutput.sendContent(HttpOutput.java:585) at org.eclipse.jetty.servlet.DefaultServlet.sendData(DefaultServlet.java:960) at org.eclipse.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:534) at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) at org.neo4j.server.web.StaticContentFilter.doFilter(StaticContentFilter.java:52) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.neo4j.server.rest.dbms.AuthorizationDisabledFilter.doFilter(AuthorizationDisabledFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.neo4j.server.rest.web.CollectUserAgentFilter.doFilter(CollectUserAgentFilter.java:69) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:499) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:748) Caused by: org.eclipse.jetty.util.SharedBlockingCallback$BlockerTimeoutException at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:195) ... 33 more 2019-09-23 15:27:43.408+0000 WARN /browser/app-37c4c192b97a512ed880.js org.eclipse.jetty.util.SharedBlockingCallback$BlockerTimeoutException java.io.IOException: org.eclipse.jetty.util.SharedBlockingCallback$BlockerTimeoutException at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:213) at org.eclipse.jetty.server.HttpOutput.sendContent(HttpOutput.java:585) at org.eclipse.jetty.servlet.DefaultServlet.sendData(DefaultServlet.java:960) at org.eclipse.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:534) at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) at org.neo4j.server.web.StaticContentFilter.doFilter(StaticContentFilter.java:52) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.neo4j.server.rest.dbms.AuthorizationDisabledFilter.doFilter(AuthorizationDisabledFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.neo4j.server.rest.web.CollectUserAgentFilter.doFilter(CollectUserAgentFilter.java:69) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:499) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:748) Caused by: org.eclipse.jetty.util.SharedBlockingCallback$BlockerTimeoutException at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:195) ... 33 more 2019-09-23 15:27:43.408+0000 WARN /browser/app-37c4c192b97a512ed880.js org.eclipse.jetty.util.SharedBlockingCallback$BlockerTimeoutException java.io.IOException: org.eclipse.jetty.util.SharedBlockingCallback$BlockerTimeoutException at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:213) at org.eclipse.jetty.server.HttpOutput.sendContent(HttpOutput.java:585) at org.eclipse.jetty.servlet.DefaultServlet.sendData(DefaultServlet.java:960) at org.eclipse.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:534) at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) at org.neo4j.server.web.StaticContentFilter.doFilter(StaticContentFilter.java:52) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.neo4j.server.rest.dbms.AuthorizationDisabledFilter.doFilter(AuthorizationDisabledFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.neo4j.server.rest.web.CollectUserAgentFilter.doFilter(CollectUserAgentFilter.java:69) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:499) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:748) Caused by: org.eclipse.jetty.util.SharedBlockingCallback$BlockerTimeoutException at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:195) ... 33 more ```

[dhimmel]

org.eclipse.jetty.util.SharedBlockingCallback$BlockerTimeoutException also seems to have been a problem in https://github.com/hetio/hetionet/issues/9, https://github.com/neo4j/neo4j/issues/9209, https://github.com/neo4j/neo4j/issues/4430.

Perhaps we should upgrade to the latest patch release of Neo4j

[dongbohu]

@dhimmel: You mean Community Edition 3.5.11? https://neo4j.com/download-center/#community

[dhimmel]

Currently we use:

https://github.com/hetio/hetionet/blob/268af3727403854d46247124f79d654dd6bf7c0f/hetnet/neo4j/docker/Dockerfile#L1

But there is a 3.2.14-enterprise release, so 6 patch versions ahead of what we're using, each with various changes.

I believe 3.2 is the last Neo4j enterprise versions that are open source. Therefore if we wanted to update to 3.5, we'd have to switch to community. There aren't any major features of enterprise that we use, but sometimes small features get made enterprise. Anyways, lot's of testing will be required to make sure the upgrade doesn't break anything, especially if incrementing major versions or from enterprise to community.

Or perhaps we wait till 4.0 is released and upgrade then.

[dongbohu]

Sooner or later the support of 3.2.X will be stopped and we'll have to do a major upgrade. If we upgrade from 3.2.X to 4.0 directly, wouldn't it be even more risky?

[dhimmel]

Sooner or later the support of 3.2.X will be stopped and we'll have to do a major upgrade

There's no reason why we have to use a supported version. If an old version works that is not supported, we can use it.

But anyways in https://github.com/hetio/hetionet/pull/26, I upgrade to 3.5.12. Didn't seem to break anything and it's nice to have the latest features. Perhaps the upgrade will also fix these downtime issues (or introduce new ones :smiling_imp:).