h2o-ops
commented
1 year ago JIRA Issue Migration Info
Jira Issue: PUBDEV-3141 Assignee: New H2O Bugs Reporter: Darren Cook State: Open Fix Version: N/A Attachments: N/A Development PRs: N/A
Open exalate-issue-sync[bot] opened 1 year ago
h2o-ops
commented
1 year ago JIRA Issue Migration Info
Jira Issue: PUBDEV-3141 Assignee: New H2O Bugs Reporter: Darren Cook State: Open Fix Version: N/A Attachments: N/A Development PRs: N/A
IMPORTANT: This is 3.8.2.11. JIRA refuses to let me set anything later than 3.8.2.9.
Trying getGrids from Flow gives the below error.
This is a deep learning grid, being made in 3 parts, and the 3rd part is still running. However I also got the below stack trace at the end of both the first and second part. (Or maybe it is the same error being returned whenever the grid completes?)
It is a 3-node EC2 cluster, c4.8xlarge, 108 cores in total. Cluster seems fine, and deep learning is still burning up all the cores.
getModels is working fine, though I see model_8 is missing. The first part of the grid should've made 12 models, and I can see model_0 to model_11 are from the first grid. So I guess model 8 crashed, and that is what is causing getGrid to now fail?
(models 12 to 17 are from the 2nd part, so that must've gone to completion.)
Error calling GET /99/Grids
ERROR MESSAGE: null
Caught exception: java.lang.NullPointerException from: hex.ModelMetrics.getMetricFromModel(ModelMetrics.java:76)
(java.lang.NullPointerException) java.lang.NullPointerException hex.ModelMetrics.getMetricFromModel(ModelMetrics.java:76) hex.ModelMetrics$MetricsComparator.compare(ModelMetrics.java:122) hex.ModelMetrics$MetricsComparator.compare(ModelMetrics.java:112) java.util.TimSort.binarySort(TimSort.java:265) java.util.TimSort.sort(TimSort.java:190) java.util.TimSort.sort(TimSort.java:173) java.util.Arrays.sort(Arrays.java:659) java.util.Collections.sort(Collections.java:217) hex.ModelMetrics.sortModelsByMetric(ModelMetrics.java:185) hex.schemas.GridSchemaV99.fillFromImpl(GridSchemaV99.java:121) water.api.GridsHandler.list(GridsHandler.java:29) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:606) water.api.Handler.handle(Handler.java:62) water.api.RequestServer.handle(RequestServer.java:654) water.api.RequestServer.serve(RequestServer.java:595) water.JettyHTTPD$H2oDefaultServlet.doGeneric(JettyHTTPD.java:734) water.JettyHTTPD$H2oDefaultServlet.doGet(JettyHTTPD.java:676) javax.servlet.http.HttpServlet.service(HttpServlet.java:735) javax.servlet.http.HttpServlet.service(HttpServlet.java:848) org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503) org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429) org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) org.eclipse.jetty.server.Server.handle(Server.java:370) org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53) org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971) org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033) org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72) org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264) org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) java.lang.Thread.run(Thread.java:745)