trivedigaurav / emr-nlp-server

This repository has been moved to http://github.com/nlpreviz/emr-nlp-server
GNU General Public License v3.0
1 stars 0 forks source link

Can't handle parenthesis in feedback #12

Open trivedigaurav opened 9 years ago

trivedigaurav commented 9 years ago

Both wordtree and highlight span feedback fail if the span includes a parenthesis. For example

Insitu Hybridization Assays (ISH & FISH)

Error trace:

22-Jul-2015 18:36:12.279 SEVERE [http-nio-8080-exec-17] com.sun.jersey.spi.container.ContainerResponse.mapMappableContainerException The RuntimeException could not be mapped to a response, re-throwing to the HTTP container
 java.lang.ArrayIndexOutOfBoundsException: 89
    at edu.pitt.cs.nih.backend.featureVector.Preprocess.separatePathologyHeaderFooter(Preprocess.java:302)
    at edu.pitt.cs.nih.backend.featureVector.ColonoscopyDS_SVMLightFormat.getFullDSFeatureVector(ColonoscopyDS_SVMLightFormat.java:308)
    at edu.pitt.cs.nih.backend.featureVector.ColonoscopyDS_SVMLightFormat.createFullDS(ColonoscopyDS_SVMLightFormat.java:96)
    at edu.pitt.cs.nih.backend.featureVector.ColonoscopyDS_SVMLightFormat.createLearningFileSet(ColonoscopyDS_SVMLightFormat.java:850)
    at edu.pitt.cs.nih.backend.feedback.TextFileFeedbackManagerLibSVM.createLearningFiles(TextFileFeedbackManagerLibSVM.java:339)
    at edu.pitt.cs.nih.backend.feedback.TextFileFeedbackManager_LibSVM_WordTree.processFeedback(TextFileFeedbackManager_LibSVM_WordTree.java:78)
    at frontEnd.serverSide.controller.Feedback_Controller.processFeedback(Feedback_Controller.java:194)
    at frontEnd.serverSide.controller.Feedback_Controller.getFeedback(Feedback_Controller.java:42)
    at frontEnd.serverSide.WSInterface.getFeedback(WSInterface.java:185)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
    at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
    at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
    at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
    at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
    at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
    at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
    at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
    at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
    at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
    at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
    at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
    at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
    at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:540)
    at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:715)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:503)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:136)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:610)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:526)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:655)
    at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:222)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1566)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1523)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:744)
emr-vis-nlp commented 9 years ago

It may not be parenthesis specifically as ""("CLIA") as" indicates any-adenoma to be false." worked just fine...

On Wed, Jul 22, 2015 at 6:40 PM, Gaurav Trivedi notifications@github.com wrote:

Both wordtree and highlight span feedback fail if the span includes a parenthesis. For example

Insitu Hybridization Assays (ISH & FISH)

Error trace:

22-Jul-2015 18:36:12.279 SEVERE [http-nio-8080-exec-17] com.sun.jersey.spi.container.ContainerResponse.mapMappableContainerException The RuntimeException could not be mapped to a response, re-throwing to the HTTP container java.lang.ArrayIndexOutOfBoundsException: 89 at edu.pitt.cs.nih.backend.featureVector.Preprocess.separatePathologyHeaderFooter(Preprocess.java:302) at edu.pitt.cs.nih.backend.featureVector.ColonoscopyDS_SVMLightFormat.getFullDSFeatureVector(ColonoscopyDS_SVMLightFormat.java:308) at edu.pitt.cs.nih.backend.featureVector.ColonoscopyDS_SVMLightFormat.createFullDS(ColonoscopyDS_SVMLightFormat.java:96) at edu.pitt.cs.nih.backend.featureVector.ColonoscopyDS_SVMLightFormat.createLearningFileSet(ColonoscopyDS_SVMLightFormat.java:850) at edu.pitt.cs.nih.backend.feedback.TextFileFeedbackManagerLibSVM.createLearningFiles(TextFileFeedbackManagerLibSVM.java:339) at edu.pitt.cs.nih.backend.feedback.TextFileFeedbackManager_LibSVM_WordTree.processFeedback(TextFileFeedbackManager_LibSVM_WordTree.java:78) at frontEnd.serverSide.controller.Feedback_Controller.processFeedback(Feedback_Controller.java:194) at frontEnd.serverSide.controller.Feedback_Controller.getFeedback(Feedback_Controller.java:42) at frontEnd.serverSide.WSInterface.getFeedback(WSInterface.java:185) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:540) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:715) at javax.servlet.http.HttpServlet.service(HttpServlet.java:725) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:503) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:136) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:610) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:526) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:655) at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:222) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1566) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1523) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:744)

— Reply to this email directly or view it on GitHub https://github.com/trivedigaurav/emr-nlp-server/issues/12.

trivedigaurav commented 9 years ago

Sending in "ISH" alone seems to problematic:

Received span map: {fbId=0, selected=ISH, matched=ISH} ISH 22-Jul-2015 18:54:38.403 SEVERE [http-nio-8080-exec-20] com.sun.jersey.spi.container.ContainerResponse.mapMappableContainerException The RuntimeException could not be mapped to a response, re-throwing to the HTTP container java.lang.ArrayIndexOutOfBoundsException

On Wed, Jul 22, 2015 at 6:53 PM, emr-vis-nlp notifications@github.com wrote:

It may not be parenthesis specifically as ""("CLIA") as" indicates any-adenoma to be false." worked just fine...

On Wed, Jul 22, 2015 at 6:40 PM, Gaurav Trivedi notifications@github.com wrote:

Both wordtree and highlight span feedback fail if the span includes a parenthesis. For example

Insitu Hybridization Assays (ISH & FISH)

Error trace:

22-Jul-2015 18:36:12.279 SEVERE [http-nio-8080-exec-17] com.sun.jersey.spi.container.ContainerResponse.mapMappableContainerException The RuntimeException could not be mapped to a response, re-throwing to the HTTP container java.lang.ArrayIndexOutOfBoundsException: 89 at edu.pitt.cs.nih.backend.featureVector.Preprocess.separatePathologyHeaderFooter(Preprocess.java:302) at edu.pitt.cs.nih.backend.featureVector.ColonoscopyDS_SVMLightFormat.getFullDSFeatureVector(ColonoscopyDS_SVMLightFormat.java:308) at edu.pitt.cs.nih.backend.featureVector.ColonoscopyDS_SVMLightFormat.createFullDS(ColonoscopyDS_SVMLightFormat.java:96) at edu.pitt.cs.nih.backend.featureVector.ColonoscopyDS_SVMLightFormat.createLearningFileSet(ColonoscopyDS_SVMLightFormat.java:850) at edu.pitt.cs.nih.backend.feedback.TextFileFeedbackManagerLibSVM.createLearningFiles(TextFileFeedbackManagerLibSVM.java:339) at edu.pitt.cs.nih.backend.feedback.TextFileFeedbackManager_LibSVM_WordTree.processFeedback(TextFileFeedbackManager_LibSVM_WordTree.java:78) at frontEnd.serverSide.controller.Feedback_Controller.processFeedback(Feedback_Controller.java:194) at frontEnd.serverSide.controller.Feedback_Controller.getFeedback(Feedback_Controller.java:42) at frontEnd.serverSide.WSInterface.getFeedback(WSInterface.java:185) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:540) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:715) at javax.servlet.http.HttpServlet.service(HttpServlet.java:725) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:503) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:136) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:610) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:526) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:655) at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:222) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1566) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1523) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:744)

— Reply to this email directly or view it on GitHub https://github.com/trivedigaurav/emr-nlp-server/issues/12.

— Reply to this email directly or view it on GitHub https://github.com/trivedigaurav/emr-nlp-server/issues/12#issuecomment-123896883 .

trivedigaurav commented 9 years ago

Marking this as CRITICAL as an error like this seems to corrupting the entire model, unless reset...

trivedigaurav commented 9 years ago

Heuristics for separating out headers in pathology reports are wrong. The search strings for pathology start cause an index out of bounds exceptions in some cases.

trivedigaurav commented 9 years ago

b8945764ef8126eb781aa08257367b2c7db9d0d2 fixes this for now. Need extensive testing. Also need to fix ResetDB.jar as it still relies on the same code...

trivedigaurav commented 9 years ago

After this fix, the retraining succeeds with the warning messages like these:

Report:0022_001 does not have pseudo instances
Report:0132_000 does not have pseudo instances
Report:0190_000 does not have pseudo instances
Report:0227_000 does not have pseudo instances
Report:0384_000 does not have pseudo instances
Report:0420_000 does not have pseudo instances
Report:0489_000 does not have pseudo instances
Report:0559_000 does not have pseudo instances
Report:0576_000 does not have pseudo instances
Report:0577_000 does not have pseudo instances
Report:0608_000 does not have pseudo instances
Report:0679_000 does not have pseudo instances
Report:0694_000 does not have pseudo instances
Report:0801_000 does not have pseudo instances
Report:0808_000 does not have pseudo instances
Report:0868_000 does not have pseudo instances
Report:0910_000 does not have pseudo instances
Report:0947_000 does not have pseudo instances
Report:0984_000 does not have pseudo instances