search

brmson / yodaqa

A Question Answering system built on top of the Apache UIMA framework.
http://ailao.eu/yodaqa
Other
619 stars 205 forks source link

New PrimarySearch CASMultiplier causing ConcurrentModificationException #23

Closed zednis closed 8 years ago

zednis commented 8 years ago

I am attempting to add a new data source which references a local ElasticSearch service.

I have created a EsDocPrimarySearch CAS multiplies and a EsDocAnswerProducer based on the current Solr classes and I have added EsDocPrimarySearch AE to the pipeline in YodaQA.java.

You can see the updates in my repo's pull request - https://github.com/brmson/yodaqa/pull/22

edit - link to pull request correctly opened in my fork - https://github.com/zednis/yodaqa/pull/1

When I run the application I get a ConcurrentModificationException after the EsDocPrimarySearch:process method has executed.

Do you have any idea why this might be happening?

Thanks, Stephan

INFO SolrDocPrimarySearch -  QUERY: +("wrote")^1.0 +("Ender's Game")^1.11  ("wrote Ender's Game "~4)^2.1100000000000003 ("wrote Ender's Game "~12)^1.0550000000000002 ("wrote Ender's Game "~36)^0.5275000000000001
Dec 07, 2015 9:04:38 PM cz.brmlab.yodaqa.pipeline.esdoc.EsDocPrimarySearch process(106)
INFO: querying elasticsearch for: [wrote, Ender's Game]
Dec 07, 2015 9:04:38 PM cz.brmlab.yodaqa.pipeline.esdoc.EsDocPrimarySearch process(117)
INFO: elasticsearch response status: OK
Dec 07, 2015 9:04:38 PM org.apache.uima.analysis_engine.impl.AggregateAnalysisEngine_impl processAndOutputNewCASes(274)
SEVERE: Exception occurred
org.apache.uima.analysis_engine.AnalysisEngineProcessException
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB$AggregateCasIterator.processUntilNextOutputCas(MultiThreadASB.java:1074)
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB$AggregateCasIterator.<init>(MultiThreadASB.java:496)
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB.process(MultiThreadASB.java:416)
    at org.apache.uima.analysis_engine.impl.AggregateAnalysisEngine_impl.processAndOutputNewCASes(AggregateAnalysisEngine_impl.java:266)
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB$AggregateCasIterator$1.call(MultiThreadASB.java:772)
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB$AggregateCasIterator$1.call(MultiThreadASB.java:754)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.util.ConcurrentModificationException
    at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429)
    at java.util.HashMap$KeyIterator.next(HashMap.java:1453)
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB$AggregateCasIterator.nextCasToProcess(MultiThreadASB.java:711)
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB$AggregateCasIterator.processUntilNextOutputCas(MultiThreadASB.java:1039)
    ... 9 more

Dec 07, 2015 9:04:38 PM org.apache.uima.analysis_engine.impl.AggregateAnalysisEngine_impl processAndOutputNewCASes(274)
SEVERE: Exception occurred
org.apache.uima.analysis_engine.AnalysisEngineProcessException
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB$AggregateCasIterator.processUntilNextOutputCas(MultiThreadASB.java:1074)
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB$AggregateCasIterator.<init>(MultiThreadASB.java:496)
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB.process(MultiThreadASB.java:416)
    at org.apache.uima.analysis_engine.impl.AggregateAnalysisEngine_impl.processAndOutputNewCASes(AggregateAnalysisEngine_impl.java:266)
    at cz.brmlab.yodaqa.flow.MultiCASPipeline.runAnalysisEngines(MultiCASPipeline.java:135)
    at cz.brmlab.yodaqa.flow.MultiCASPipeline.runPipeline(MultiCASPipeline.java:95)
    at cz.brmlab.yodaqa.YodaQA_Web.main(YodaQA_Web.java:45)
Caused by: java.util.concurrent.ExecutionException: org.apache.uima.analysis_engine.AnalysisEngineProcessException
    at java.util.concurrent.FutureTask.report(FutureTask.java:122)
    at java.util.concurrent.FutureTask.get(FutureTask.java:192)
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB$AggregateCasIterator.collectCasInFlow(MultiThreadASB.java:814)
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB$AggregateCasIterator.casInFlowFromFuture(MultiThreadASB.java:603)
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB$AggregateCasIterator.nextCasToProcess(MultiThreadASB.java:635)
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB$AggregateCasIterator.processUntilNextOutputCas(MultiThreadASB.java:1039)
    ... 6 more
Caused by: org.apache.uima.analysis_engine.AnalysisEngineProcessException
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB$AggregateCasIterator.processUntilNextOutputCas(MultiThreadASB.java:1074)
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB$AggregateCasIterator.<init>(MultiThreadASB.java:496)
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB.process(MultiThreadASB.java:416)
    at org.apache.uima.analysis_engine.impl.AggregateAnalysisEngine_impl.processAndOutputNewCASes(AggregateAnalysisEngine_impl.java:266)
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB$AggregateCasIterator$1.call(MultiThreadASB.java:772)
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB$AggregateCasIterator$1.call(MultiThreadASB.java:754)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.util.ConcurrentModificationException
    at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429)
    at java.util.HashMap$KeyIterator.next(HashMap.java:1453)
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB$AggregateCasIterator.nextCasToProcess(MultiThreadASB.java:711)
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB$AggregateCasIterator.processUntilNextOutputCas(MultiThreadASB.java:1039)
    ... 9 more

Exception in thread "main" org.apache.uima.analysis_engine.AnalysisEngineProcessException
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB$AggregateCasIterator.processUntilNextOutputCas(MultiThreadASB.java:1074)
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB$AggregateCasIterator.<init>(MultiThreadASB.java:496)
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB.process(MultiThreadASB.java:416)
    at org.apache.uima.analysis_engine.impl.AggregateAnalysisEngine_impl.processAndOutputNewCASes(AggregateAnalysisEngine_impl.java:266)
    at cz.brmlab.yodaqa.flow.MultiCASPipeline.runAnalysisEngines(MultiCASPipeline.java:135)
    at cz.brmlab.yodaqa.flow.MultiCASPipeline.runPipeline(MultiCASPipeline.java:95)
    at cz.brmlab.yodaqa.YodaQA_Web.main(YodaQA_Web.java:45)
Caused by: java.util.concurrent.ExecutionException: org.apache.uima.analysis_engine.AnalysisEngineProcessException
    at java.util.concurrent.FutureTask.report(FutureTask.java:122)
    at java.util.concurrent.FutureTask.get(FutureTask.java:192)
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB$AggregateCasIterator.collectCasInFlow(MultiThreadASB.java:814)
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB$AggregateCasIterator.casInFlowFromFuture(MultiThreadASB.java:603)
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB$AggregateCasIterator.nextCasToProcess(MultiThreadASB.java:635)
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB$AggregateCasIterator.processUntilNextOutputCas(MultiThreadASB.java:1039)
    ... 6 more
Caused by: org.apache.uima.analysis_engine.AnalysisEngineProcessException
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB$AggregateCasIterator.processUntilNextOutputCas(MultiThreadASB.java:1074)
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB$AggregateCasIterator.<init>(MultiThreadASB.java:496)
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB.process(MultiThreadASB.java:416)
    at org.apache.uima.analysis_engine.impl.AggregateAnalysisEngine_impl.processAndOutputNewCASes(AggregateAnalysisEngine_impl.java:266)
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB$AggregateCasIterator$1.call(MultiThreadASB.java:772)
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB$AggregateCasIterator$1.call(MultiThreadASB.java:754)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.util.ConcurrentModificationException
    at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429)
    at java.util.HashMap$KeyIterator.next(HashMap.java:1453)
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB$AggregateCasIterator.nextCasToProcess(MultiThreadASB.java:711)
    at cz.brmlab.yodaqa.flow.asb.MultiThreadASB$AggregateCasIterator.processUntilNextOutputCas(MultiThreadASB.java:1039)
    ... 9 more
zednis commented 8 years ago

I found several small issues with by branch and after fixing them I am no longer getting exceptions and the application appears to be working normally.

I am not 100% sure which of the bugs in my code were causing this issue, but I did resolve the following problems

  1. CAS Multiplier next( ) now returning empty JCas instead of null when there are no hits from ES
  2. "empty answer" CAS now correctly has AnswerInfo and ResultInfo being added to indexes
  3. Now using static index during next( ) processing like in SolrDocPrimarySearch
pasky commented 8 years ago

Hi! Sorry I couldn't have been of more help! :-) Hope things work well now. If you are sometime in the mood for talking about what you are working on, I'd love to hear that! (By email, IRC, or here.) Or brainstorm how to improve accuracy.