search

Waikato / moa

MOA is an open source framework for Big Data stream mining. It includes a collection of machine learning algorithms (classification, regression, clustering, outlier detection, concept drift detection and recommender systems) and tools for evaluation.
http://moa.cms.waikato.ac.nz/
GNU General Public License v3.0
609 stars 353 forks source link

ConcurrentModificationException in MOA #170

Closed moussaoui07 closed 3 years ago

moussaoui07 commented 5 years ago

Hi,

I use MCOD for anomalies detection in stream with flink but I have an error in the code when i pass the instance to MCOD:

Caused by: java.util.ConcurrentModificationException: null at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909) at java.util.ArrayList$Itr.next(ArrayList.java:859) at moa.clusterers.outliers.MCOD.MCOD.ProcessExpiredNode(MCOD.java:360) at moa.clusterers.outliers.MCOD.MCOD.ProcessNewStreamObj(MCOD.java:398) at moa.clusterers.outliers.MyBaseOutlierDetector.processNewInstanceImpl(MyBaseOutlierDetector.java:157) at etl.samoa.SamoaAnomalyDetection.mapPartition(SamoaAnomalyDetection.java:84) at org.apache.flink.runtime.operators.MapPartitionDriver.run(MapPartitionDriver.java:103) at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:503) at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704) at java.lang.Thread.run(Thread.java:748)

the line of code that causes this error is: outlierDetector.processNewInstanceImpl(samoaInstance);

how can we avoid this error or fix it ?

Thank's in advance !

Cordially, Anissa

csalperwyck commented 5 years ago

Hi,

I am not sure if this can help, but you could also try to plug Flink directly to MOA: https://moa.cms.waikato.ac.nz/moa-with-apache-flink/

For testing purpose, you could also try to limit SAMOA to use only one core so that you don't have concurrent access to the "list".

Best, Christophe

Le mer. 27 mars 2019 à 18:10, moussaoui07 notifications@github.com a écrit :

Hi,

I use MCOD for anomalies detection in stream with flink but I have an error in the code when i pass the instance to MCOD:

Caused by: java.util.ConcurrentModificationException: null at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909) at java.util.ArrayList$Itr.next(ArrayList.java:859) at moa.clusterers.outliers.MCOD.MCOD.ProcessExpiredNode(MCOD.java:360) at moa.clusterers.outliers.MCOD.MCOD.ProcessNewStreamObj(MCOD.java:398) at moa.clusterers.outliers.MyBaseOutlierDetector.processNewInstanceImpl(MyBaseOutlierDetector.java:157) at etl.samoa.SamoaAnomalyDetection.mapPartition(SamoaAnomalyDetection.java:84) at org.apache.flink.runtime.operators.MapPartitionDriver.run(MapPartitionDriver.java:103) at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:503) at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704) at java.lang.Thread.run(Thread.java:748)

the line of code that causes this error is: outlierDetector.processNewInstanceImpl(samoaInstance);

how can we avoid this error or fix it ?

Thank's in advance !

Cordially, Anissa

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Waikato/moa/issues/170, or mute the thread https://github.com/notifications/unsubscribe-auth/AKC21-p3_VRhdMgUs1izuaqpLqkZZVV5ks5va6X-gaJpZM4cOVhC .

moussaoui07 commented 5 years ago

Hello,

Thank you for your reply !

I have already tried to put a single heart with a flink setParallelism at 1 but I always have the competition error !!

So I do not know how to correct this error.

Best, Aniisa

csalperwyck commented 5 years ago

Hello,

I think it's a bug in their code... There a proposal to fix it there but you need to modify the source code: https://groups.google.com/forum/#!topic/moa-users/QhvNcpIRIFg

Best, Christophe

Le sam. 6 avr. 2019 à 10:59, moussaoui07 notifications@github.com a écrit :

Hello,

Thank you for your reply !

I have already tried to put a single heart with a flink setParallelism at 1 but I always have the competition error !!

So I do not know how to correct this error.

Best, Aniisa

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Waikato/moa/issues/170#issuecomment-480487827, or mute the thread https://github.com/notifications/unsubscribe-auth/AKC21x2bExNWIgOnogEb1Q0ly0GrV5_0ks5veGIPgaJpZM4cOVhC .