nsoft / jesterj

Document Ingestion Framework for Search Systems
Apache License 2.0
34 stars 33 forks source link

Handle large numbers of error/processing/batched files #157

Closed nsoft closed 3 years ago

nsoft commented 4 years ago
java.lang.IllegalStateException: Batch statement cannot contain more than 65535 statements.                                                                                                 
  at com.datastax.oss.driver.internal.core.cql.DefaultBatchStatement.addAll(DefaultBatchStatement.java:186)                                                                   
  at java.base/java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:608)                                                                                                    
  at org.jesterj.ingest.model.impl.PlanImpl.activate(PlanImpl.java:132)                                                                                                  
  at org.jesterj.ingest.Main.lambda$main$0(Main.java:144)                                                                                                                      
  at java.base/java.lang.Thread.run(Thread.java:834)     
nsoft commented 4 years ago

I saw this limit when I wrote this and didn't think it would be a problem... silly me :)

nsoft commented 3 years ago

Fixed in https://github.com/nsoft/jesterj/commit/b043159b6895ab7455ce2388c40c8c4fbac36a5f#diff-1ca04d7b9652e2bef988bb3ac4e22a9e55369200bfc690d0d20a7ee285b3d211R146