BatchInsert java.lang.StackOverflowError: null

[btbox]

• 批量插入导致 java.lang.StackOverflowError: null

• 我的逻辑是从数据库中分页查询出数据，然后分批次 多线程批量插入，例如：九十万数据，分 2000条数据，总共450页。

• 错误栈如下

Unexpected exception occurred invoking async method: public void com.box.modules.work.graph_import.async.GraphImportAsync.asyncBatchInsertGraph(java.lang.String,java.lang.String,java.lang.String,java.util.Map,java.util.List)

java.lang.StackOverflowError: null
    at com.box.modules.work.graph_import.service.impl.GraphBaseApiImpl.batchInsertVertex(GraphBaseApiImpl.java:126)
    at com.box.modules.work.graph_import.async.GraphImportAsync.asyncBatchInsertGraph(GraphImportAsync.java:45)
    at com.box.modules.work.graph_import.async.GraphImportAsync$$FastClassBySpringCGLIB$$5ed53427.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
    at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

[btbox]

很奇怪的是，我之前是2000为一次batch插入是发生这个异常，换成 1W一次batchInsert没有问题

[btbox]

• 这是我的mapper语句

<insert id="batchInsertVertex" >
        INSERT VERTEX ${tagName}
         (
            @for ( param in params ) {
                ${param}${paramLP.last?"":"," }
            @}
         )
        VALUES
        ${dataStr};
    </insert>

[CorvusYe]

感觉有可能是因为2000条一页的时候开的线程数太多了？

[btbox]

但是我用的是线程池，6 core, 12 线程 队列 是 9999

如上所述，450页的话，就会循环450次 @Async里面的线程方法去batchInsert

@Bean(name = "graphAsyncImportExecutor")
    public Executor graphAsyncImportExecutor() {
        //在这里修改
        ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
        //配置核心线程数
        executor.setCorePoolSize(Runtime.getRuntime().availableProcessors());
        //配置最大线程数
        executor.setMaxPoolSize(Runtime.getRuntime().availableProcessors() * 2);
        //配置队列大小
        executor.setQueueCapacity(queueCapacity);
        //配置线程池中的线程的名称前缀
        executor.setThreadNamePrefix(namePrefix);
        // rejection-policy：当pool已经达到max size的时候，如何处理新任务
        // CALLER_RUNS：不在新线程中执行任务，而是有调用者所在的线程来执行
        executor.setRejectedExecutionHandler( new ThreadPoolExecutor.CallerRunsPolicy());
        //执行初始化
        executor.initialize();
        return executor;
    }

[CorvusYe]

既然线程有不大的上限，所开线程一致的话，按道理2000一次所占用的内存会更小才对，是有些反直觉。 不好意思，刚刚看错了... 还以为是OutOfMemoryException。 报错有 Cause by 信息吗

[btbox]

• 这个是报错信息，我注释掉了batchInsert的语句后，并没有出现该异常了，是不是我的xml语句不正确呢

  
  Unexpected exception occurred invoking async method: public void com.box.modules.work.graph_import.async.GraphImportAsync.asyncBatchInsertGraph(java.lang.String,java.lang.String,java.lang.String,java.util.Map,java.util.List)

java.lang.StackOverflowError: null at com.box.modules.work.graph_import.service.impl.GraphBaseApiImpl.batchInsertVertex(GraphBaseApiImpl.java:126) at com.box.modules.work.graph_import.async.GraphImportAsync.asyncBatchInsertGraph(GraphImportAsync.java:45) at com.box.modules.work.graph_import.async.GraphImportAsync$$FastClassBySpringCGLIB$$5ed53427.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753) at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

[CorvusYe]

xml 看上去没有问题，如果dataStr每条写入值的栏位跟params一致，应该不会有太大问题。