Limit the number of thread which upload and copy data to redshift due to the max connection error

[giwa]

Please see the details: https://github.com/trocco-io/embulk-output-jdbc/pull/2

Short Summary

• The embulk-output-redshift transfers data by uploading them to S3(called UpoadTask) and then scaning the data from s3 for storing(called CopyTask).
• Both UploadTask and CopyTask would be submitted into Thread Pool to execute ansynchronously.
• since batch_size is specified as a relative small value such as 16384(16kb), hence the UploadTask and CopyTask incurred frequently. That results in a high-frequent submission of threads of these tasks while the thread pool just create new thread to execute the tasks as current threads are all occupied.
• All of those above ends in a situation of org.embulk.exec.PartialExecutionException: java.lang.RuntimeException: org.postgresql.util.PSQLException: FATAL: Sorry, too many connections. Please slow down and try again later.

how to fix

• As newCachedThreadPool would create new thread for execution if and only if current thread are all occupied, it seems tenable to pass a maximum of thread to newCachedThreadPool. However, due to the execution mechanism of embulk is ansync we may have troubles in how to determine the max of it. The most simple way to resolve is just make the maximum of each thread pool as a argument. After some performance test, it seems fine to make the default value of thread_maximum as 5.

[giwa]

@dmikurube Please review this PR:)

[hiroyuki-sato]

I read this comment. https://github.com/trocco-io/embulk-output-jdbc/pull/2#issuecomment-806404200

It creates 40 threads( task(8) * thread_maximum(5) ) by default, Correct? (When the number of tasks is 8)

Adding this description may be helpful for users.

[dmikurube]

Ah, if the number of threads is multiplied by the number of tasks, we should take care of that, in addition. Maybe some more updates in the config name, such as: max_s3_upload_threads_per_task