If we create a temporary table for each task, we will end up with a lot of temporary tables when there are many tasks. Therefore, depending on the workload, the number of temporary tables may reach the limit of the number of tables created in the destination database. From what I've read in the code, it looks like it's okay to share temporary tables between tasks if we ignore the resume feature. So, I think we should create an option recycle_temp_table or share_temp_table to allow sharing of temporary tables between tasks. How do you think?
Hi maintainers,
I create an issue related to https://github.com/embulk/embulk-output-jdbc/issues/299.
If we create a temporary table for each task, we will end up with a lot of temporary tables when there are many tasks. Therefore, depending on the workload, the number of temporary tables may reach the limit of the number of tables created in the destination database. From what I've read in the code, it looks like it's okay to share temporary tables between tasks if we ignore the
resumefeature. So, I think we should create an optionrecycle_temp_tableorshare_temp_tableto allow sharing of temporary tables between tasks. How do you think?