Allowing Jdbc FetchSize Configuration for handling large rows.
Overview
Default fetch size used by JdbcIO is 50_000 rows.
Large (in terms of memory size) rows can lead to memory errors and JdbcIO recommends tuning fetch size in case of memory errors. Please see here.
Here we allow the user to tune the fetch size via parameters.
Auto inference of fetchsize will be taken as a separate task as it needs careful scale testing.
Allowing Jdbc FetchSize Configuration for handling large rows.
Overview
Default fetch size used by
JdbcIO
is 50_000 rows. Large (in terms of memory size) rows can lead to memory errors andJdbcIO
recommends tuning fetch size in case of memory errors. Please see here. Here we allow the user to tune the fetch size via parameters. Auto inference of fetchsize will be taken as a separate task as it needs careful scale testing.